Disabled IPv6 and services doesn't start

So I did the remove the package and went through with the update and after a reboot the router is bricked as in not accessible via ssh or any web interface.

Thanks for edits! We will look into it, but there is a high possibility that we would need some more details. So, we will get in touch with you.

Figured out the root cause - bootargs ipv6.disable=1. Tested this now though with https://repo.turris.cz/omnia-hbk/medkit/omnia-medkit-latest.tar.gz but reckon it is the same with 3.11

Apparently none of the servers are booted with the above directive in place.

from netstat with bootargs ipv6.disable=1

Summary

Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 4186/sshd
tcp 0 0 127.0.0.1:9080 0.0.0.0:* LISTEN 4344/python3.6
netstat: /proc/net/tcp6: No such file or directory
udp 0 0 0.0.0.0:514 0.0.0.0:* 4030/syslog-ng
udp 0 0 0.0.0.0:67 0.0.0.0:* 4063/dnsmasq
udp 0 0 224.0.0.251:5353 0.0.0.0:* 4496/umdns
udp 0 0 192.168.1.1:5353 0.0.0.0:* 4496/umdns
netstat: /proc/net/udp6: No such file or directory

from netstat without (removed) bootargs ipv6.disable=1

Summary

Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 4357/lighttpd
tcp 0 0 0.0.0.0:53 0.0.0.0:* LISTEN 3381/kresd
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 4178/sshd
tcp 0 0 127.0.0.1:9080 0.0.0.0:* LISTEN 4315/python3.6
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 4357/lighttpd
tcp 0 0 :::80 :::* LISTEN 4357/lighttpd
tcp 0 0 :::53 :::* LISTEN 3381/kresd
tcp 0 0 :::22 :::* LISTEN 4178/sshd
tcp 0 0 :::443 :::* LISTEN 4357/lighttpd
udp 0 0 0.0.0.0:50176 0.0.0.0:* 4564/ntpd
udp 0 0 0.0.0.0:514 0.0.0.0:* 4026/syslog-ng
udp 0 0 0.0.0.0:53 0.0.0.0:* 3381/kresd
udp 0 0 0.0.0.0:67 0.0.0.0:* 4058/dnsmasq
udp 0 0 224.0.0.251:5353 0.0.0.0:* 4467/umdns
udp 0 0 192.168.1.1:5353 0.0.0.0:* 4467/umdns
udp 0 0 :::53 :::* 3381/kresd
udp 0 0 ff02::fb:5353 :::* 4467/umdns
udp 0 0 fe80::da58:d7ff:fe00:874b:5353 :::*

I would appreciate not to turn this into a discussion of ipv6 and neither that having turned ipv6 turned off in the kernel is rendering the router useless. Instead I sincerely hope that this bug will be patched in the spirit of what TO is promoting.

you have an open Linux distribution in your device and can do anything you would with a normal Linux server.

And since I got ipv6 turned off on any of my other Linux nodes I would expect the to same be feasible here, particularly since it worked until 3.10.8

Did you tried other protocols or just web?

sshd is not impeded and was reachable but without being able to set a root password after medkit reset it would end at the password input.

Apparently lighttpd, kresd and ntpd are not booting up, not sure about unbound since I could not try it.

I hope all our daemons are configured to listen on IPv6 as well, then it depends on the daemon itself what it does when you instead of not providing IPv6 routable addresses disable IPv6 stack completely. My suggestion would be not to put it in bootargs but if you need to then put it in systems sysctl so in case of factory reset/medkit, everything would work, you can reconfigure daemons not to listen on IPv6 and then you can disable IPv6 stack.

That is not the same as disabling the ipv6 stack in the kernel it is only preventing routing of ipv6. There has been no issue with bootargs surviving either TO upgrades or factory/medkit resets. And things worked peachy in this regard until 3.11 came along. And neither sshd nor dnsmasq has any qualms about it, unbound I would reckon also not.

But since ntpd is also impeded and now being provided by busybox I would reckon the culprit somewhere along that line, not sure whether kresd and lighttpd depend on a/o interact with busybox. Those apps should not fail to start in lieu of an ipv6 stack.

Thank you for the suggestion but that is not an option. If TO is forcing ipv6 now this way and diverting from what is being promoted then it is perhaps time to part ways.

So from the description it most likely looks like some daemons like lighttpd and kresd fail to start if you instruct them to bind on IPv6 and they can’t which sounds like reasonable behavior. This might well be caused by some update to the respected software where upstream fixed bug when failing to bind didn’t produced severe enough error.

In the future we plan to update Omnia rescue image to reset u-boot environment during factory/medkit restore. In the meantime you have to either revert bootargs or flash modified medkit where you disable binding to IPv6 for services you need. Omnia is universal and you can do whatever you would do with any Linux distribution, you just have to understand the consequences of what are you doing.

We are not going to disable IPv6 by default, that doesn’t make sense as it would break much more setups. And we are most likely not going to dedicate resources to create a workarounds for exotic setups like yours. But feel free to send a pull request with workaround and if it doesn’t break other setups, we will include it.

1 Like

It most certainly is not, rather the contrary proven by other applications not suffering from such design. It is entirely illogic to break just because there is no ipv6 stack available but the ipv4 stack instead.

Operating a couple of Linux nodes, bare metal/kvm/lxc and all of them have the ipv6 stack disabled as boot argument and not a single package breaking.

It does not make sense that it breaks applications, by no means.

Nothing exotic about it in the wider Linux space. TO is just not delivering on what it is promoting since this

apparently does not hold true, this being a point in case.

Well, isn’t that the usual response, it is getting old really.

If configuration requests binding on IPv6 and it’s not possible, I agree the service should fail. Whether the default configuration should be more flexible in this respect, that’s perhaps questionable, but here I really understand the desire to focus on less exotic use cases.

Since when sole ipv4 app performance became exotic or the other way around where is the design guideline for apps to fail in lieu of an ipv6 stack whilst the ipv6 stack is available?

It has been voiced earlier by TO that they would prefer ipv4 being phased/chased out and this is certainly one way to go about.

If it can’t bind to the address it is instructed to, yes it is.

And you set those packages to listen on IPv6 and they all failed silently right?

Reread explanation. Big parts of the world are embracing future and not fearing the change, so we need to ensure that the common setup works.

If you want something done and you are the only one that wants it that way, you either reconsider your desires or you fix it.

This is my last response in this conversation, everything was explained.

Nope, but then I do not utilise the packages failing here in my other other nodes, there it is unbound, nginx and ntpd (stand-alone and not part of busybox).

Which seems more of a design/logic error.

@miska @vcunat

Out of curiosity I just installed knot resolver on a node (centos 7.x and 4.18.x kernel) with the ipv6 stack in the kernel boot arg disabled and without adjusting any settings of kresd, which I am not familiar with anyway, kresd starts up and is running/performing out of the box without any issue. Thus there is no misconception in the application design and it is working as expected.

The configuration is completely different. There kresd doesn’t bind anything – that’s done by systemd and the sockets are passed to unprivileged kresd.

Can you explain/link how it’s related to nontrivial change in IPv4 performance? I understand that it’s not ideal to distribute IPv6 addresses locally when you don’t have IPv6 connectivity, but you don’t need to disable the stack to avoid that.

BTW, thanks Pepe for splitting this thread.

@Pepe thanks for the split :+1:

@vcunat it is a Linux system like it is being claimed for TOS. And I do not know how such design of killing the app rather than dropping the ipv6 stack and continuing with the ipv4 stack only is the golden standard/future. Any other app not suffering such design must be wrong then.

Unbound has an option to integrate with systemd (use-systemd) but even if not utilizing systemd it does not get killed, just as an example.

Disabling the stack is not about performance but a security measure. Any non-existent stack is one less potential attack surface. And since my network designs for the time being are solely based on ipv4 there is no need for any ipv6 socks hanging about.

For instance wireguard is not binding and spawning all over the place and ipv6 cannot be turned off. Their developers trust this being a good design and prefer not to add nobs but let other apps handle the fencing. Thus turning off ipv6 in the kernel boot arg is one big knob for less to be concerned about and has not had any inclement effect on any of the Linux nodes I am administrating, except now for this case.

And of course with the settings in sysctl preventing the ipv6 routing the kernel would still have to deal with the packets and thus adding cost whereas with the bootarg it does not.

If you don’t instruct kresd to bind to IPv6 addresses, it won’t even attempt to do so (and won’t crash). It’s about configuration in TurrisOS really.

I would not concur with that perspective. Binding is good but the app’s discovery and handling of ipv4/6 socks seems illogical. If my network design was today ipv6 solely I would drop ipv4 socks from the kernel the same way. Does that crash kresd too? It cannot be expected that each node runs a hybrid, there are indeed folks opting for one or the other.

If one of either is discovered as available and the other not what is the logic behind the decision to cease app’s activity entirely as opposed to continue with whichever socket is usable?

What about those apps that do not crash in such setup - wrong in their design?

To be short, you probably want to change your /etc/config/resolver

config resolver 'common'
        list interface '0.0.0.0'
        list interface '::0'

and adapt the addresses to your particular need :wink: