Turris OS 5.3.3 auto update from 3 - issues encountered (and some to resolve)

Imagine my surprise this morning to find all my websites down. A quick look and last night at 2am:

##### News announcements #####
The automatic migration to the latest Turris OS version was enabled for your router. The next update is going to install package 'tos3to4' and initiate the migration.

More info is available in the official documentation https://docs.turris.cz/geek/tos3-migration/.

##### Update notifications #####
 • Installed version 1.5 of package tos3to4-notify

So panic stations this morning to find out what was wrong.

Problem 1: kresd name resolution failed

Lord knows why. It came good all by itself without a reboot before I could even find out why. All I did was try to ssh to my router by its domain name and it failed to resolve, so I ssh to it using its IP address and then poke around a few logs and check port listeners netstat -tulpen being my goto and I see kresd is there listening so I try again and names now resolve. We’ll write that one off as a transient hiccup.

Problem 2: Foris and Luci down

Turns out lighttpd is not up. Grrr. Nothing in logs, so I run it manually:

lighttpd -f /etc/lighttpd/lighttpd.conf

and I get ssl errors. Lots of on-line searching (not much help) and poking around locally and head scratching and happens that lighttpd’s default config is broken. In:

/etc/lighttpd/conf.d/ssl-enable.conf

which is part of the lighttpd-https-cert package

the config:

    ssl.openssl.ssl-conf-cmd = (
      "MinProtocol" => "TLSv1.2",
      "Options" => "-ServerPreference",
      "CipherString" => "ECDHE+AESGCM:ECDHE+AES256:CHACHA20:!SHA1:!SHA256:!SHA384"
    )

must now be global or wrapped in $SERVER["socket"]. Efforts to make it global by wrapping it global { ... } did not help so moved into the $SERVER["socket"] sections and then (after one other small tweak on account of a missing Foris conf) lighttpd started and Luci and Foris were up again.

Conclusion: the lightttpd ssl module got an upgrade but the lighttpd-https-cert did not.

Problem 3: DDNS updates failed

I rely on DDNS. And Luci wasn’t even showing me DDNS details any more. Aaarg. Checked packages and ddns-scripts and luci-app-ddns packages were gone. Reinstalled them. Rebooted and all my DDNS updates went ahead and the Lucy app was back.

Problem 4: reforis is broken

This is new. In fact I know that because I use the router as a reverse proxy and had issues years ago with Foris SSL so have this in my in lighttpd.conf:

# Prohibit foris and luci from any WAN ip (server only to LAN ips)
$HTTP["remoteip"] == "192.168.0.0/16" {
    # SSL fix needed alas. Foris depends on a package that creates
    #    /etc/lighttpd/conf.d/ssl-enable.conf
    # whenever an update is installed. This file delivers an SSL certificate that the FORIS
    # web interface currently depends on. Bothersome to say the least. So we have to not load
    # that file if we're delivering an SSL certificate for one of the sites we configured
    # in LAN-servers.conf that needs a certificate. Otherwise lighttpd delivers two
    # certificates.
    #
    # So we can't do the following:
    #
    #include "/etc/lighttpd/conf.d/*.conf"

    # Don't load the foris and luci configs if accessing a site that uses its own SSL
    # certificates. As we we don't want to deliver two certificates.
    $HTTP["host"] !~ "(my WAN domains)" {
        include "/etc/lighttpd/conf.d/foris-config.conf"
        include "/etc/lighttpd/conf.d/foris-root.conf"
        include "/etc/lighttpd/conf.d/foris-ws.conf"
        include "/etc/lighttpd/conf.d/ssl-enable.conf"
        include "/etc/lighttpd/conf.d/luci.conf"
    }
}

But foris-root.conf disappeared and so I replace it with reforis.conf to get lighttpd to start earlier. Still on Foris the reforis links like:

http://myrouterlan/reforis/openvpn/client-settings/

return a 404 and the requests leave a trace in the lighttpd access log but nothing in the error log.

I’d love some tips on getting reforis to work, I’ll have to park that as simply broken for now as I’ve spent all morning getting this far and have an operational router again bar this issue as far as I can tell.

The up side

Foris reports I’m on Turris OS 5.3.3.

Thanks for the update, it’s about time. A shame it cost me so much time and is still a little broken

The down side

Majordomo is a loss. Why on earth was it not maintained? Pakon performs dreadfully by comparison (I do use it) and Luci stats is not a replacement.

And the reforis issue is resolved. Found this on another thread:

opkg-cl update
opkg install reforis-netboot-plugin

and after that reforis is working. Imagine my surprise, that it looks like a whole reimplemetation of foris ;-). Well, not quite, missing some Foris things but hey.

1 Like

Ad 2) It’s interesting to me to hear that lighttpd-https-cert was not updated. While looking at Turris OS 3.x and Turris OS 5.x, there is a different version of lighttpd-https-cert. :frowning_face:

Ad 3) It would be good to know how did you installed DDNS packages. Was it installed manually by using CLI/LuCI or using package list from reForis?

Ad 4) I am sorry, but I am afraid that we are not able to do anything with it as this is a custom configuration and it is not good if we touch it anyhow.

Not everything is lost since routing&switching was working. :+1:

People are leaving, LuCI package was not maintained for more than 6 years, majordomo itself 4 years and it would need complete refactor as it was not compatible with newer LuCI. I can speak of Pakon and there will be some promising improvements. Later, we will not be using Suricata as it requires Rust, which is not available in OpenWrt.


But hey thanks for summarising it here. It is detailed and with solutions. This is how it should look. I will arrange to ship you a few goodies. I will get in touch with you soon.

Re: 2, yep that was a surprise.
Re: 3, I used opkg.
Re: 4, fret not it is resolved as explained in my reply to my OP. That said, I suspect it would be a good thing if Turris tested the probably not that uncommon use case for such a wonderfully configurable and powerful gateway router as a reverse proxy serving SSL certs for sites behind it. It’s because I did this and the Foris config sent by default a conflicting SSL cert and things did not work that I need that custom config block which loads the Foris confs only when it’s a site I’m serving behind the reverse proxy. And sure, that’s my thing to manage, but I suspect it’s not a super rare use case, and the clash should be more managable and a tested use case IMHO. IN fact it would be a wonderful feature if Foris has a nice SSL cert configuration tool built in, that let you specify the domain name and set up the cert with certbot and letsencrypt so that the challenges work and the certs are created and then renewed as needed. I mean I do all that and am not fussed, just that I figure it’s not a rare use case and would make that whole process so much easier and hence make it even easier for people to do that from behind a Turris router. Instead, currently after a lot of reading and testing it sort of fails because of a Foris conf which I therefore wrapped years ago in condition that prevented them loading for any of the domains I serve.

Re: Pakon, I love it in principle and I want it, I have kids behind this same router and want to keep an eye on responsible use and be able to block certain domains etc. Alas I clicked on Pakon this morning and now easily 8 or more hours later I still see:

which is possibly because i have a LOT of data and it’s trying to load it all. Which is clearly problematic in any case, and this loader could be smarter. Ideas include a background process that analyzes the data and produces summaries for a quick Pakon page load, that the loader times out after as little as 30 seconds (already a really slow page load) and uses just the data it’s loaded to that point with a notice that there was a lot more data (maybe some summary of how much) but only displaying some of it. Or defailinting to a view of the last day or week only and permitting a drill down into older data etc … loads of ideas. Whatever is causing it to hang for over 8 hours spinning a wheel is not good, lacking feedback as to what is up. And that sort of thing should be easy to avoid with a smarter approach to “Loading data”.

Anyhow on the whole I am happy. I mean it’s important for us to keep up to date with OpenWRT evolutions and Turris OS evolution and we were stuck in v3 backwater for a long time and are back on the train and that is great. And it could have gone much worse, I do have a functional router and all is working as far as I can tell as it was which rocks. That it only cost me a few hours to bring up again is a blessing of sorts, it might have gone very pair shaped.

I was umming an ahing at what point I stop ploughing forward and bail to restore a schnapps snapshot. Schnapps alone makes the Turris such a darling product! But I saw light at the end of the tunnel before bailing and ploughed on.Every diagnostic step as a rule was rewarded with an insight and new thing to look at so I didn’t get stuck.

I did find schnapps shortcoming. It seems I can’t get a diff of a specific file between the current file and the last snapshot. I can only do it by getting a full diff, not for a specific file. Would be grand if there was a CLI shortcut documented for “the current filesystem” in lieu of a snapshot number. I though their might be one and tried a few things like -, #, etc but had no luck! So had to do a full diff to find what I wanted (how a file looked before I started tweaking it ;-).

1 Like