Turris 1.x - kernel 5.15: router does not want start properly

Software SW bugs discussion
,
#1

Hi, i am using Turris OS 6.0 (HBL) on the Turris 1.x and this morning my router was not able to boot correctly. Also DHCP was broken, so i was able to connect it only with the static ip and no internet was working. After connection to the router i found that router failed to start properly with a huge CPU load and LA > 8. Commands like ifconfig or ip addr were hanging. Reboot did not help. After all i found that router was upgraded this night and problem started after reboot, so i downgraded to the latest pre-upgrade snapshot and disabled automatic upgrades for now.

Below is an os-release diff between working and failed openwrt. Hope this would help:

--- @287/etc/os-release 2022-07-17 18:45:26.000000000 +0200
+++ @288/etc/os-release 2022-08-03 19:07:00.000000000 +0200
@@ -7,7 +7,7 @@
 HOME_URL="https://www.turris.cz/"
 BUG_URL="https://gitlab.nic.cz/groups/turris/-/issues/"
 SUPPORT_URL="https://www.turris.cz/support/"
-BUILD_ID="r16616+81-6f89233c41"
+BUILD_ID="r16625+110-f94b30d83c"
 OPENWRT_BOARD="mpc85xx/p2020"
 OPENWRT_ARCH="powerpc_8540"
 OPENWRT_TAINTS="busybox"
@@ -15,4 +15,4 @@
 OPENWRT_DEVICE_MANUFACTURER_URL="https://www.turris.cz/"
 OPENWRT_DEVICE_PRODUCT="Turris 1.x"
 OPENWRT_DEVICE_REVISION="v0"
-OPENWRT_RELEASE="TurrisOS 6.0 6f89233c41ae8af4537e64684fa274ab4a21e9d4"
+OPENWRT_RELEASE="TurrisOS 6.0 f94b30d83c2aa401b3ff9c4914bb2ef6386df8aa"

Also i see that kernel was upgraded from 5.4.203 to the 5.15.58 what likely is a root cause.

-Version: 5.4.203+5.10.110-1-1-f70e9c9745643e220f2338b431a1b5ff
-Depends: kernel (=5.4.203-1-f70e9c9745643e220f2338b431a1b5ff), kmod-mac80211
+Version: 5.15.58+5.15.33-1-1-9a2687e627ae31779792480683f13d02
+Depends: kernel (=5.15.58-1-9a2687e627ae31779792480683f13d02), kmod-mac80211

cc: @Pepe

#2

Thanks for reporting, I’ll look at it.

1 Like
#3

Commands like ifconfig or ip addr were hanging.

This I can reproduce, but just because the network stack takes too long until it is fully loaded. In my case, it took almost 9 minutes after everything is settled down. After that, it works as it should.

We are investigating that.

#4

Thank you for checking. In my case internet was broken completely, thats how i found that there is an issue. Also i have a syslog forwarding set to another host (RPI) but i think syslog-ng was never able to start, as i found that all logs after upgrade are missing. I will be on vacation next 2 weeks, but happy to test whatever is needed on return ) And hope that ToS 6 will be released eventually, so will switch back to stable.

#5

Yes I experience same issues on both Turris 1 and Omnia as well.
It works like after reboot I had connection to wan for while then it disappear and sometimes it get back. I did not know that it could take up to 9 minutes but I experienced same behaviour.
Not knowing this is a new feature I get quite mad fiddling with both devices for past two weeks.
I was suspicious that my old images of TOS 6 HBL from march on omnia went somehow broken over time with all the daily updates in HBL so I decided to rolback to fresh factory medkit and do new configuration. Not knowing such important information you could imagine how much time I spent refreshing end experimenting and trying to figure out what is wrong. Hope that this will not be marked as another offtopic story.

#6

No, this happens only on Turris 1.x routers. We don’t know the culprit yet, but we are working on it. If you experience it on Turris Omnia router, then I am pretty confident that it is related to the misconfiguration on your end.