Reboot fails, but automatic update works - despite being disabled

Hi,

I was logged in today to my home router through the WAN connection which is an LTE USB stick, when I got a mail from the automatic updater:

##### Restart is needed #####
The OS kernel has been updated to version 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0. Please reboot the device.

The device will be restarted automatically on Friday, September  8 at 03:30 AM.

Interestingly, the updater did not install any of the 4.4.79 kernel modules, but it removed the current kernel, which is 4.4.74. Anyways, I thought it knew what it would do, so I issued a “reboot” command.

After a couple of seconds I got kicked out from SSH, as expected, but the router has been still running since then. I can still ping its IP address.

As I said, automatic updates were disabled some months ago (as it screwed up/overwrote my customised configuration without notice), so it might have been fixed already. I cannot even tell you the OS version I run currently, because I cannot log in back.

Did anyone else experience something similar? Reboot does not work, but the disabled auto updater still works?

Oh, BTW, and cron seems to be dead, too. I had a cron job which runs at every 5 minutes, grabs a URL on an external webserver I manage (the point is to log the router’s external IP). This job has also stopped since the reboot.

Update: currently it looks like the router is screwed up. I asked a family member to reboot it and got the information back that the front LEDs change colours between all blue / some of them turn to green. Which would be fine, because I set green for the GE ports.

But the WAN connection does not come up, and the wifi LEDs (yellow) also stay dark.

If this reboot happens on ths Friday early morning as stated in the mail, the whole process would have gone down in an uncontrolled way. I wouldn’t call the automatic updater a success story at the moment.

What is the status of committing back the Omnia-specific changes to OpenWRT, so we can use the stock firmware?

I managed to roll back to the previous snapshot, kudos for this feature. I wish it was already so integrated in Linux like beadm in Solaris and FreeBSD.

But the automatic updater fucked up my system and it got into a reboot loop. So some additional information for debugging:

  • Turris OS is 3.7.1.
  • Automatic updates are DISABLED.
  • Yet it updated ONLY the kernel, WITHOUT first making a snapshot.

My bet is that some kernel module was missing, so the watchdog resetted the router (as I read it in some other thread). Here are the differring files between the 2 snaps, it’s clearly visible that there are no kernel modules.

root:# schnapps list
    # | Type      | Date                      | Description
------+-----------+---------------------------+---------------------------------
    2 | rollback  | 2017-02-19 22:59:56 +0000 | Rollback to snapshot 1
    3 | rollback  | 2017-02-19 23:06:03 +0000 | Rollback to snapshot factory
   12 | pre       | 2017-06-11 17:23:12 +0200 | Automatic pre-update snapshot
   13 | pre       | 2017-06-11 18:33:40 +0200 | Automatic pre-update snapshot
   14 | pre       | 2017-06-26 22:47:30 +0200 | Automatic pre-update snapshot
   15 | post      | 2017-06-26 23:02:09 +0200 | After updating to 3.7 to diff modifications
   16 | pre       | 2017-06-26 23:24:10 +0200 | Automatic pre-update snapshot
   17 | pre       | 2017-06-30 21:48:15 +0200 | Automatic pre-update snapshot
   18 | single    | 2017-07-01 10:46:34 +0200 | After update to 3.7.1
   21 | time      | 2017-07-30 01:05:02 +0200 | Snapshot created by cron
   22 | time      | 2017-08-06 01:05:02 +0200 | Snapshot created by cron
   23 | time      | 2017-08-13 01:05:01 +0200 | Snapshot created by cron
   24 | time      | 2017-08-27 01:05:01 +0200 | Snapshot created by cron
   25 | time      | 2017-09-03 01:05:01 +0200 | Snapshot created by cron
   26 | rollback  | 2017-09-05 15:32:00 +0000 | Rollback to snapshot 25
root:# schnapps cmp 25 26
Comparing snapshots 25 and 26
This can take a while, please be patient.
Meaning of the lines is following:

   - file    file present in 25 and missing in 26
   + file    file not present in 25 but exists in 26
   ~ file    file in 25 differs from file in 26

 - /boot/zImage-4.4.74-1-967673b9d511e4292e3bcb76c9e064bc-0
 + /boot/zImage-4.4.79-1-967673b9d511e4292e3bcb76c9e064bc-0
 ~ /etc/config/system
 + /etc/crontabs/root
 ~ /etc/hosts
 ~ /root/.viminfo
 ~ /run/blkid/blkid.tab
 ~ /run/blkid/blkid.tab.old
 ~ /srv/lxc/deimos/deimos.log
 ~ /srv/lxc/deimos/rootfs/var/log/messages
 ~ /srv/lxc/lxc-monitord.log
 ~ /usr/lib/opkg/info/kernel.control
 ~ /usr/lib/opkg/info/kernel.files-md5sum
 ~ /usr/lib/opkg/info/kernel.list
 ~ /usr/lib/opkg/info/kernel.postinst-pkg
 ~ /usr/lib/opkg/status
root:# uname -r
4.4.74-967673b9d511e4292e3bcb76c9e064bc-0
1 Like

A hopefully final update.

After reenabling automatic updates from Foris I ran updater.sh by hand and it could successfully upgrade the OS to 3.7.5. After reboot everything came back as it should, I really wish the updater script will not go bonkers again as it did previously.

1 Like

The same bug is present in 3.8, too.

Meanwhile I guess I found the problem: 3.8 was published and, unlike in stock OpenWRT, the repo configurations are not wired to a specific OS version. So when I install a kernel module by hand, it will pull in the new kernel as a dependency and remove the older one. For some reason in the previous occasions I overlooked it (curl is too chatty), but tracing back the file time stamps under /usr/lib/opkg/info made it clear.

# opkg list-installed|grep 4.4.7
kmod-ath - 4.4.79+2016-10-08-9-967673b9d511e4292e3bcb76c9e064bc-0
kmod-ath10k - 4.4.79+2016-10-08-9-967673b9d511e4292e3bcb76c9e064bc-0
kmod-ath9k - 4.4.79+2016-10-08-9-967673b9d511e4292e3bcb76c9e064bc-0
kmod-ath9k-common - 4.4.79+2016-10-08-9-967673b9d511e4292e3bcb76c9e064bc-0
kmod-atm - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-bridge - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-cfg80211 - 4.4.79+2016-10-08-9-967673b9d511e4292e3bcb76c9e064bc-0
kmod-crypto-aead - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-crypto-authenc - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-crypto-des - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-crypto-ecb - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-crypto-hash - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-crypto-manager - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-crypto-marvell-cesa - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-crypto-null - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-crypto-pcompress - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-crypto-sha1 - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-crypto-user - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-cryptodev - 4.4.79+1.8-mvebu-2
kmod-ebtables - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-gre - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-hwmon-core - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-i2c-core - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-i2c-mv64xxx - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-ifb - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-input-core - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-ip6-tunnel - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-ip6tables - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-ipt-conntrack - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-ipt-conntrack-extra - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-ipt-core - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-ipt-ipopt - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-ipt-nat - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-iptunnel - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-iptunnel4 - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-iptunnel6 - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-ledtrig-default-on - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-ledtrig-gpio - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-ledtrig-heartbeat - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-ledtrig-morse - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-ledtrig-netdev - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-ledtrig-netfilter - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-ledtrig-oneshot - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-ledtrig-timer - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-ledtrig-transient - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-ledtrig-usbdev - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-lib-crc-ccitt - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-lib-textsearch - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-libphy - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-llc - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-mac80211 - 4.4.79+2016-10-08-9-967673b9d511e4292e3bcb76c9e064bc-0
kmod-mii - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-mmc - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-mppe - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-mvsdio - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-nf-conntrack - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-nf-conntrack-netlink - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-nf-conntrack6 - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-nf-ipt - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-nf-ipt6 - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-nf-nat - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-nf-nathelper - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-nf-nathelper-extra - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-nfnetlink - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-nls-base - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-ppp - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-pppoa - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-pppoe - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-pppox - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-rtc-marvell - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-sched - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-sched-core - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-scsi-core - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-sit - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-slhc - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-sound-core - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-stp - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-swconfig - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-switch-mvsw61xx - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-thermal - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-thermal-armada - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-tun - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-usb-core - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-usb-net - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-usb-net-cdc-ether - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-usb-net-qmi-wwan - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-usb-net-rndis - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-usb-serial - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-usb-serial-option - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-usb-serial-qualcomm - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-usb-serial-wwan - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-usb-storage - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-usb-uhci - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-usb-wdm - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-usb3 - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-veth - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
kmod-wdt-orion - 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0
# opkg list-installed|grep kernel
kernel - 4.4.87-1-cb5e816fa6b1a6b5342df69755869d71-2
# pwd
/boot
# ls -la
drwxr-xr-x    1 root     root           118 Sep 17 21:24 .
drwxrwxr-x    1 root     root           156 Sep  5 19:40 ..
-rw-r--r--    1 root     root         17306 Sep 13 18:49 dtb
lrwxrwxrwx    1 root     root            50 Sep 17 21:24 zImage -> zImage-4.4.87-1-cb5e816fa6b1a6b5342df69755869d71-2
-rwxr-xr-x    1 root     root       2557944 Sep 13 18:52 zImage-4.4.87-1-cb5e816fa6b1a6b5342df69755869d71-2
# 

Anyways, I fixed this problem by modifying the repo descriptors to stick to 3.7.5.

Ugh have you been using opkg for that? It shouldn’t allow you to do that and updater should fail miserably because every kmod explicitly depends on it’s kernel version. For example:

Package: kmod-ath10k
Version: 4.4.79+2016-10-08-9-967673b9d511e4292e3bcb76c9e064bc-0
Depends: kernel (=4.4.79-1-967673b9d511e4292e3bcb76c9e064bc-0), kmod-ath

Did you used dependency override? Or are those packages from Lede? I am confused.

Of course I used opkg. I try to stick to the stock utilities and not reinvent the wheel.

I fixed it by pointing the distfeeds.conf to 3.7.5 (see the snippet below), removing the newer kernel module, force removing the kernel, then reinstalling the proper version.

root:/etc/opkg# ls -la
drwxr-xr-x    1 root     root           144 Sep 17 22:03 .
drwxr-xr-x    1 root     root          1702 Sep  1 16:43 ..
-rw-r--r--    1 root     root           103 Jun 21 01:04 customfeeds.conf
lrwxrwxrwx    1 root     root            20 Sep 17 22:03 distfeeds.conf -> distfeeds.conf.3.7.5
-rw-r--r--    1 root     root           760 Sep 17 21:53 distfeeds.conf.3.7.5
-rw-r--r--    1 root     root           634 Sep 17 21:53 distfeeds.conf.bak
drwxr-xr-x    1 root     root            64 May 10  1968 keys
root:/etc/opkg# head -1 distfeeds.conf.3.7.5 distfeeds.conf.bak 
==> distfeeds.conf.3.7.5 <==
src/gz omnia_base https://repo.turris.cz/archive/omnia/3.7.5/packages//base

==> distfeeds.conf.bak <==
src/gz omnia_base https://repo.turris.cz/omnia/packages//base
root:/etc/opkg# 

I copied the original distfeeds.conf to .bak, but otherwise did not modify it.

Even if that wheel is pentagon and new version is hexagon? Because I have never seen round wheel in packages management.

Opkg is in lot of cases flawed and for example updater would never allowed you to do such mistake.

Heh, and on the other hand, the manual kernel downgrade triggered the usual notification mail about the necessary reboot:

"##### Restart is needed #####
The OS kernel has been updated to version 4.4.79+0-1-967673b9d511e4292e3bcb76c9e064bc-0. Please reboot the device.

The device will be restarted automatically on Wednesday, September 20 at 03:30 AM."

What do you mean by that? It’s just a single line in postinst script that was moved to more informed updater at 3.8 version (some users were rebooting router even if it wasn’t yet fully updater simply because this message was generated after kernel was installed but not after whole update including all kernel mods was installed). That has nothing to do with you installing packages by hand or not.

No matter what you think of opkg, it did pull in the newer kernel and removed the current one. Next time I’m going to be more observant and catch the details, at least now I can see the pattern here.

So please on top of inconsistent dependencies add to your list of potential problems also no files collision checking, no distinction between explicit and implicit packages, somewhat random order of updated packages and no transaction recovery. And those are just the major problems I put from top of my mind now. I am not saying that opkg is bad, it’s good for what it was used for in OpenWRT/Lede but it’s no way dnf or apt.

We tried to use opkg for updates for more than two years so we know those problems. And if you think that you can handle that better than us then I am saying good luck but we don’t provide any support for it.

1 Like

Hang on mate, I’m afraid you didn’t get my point. I did not want to update with opkg, just simply install a couple of packages (in fact, to try the cake packet scheduler), but it went wrong, because meanwhile you bumped the OS version to 3.8 and opkg seemingly cannot handle package version dependencies properly.

Ok in that case updater should fix it easily. It does full dependency check and reinstalls packages and if it founds out that some packages in system are not compatible it reinstalls them to correct versions.