Reboot at night - why?

yes, just for thissimilar case I have set up syslog-ng to save logs on usb stick.

My NAS is up for 54 days, so no.

But thanks - I didn’t think of the hint, I’ll set up nut to log UPS status. my UPS is already connected to the router since it’s the only device 24/7 on

Wasn’t there some internet outage? My Omnia’s kernel regularly segfaults when there is WAN outage (VDSL). I’m working on it with support, it looks like some memory corruption caused by the mvneta driver.

Just a note … I have like two unexpected reboot, due incoming HDO signal (nightly el.power tarif). So no real el.power outage, that hdo was somehow so strong, it shutted down also my main computer (while others were stil up and running).

Internet outage shouldn’t cause router reboot.

this happened at 03:33, while after packages update router reboots as 03:30. While no packages were updatet, maybe the issue 21 can cause router reboot even without update happening.

Yes, it shouldn’t, but at least in my case it does.

My router also affected by issue 21 (with update approvals still configured) does not reboot spuriously.

By default, if you have the router set to automatically install updates, it will reboot by itself in a few days, unless you reboot it yourself, if the update requires restarting (usually when the kernel is updated). Mine’s set, for instance, to reboot in 3 days at 3:30am.
I know there was an update to 5.1.4 a few days ago which required a reboot; might’ve been that.

Note, I’ve no clue if you can configure that in reForis, but in Forius you definitely can.

Personally, I set updates to “approval needed” and have the router send me an email notifications; that way, I’m never caught by surprise.

I set updates to “approve manually” (can do in both foris and reforis).

However no updates were done last week, but the reboot happened about the time it’s usually done after kernel update.

well, I get disconnected by ISP after 24 hours, so teoretically it could cause that. But it won’t explain why it happened once in half year I own omnia…

As far as we got with the kernel crash investigations, it seems that it might be caused by the ethernet driver unable to allocate memory… At first,this seemed odd to me because there were usually 100-200 MB free RAM,but then we found out that the memory is pretty fragmented and the driver asks for a larger chunk than is available. You can watch the number of free chunks of memory in /proc/buddyinfo - it contains a list of numbers denoting the number of free blocks of sizes starting with 4 kiB and doubling with each entry… Given this way of reasoning, it could explain why it only happened now - it’s just pretty much dependent on how long has the router been running and how much did its free memory get fragmented.

that could help - do you know anything available as opkg that monitors that info?
internet search colletcl that seems not to be available

I’d go just with a simple background script that saves the data to some persistent storage. E.g. something like:

while true; do cat /proc/buddyinfo >> /srv/buddyinfo; sleep 5; done

Be aware that /srv should be some other memory than the internal drive (mmcblk0). It can even be a flash drive in USB.

happened today again. Happened in the afternoon, about the time ISP was going to disconnect me after 24 hours (we’ve had power outage 3 weeks ago, power went on ~13:00
That indicated the problem appears after disconnect.

unfortunately, USB flash decided to fail yesterday and seems to be empty (dd from flash shows only zeroes).

That would really point towards a similar problem I have with the crashes on wan outage… @mbehun ?

i set up buddyinfo to be posted to syslog.
Since I save syslog messages to permanent storage, it could help.

root@turris:/etc/cron.d# cat /etc/cron.d/buddyinfo 
*/5	*	*	*	*	root	logger -t buddyinfo < /proc/buddyinfo
1 Like

ok, I had unexpected reboot 2 days ago. This is what I have in log:

Feb 17 02:40:01 turris crond[3643]: (root) CMD (logger -t buddyinfo < /proc/buddyinfo )
Feb 17 03:40:01 turris buddyinfo: Node 0, zone   Normal    317   2855   2241    574     22      0      0      0      0      0      0
Feb 17 03:40:01 turris buddyinfo: Node 0, zone  HighMem    137     81     24      5      3      2      2      0      0      1      1
Feb 17 02:40:01 turris crond[3647]: (root) CMD (/usr/bin/notifier)
Feb 17 02:40:01 turris crond[3641]: (root) CMDOUT (There is no message to send.)
Feb 17 03:42:29 turris kernel: [    0.000000] Booting Linux on physical CPU 0x0

Are you on HBT or HBS?

HBS. I approved updates, so at night I’ll get another reboot, wanted for now.

I have buddyinfo logged since Jan 4, numbers from before didn’t show any extreme numbers

I got an unexpected reboot when updating from 5.1.9 RC1 to RC3 (in HBT). This might be related. But I haven’t figured out what was the reason for the reboot (last lines in log were from wpad, which is a package that got updated).

It seems to be more common than I thought. Luckily the ISP disconnects after 24 hours and luckily TOS reboots ad 03:30, so it does not happen during day.
This is the log from updating kernel (causes reboot next night) and reboots:

Jan 14 19:16:37 turris updater[26977]: src/lib/logging.c:201 (log_subproc_open): Running postinst of kernel
Jan 15 03:30:31 turris kernel: [    0.000000] Linux version 4.14.212 (packaging@turris.cz) (gcc version 7.5.0 (OpenWrt GCC 7.5.0 e290024)) #0 SMP Tue Jan 12 00:24:34 2021
Jan 17 03:31:20 turris kernel: [    0.000000] Linux version 4.14.212 (packaging@turris.cz) (gcc version 7.5.0 (OpenWrt GCC 7.5.0 e290024)) #0 SMP Tue Jan 12 00:24:34 2021
Jan 19 19:48:21 turris updater[3085]: src/lib/logging.c:201 (log_subproc_open): Running postinst of kernel
Jan 20 03:30:30 turris kernel: [    0.000000] Linux version 4.14.214 (packaging@turris.cz) (gcc version 7.5.0 (OpenWrt GCC 7.5.0 c9388fa)) #0 SMP Thu Jan 14 09:13:08 2021
Jan 22 19:08:31 turris updater[2205]: src/lib/logging.c:201 (log_subproc_open): Running postinst of kernel
Jan 23 03:30:31 turris kernel: [    0.000000] Linux version 4.14.215 (packaging@turris.cz) (gcc version 7.5.0 (OpenWrt GCC 7.5.0 c21d59d)) #0 SMP Wed Jan 20 10:47:26 2021
Jan 28 17:46:33 turris updater[24349]: src/lib/logging.c:201 (log_subproc_open): Running postinst of kernel
Jan 29 03:30:32 turris kernel: [    0.000000] Linux version 4.14.216 (packaging@turris.cz) (gcc version 7.5.0 (OpenWrt GCC 7.5.0 11f4918)) #0 SMP Tue Jan 26 07:47:52 2021
Jan 31 03:31:23 turris kernel: [    0.000000] Linux version 4.14.216 (packaging@turris.cz) (gcc version 7.5.0 (OpenWrt GCC 7.5.0 11f4918)) #0 SMP Tue Jan 26 07:47:52 2021
Feb  7 03:33:03 turris kernel: [    0.000000] Linux version 4.14.216 (packaging@turris.cz) (gcc version 7.5.0 (OpenWrt GCC 7.5.0 11f4918)) #0 SMP Tue Jan 26 07:47:52 2021
Feb 12 03:34:23 turris kernel: [    0.000000] Linux version 4.14.216 (packaging@turris.cz) (gcc version 7.5.0 (OpenWrt GCC 7.5.0 11f4918)) #0 SMP Tue Jan 26 07:47:52 2021
Feb 14 03:41:29 turris kernel: [    0.000000] Linux version 4.14.216 (packaging@turris.cz) (gcc version 7.5.0 (OpenWrt GCC 7.5.0 11f4918)) #0 SMP Tue Jan 26 07:47:52 2021
Feb 17 03:42:29 turris kernel: [    0.000000] Linux version 4.14.216 (packaging@turris.cz) (gcc version 7.5.0 (OpenWrt GCC 7.5.0 11f4918)) #0 SMP Tue Jan 26 07:47:52 2021
Feb 19 08:09:34 turris updater[21399]: src/lib/logging.c:201 (log_subproc_open): Running postinst of kernel
Feb 20 03:30:31 turris kernel: [    0.000000] Linux version 4.14.221 (packaging@turris.cz) (gcc version 7.5.0 (OpenWrt GCC 7.5.0 6bf5bfc)) #0 SMP Thu Feb 18 03:57:20 2021