Debugging unscheduled reboots

[s11erm]

My router keeps rebooting spontaneously. I know this because I get multiple small episodes of unscheduled downtime and when I SSH in to the router, I can see a reboot has happened using the uptime command. This happens a couple of times per day.

I’m running hbt (TurrisOS 6.0.1).

I’ve set up a external rsyslog instance on a separate device to collect the logging information (as the in-memory logging information gets cleared on reboot). However, I can’t find any useful information in the externally stored logs, in the run up to the problem.

What’s the best way to investigate this further? I’m not sure if the standard logging uci options in /etc/config/system do anything on a Turris. There is excellent documentation for /etc/config/system options on the OpenWrt but I couldn’t easily find comprehensive documentation on Turris-specific logging (surprisingly). It seems that TurrisOS uses syslog-ng and the configuration for this is not currently integrated into uci.

[peci1]

Is it a MOX? If yes, this is a known problem. The best thing you can do to debug unscheduled reboots is connecting a serial console and watching what the system writes to kernel log.

[s11erm]

No it’s an Omnia.

The unscheduled reboots happen intermittently and unpredictably. I suppose I could attach a Pi to serial and log the output but network logging is easier for me to implement as I already have the facility set up.

Is it possible to up the syslog output level?

[peci1]

Typically, mounted filesystems last longer than network-attached devices. So I’d connect a USB stick and log onto it (if you don’t already have an SSD inside). However, if it’s some kernel panic or so, my experience is that there’s never enough time to log it anywhere but the serial console.

[s11erm]

OK, so I have UART serial access and a Pi plugged in. What’s the best way to save all serial output onto the Pi without having to manually monitor a TTY? (I’m connected to the Pi over SSH which will disconnect when the router fails)

I’ve tried stty 115200; nohup cat /dev/ttyUSB0 > serial.out & but this doesn’t seem to provide reliable and continuous output.

Can I turn up the serial logging so it produces debug-level output?

[peci1]

I’m not an expert on this low-level stuff, but you can try connecting to the console from screen. This should survive the network downtime. It is also possible to save the whole buffer of screen to a file (just google for how to do that).