System hangs on any filesystem actions in command line

Omnia hardware Omnia HW problems
1

Hi all! I am on TurrisOS 5.3.5

Last week I encountered a problem, don’t know exactly when it appeared because router just works, but when I came to router via SSH and tried to edit or copy any files it just hangs.
Trying to SSHFS/SCP any files (I tried to backup any configurations) just hangs sshd server to death until hard reboot.

Little exampe

root@cerberus:~# ls -la /etc/config
drwxr-xr-x    1 root     root           654 Apr 29 17:46 .
drwxr-xr-x    1 root     root          2282 May  4 21:03 ..
-rw-------    1 root     root            98 Jan 10  2021 acme
-rw-------    1 root     root           435 Oct 20  2021 adblock
-rw-------    1 root     root          1038 Nov  1  2021 ahcpd
-rw-------    1 root     root           598 Nov  1  2021 backups
-rw-------    1 root     root           897 Nov  1  2021 bcp38
-rwxr-xr-x    1 root     root           110 Nov 23 16:52 cronie
-rw-------    1 root     root          2623 Apr 29 17:46 dhcp
Timeout, server 172.28.0.250 not responding.

Tried in panic to export backup snapshot on to external SSD mounted to /srv

root@cerberus:~# schnapps list
    # | Type      | Size        | Date                      | Description
------+-----------+-------------+---------------------------+------------------------------------
    1 | single    |    10.93MiB | 2022-03-18 17:41:07 +0700 | Point of No Return (2022-03-18)
    4 | time      |    10.91MiB | 2022-04-03 01:05:01 +0700 | Snapshot created by cron
    5 | time      |    11.22MiB | 2022-04-10 01:05:01 +0700 | Snapshot created by cron
    6 | time      |    11.45MiB | 2022-04-17 01:05:02 +0700 | Snapshot created by cron
    7 | time      |    10.95MiB | 2022-04-24 01:05:01 +0700 | Snapshot created by cron
    8 | time      |    10.93MiB | 2022-05-01 01:05:01 +0700 | Snapshot created by cron

root@cerberus:~# schnapps export 8 /srv/
./
./rom/
./rom/note
./tmp/
./tmp/lock/
./tmp/lock/procd_lighttpd.lock
./tmp/lock/procd_dnsmasq.lock
./tmp/lock/procd_gpio_switch.lock
./tmp/lock/procd_ucitrack.lock
./tmp/lock/procd_haveged.lock
./tmp/lock/procd_rpcd.lock
./tmp/lock/procd_foris-controller.lock
Timeout, server 172.28.0.250 not responding.

So, what happened? Is that case when eMMC is dead? How to check it?
If eMMC is dead why any fs operations hangs on /tmp path? RAM dead/broken?

2

I’d look at logs, mainly /var/log/messages I think.

3

It’s not possible, I tried tail /var/log/messages but it hangs.
// dmesg also hangs

After posting this, one hour later I found file (probably partial or full export of schnapps export 8 /srv/)

root@cerberus:~# ls -la /srv
-rw-r--r--    1 root     root      39854080 May  5 11:22 omnia-medkit-cerberus-8.tar.gz

Also tried this

root@cerberus:~# btrfs subvolume snapshot -r / /dump
Create a readonly snapshot of '/' in '//dump'
root@cerberus:~# btrfs send /dump | btrfs receive /srv
At subvol /dump
At subvol dump

But when going to /srv/dump to access some files at least it finally hangs on ls

4 
root@cerberus:/srv# dmesg
[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 4.14.264 (packaging@turris.cz) (gcc version 7.5.0 (OpenWrt GCC 7.5.0 r11397+89-524cbcf6f5)) #0 SMP Tue Feb 8 00:42:25 2022
[    0.000000] CPU: ARMv7 Processor [414fc091] revision 1 (ARMv7), cr=10c5387d
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[    0.000000] OF: fdt: Machine model: Turris Omnia
[    0.000000] Memory policy: Data cache writealloc
[    0.000000] On node 0 totalpages: 524288
[    0.000000] free_area_init_node: node 0, pgdat c0b3c7c0, node_mem_map eedf9000
[    0.000000]   Normal zone: 1728 pages used for memmap
[    0.000000]   Normal zone: 0 pages reserved
[    0.000000]   Normal zone: 196608 pages, LIFO batch:31
[    0.000000]   HighMem zone: 327680 pages, LIFO batch:31
[    0.000000] random: get_random_bytes called from start_kernel+0x8c/0x4cc with crng_init=0
[    0.000000] percpu: Embedded 11 pages/cpu s15372 r8192 d21492 u45056
[    0.000000] pcpu-alloc: s15372 r8192 d21492 u45056 alloc=11*4096
[    0.000000] pcpu-alloc: [0] 0 [0] 1
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 522560
<-- hangs here
5

Best to write to Turris support to get help (see Bug reports must be reported to Technical support department or Gitlab)

1 Like
6

Accidentally, problem has been solved by turning off load from USB port. My cat pulled wire out of the port, I guess he knows something.
This is kinda strange, it was fine before.