TOS5 - HDD disks re-mapped

Software SW help
1

Hi all,
since I upgraded Omnia to TOS5 I’m facing the following issue:

  • there is mSata disk discovered as /dev/sda and it works fine
  • then there are 2 SATA disks connected over USB3 and they are discovered as /dev/sdb and /dev/sdc after router starts
  • partitions on all 3 disks are BTRFS

Now the issue - the disks connected over USB3 suddently appear as /dev/sde and /dev/sdf and devices /dev/sdb and /dev/sdc no longer exist. Of course they are not mapped any longer.

I cannot see any obvious reason from the logs - some example entries when kernel “rediscovers” the disk again

2020-08-10T13:55:38+02:00 2020-08-10T13:55:38+02:00 omnia kern notice kernel[]: [89271.790698] sd 4:0:0:1: [sdf] 3907029168 512-byte      logical blocks: (2.00 TB/1.82 TiB)
2020-08-10T13:55:38+02:00 2020-08-10T13:55:38+02:00 omnia kern notice kernel[]: [89271.828415] sd 4:0:0:0: [sde] Write Protect is off
2020-08-10T13:55:38+02:00 2020-08-10T13:55:38+02:00 omnia kern debug kernel[]: [89271.833224] sd 4:0:0:0: [sde] Mode Sense: 47 00 10 08
2020-08-10T13:55:39+02:00 2020-08-10T11:55:39+02:00 omnia daemon err block[]: mounting /dev/sde1 (btrfs) as /srv/data failed (16) - Not a tty

Do you have any suggestion on what might trigger this behavior? I always lose access to the disks through mounted points and have to restart the router.

2

In general that is caused by USB subsytem reload or reset of bus and so on (I suspect). In short: disconnect of usb drive. You should look for kernel messages about usb.

First of all I would suggest you to use UUID instead of dev file to mount drives as those are stable and with that just calling block mount mounts them back.

Why drives are mapped as not their original files is because of them being mounted at the time of disconnect. Kernel can’t safelly reuse those files so it uses new ones.

3

I already mount the partitions through uuid. E.g. in /etc/config/fstab I have:

config mount
        option uuid '78a02baa-6ad2-4721-ac84-da25e3b3071b'
        option target '/srv/data'
        option fstype 'btrfs'
        option enabled '1'

The mount point is visible even after the disks are re-discovered (see sdf1 below):

NAME         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda            8:0    1 232.9G  0 disk
└─sda1         8:1    1 232.9G  0 part /srv
sdd            8:48   1   3.8G  0 disk
└─sdd1         8:49   1   3.8G  0 part
sde            8:64   0   2.7T  0 disk
├─sde1         8:65   0   1.8T  0 part
└─sde2         8:66   0 931.5G  0 part
sdf            8:80   0   1.8T  0 disk
└─sdf1         8:81   0   1.8T  0 part /srv/data
mtdblock0     31:0    0     1M  0 disk
mtdblock1     31:1    0     7M  0 disk
mmcblk0      179:0    0   7.3G  0 disk
└─mmcblk0p1  179:1    0   7.3G  0 part /
mmcblk0boot0 179:8    0     4M  1 disk
mmcblk0boot1 179:16   0     4M  1 disk

The problem is that the data on the partition is not accessible:

root@omnia:/srv/data# ls
ls: reading directory '.': I/O error

root@omnia:/srv/data# mount -t btrfs /dev/sdf1 /mnt/sdf1
mount: /mnt/sdf1: wrong fs type, bad option, bad superblock on /dev/sdf1, missing codepage or helper program, or other error.

root@omnia:/srv/data# btrfs filesystem df /srv/data
Data, RAID1: total=448.00GiB, used=447.92GiB
System, RAID1: total=32.00MiB, used=96.00KiB
Metadata, RAID1: total=2.00GiB, used=891.12MiB
GlobalReserve, single: total=498.73MiB, used=0.00B

root@omnia:/srv/data# btrfs filesystem usage /srv/data
ERROR: cannot look up chunk tree info: I/O error

but when I reboot the router the disks work fine again

root@omnia:~# lsblk
NAME         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda            8:0    1 232.9G  0 disk
└─sda1         8:1    1 232.9G  0 part /srv
sdb            8:16   0   2.7T  0 disk
├─sdb1         8:17   0   1.8T  0 part
└─sdb2         8:18   0 931.5G  0 part
sdc            8:32   0   1.8T  0 disk
└─sdc1         8:33   0   1.8T  0 part /srv/data
sdd            8:48   1   3.8G  0 disk
└─sdd1         8:49   1   3.8G  0 part
mtdblock0     31:0    0     1M  0 disk
mtdblock1     31:1    0     7M  0 disk
mmcblk0      179:0    0   7.3G  0 disk
└─mmcblk0p1  179:1    0   7.3G  0 part /
mmcblk0boot0 179:8    0     4M  1 disk
mmcblk0boot1 179:16   0     4M  1 disk
root@omnia:~# cd /srv/data
root@omnia:/srv/data# ls
backup  computers  data  media  zdroje

So it’s not only about rescanning / remapping the disks but about the data not being accessible any more.

4

In that case it looks more like broken USB connection. Try to manually unplug and plug it back.

Is it connected trough some USB hub? Does that HUB provides enough power? Are those drives HHDs or SSDs? What are their nominal power consumption?

You should really look for errors on USB bus in kernel log. This looks like problem with USB connection rather than with software. Is it possible that they draw too much power or are they connected together with some other device that could potentially behave badly?

5

Power is not a problem - both disks in external HDD 2 Bay HDD Case (see https://fantec.com/product/fantec-qb-x2us3r/) with external power supply. The disks are SATA (1x WD RED and 1 x WD Green) - from the external case perspective configured as JBOD but one of the BTRFS partitions is configured as Raid1 over these 2 disks (as one disk is larger the rest of the space is another BTRFS partition with single disk)

Below is the extract from the log after restart filtering out ‘usb’. There is problem with UAS which was reported earlier in Rozbitý USB3 UAS po update (restartu) na 3.11.3 but otherwise I cannot see anything else from the logs. There are no USB related events in the log when the disks are “remapped”. When I disconnect the case then there are no messages in the log either. When I connect it back I can the messages that new USB Mass Storage device detected and that the UAS is not available.

The external case worked fine when I was on 3.x version - but at that time I had a single disk inside so there was a change there too. I can try to remove the second disk as the data is mirrored to rule it out.

2020-08-13T13:33:12+02:00 2020-08-13T13:33:12+02:00 omnia kern info kernel[]: [    3.120600] usbcore: registered new interface driver usbfs
2020-08-13T13:33:12+02:00 2020-08-13T13:33:12+02:00 omnia kern info kernel[]: [    3.120626] usbcore: registered new interface driver hub
2020-08-13T13:33:12+02:00 2020-08-13T13:33:12+02:00 omnia kern info kernel[]: [    3.120652] usbcore: registered new device driver usb
2020-08-13T13:33:12+02:00 2020-08-13T13:33:12+02:00 omnia kern info kernel[]: [    4.478984] orion-ehci f1058000.usb: EHCI Host Controller
2020-08-13T13:33:12+02:00 2020-08-13T13:33:12+02:00 omnia kern info kernel[]: [    4.484412] orion-ehci f1058000.usb: new USB bus registered, assigned bus number 1
2020-08-13T13:33:12+02:00 2020-08-13T13:33:12+02:00 omnia kern info kernel[]: [    4.492048] orion-ehci f1058000.usb: irq 40, io mem 0xf1058000
2020-08-13T13:33:12+02:00 2020-08-13T13:33:12+02:00 omnia kern info kernel[]: [    4.521334] orion-ehci f1058000.usb: USB 2.0 started, EHCI 1.00
2020-08-13T13:33:12+02:00 2020-08-13T13:33:12+02:00 omnia kern info kernel[]: [    4.535646] xhci-hcd f10f0000.usb3: xHCI Host Controller
2020-08-13T13:33:12+02:00 2020-08-13T13:33:12+02:00 omnia kern info kernel[]: [    4.540978] xhci-hcd f10f0000.usb3: new USB bus registered, assigned bus number 2
2020-08-13T13:33:12+02:00 2020-08-13T13:33:12+02:00 omnia kern info kernel[]: [    4.548548] xhci-hcd f10f0000.usb3: hcc params 0x0a000990 hci version 0x100 quirks 0x0000000000010010
2020-08-13T13:33:12+02:00 2020-08-13T13:33:12+02:00 omnia kern info kernel[]: [    4.557817] xhci-hcd f10f0000.usb3: irq 46, io mem 0xf10f0000
2020-08-13T13:33:12+02:00 2020-08-13T13:33:12+02:00 omnia kern info kernel[]: [    4.576942] xhci-hcd f10f0000.usb3: xHCI Host Controller
2020-08-13T13:33:12+02:00 2020-08-13T13:33:12+02:00 omnia kern info kernel[]: [    4.582280] xhci-hcd f10f0000.usb3: new USB bus registered, assigned bus number 3
2020-08-13T13:33:12+02:00 2020-08-13T13:33:12+02:00 omnia kern info kernel[]: [    4.589789] xhci-hcd f10f0000.usb3: Host supports USB 3.0  SuperSpeed
2020-08-13T13:33:12+02:00 2020-08-13T13:33:12+02:00 omnia kern info kernel[]: [    4.596279] usb usb3: We don't know the algorithms for LPM for this host, disabling LPM.
2020-08-13T13:33:12+02:00 2020-08-13T13:33:12+02:00 omnia kern info kernel[]: [    4.612469] xhci-hcd f10f8000.usb3: xHCI Host Controller
2020-08-13T13:33:12+02:00 2020-08-13T13:33:12+02:00 omnia kern info kernel[]: [    4.617798] xhci-hcd f10f8000.usb3: new USB bus registered, assigned bus number 4
2020-08-13T13:33:12+02:00 2020-08-13T13:33:12+02:00 omnia kern info kernel[]: [    4.625356] xhci-hcd f10f8000.usb3: hcc params 0x0a000990 hci version 0x100 quirks 0x0000000000010010
2020-08-13T13:33:12+02:00 2020-08-13T13:33:12+02:00 omnia kern info kernel[]: [    4.634620] xhci-hcd f10f8000.usb3: irq 47, io mem 0xf10f8000
2020-08-13T13:33:12+02:00 2020-08-13T13:33:12+02:00 omnia kern info kernel[]: [    4.648433] xhci-hcd f10f8000.usb3: xHCI Host Controller
2020-08-13T13:33:12+02:00 2020-08-13T13:33:12+02:00 omnia kern info kernel[]: [    4.653772] xhci-hcd f10f8000.usb3: new USB bus registered, assigned bus number 5
2020-08-13T13:33:12+02:00 2020-08-13T13:33:12+02:00 omnia kern info kernel[]: [    4.661279] xhci-hcd f10f8000.usb3: Host supports USB 3.0  SuperSpeed
2020-08-13T13:33:12+02:00 2020-08-13T13:33:12+02:00 omnia kern info kernel[]: [    4.667767] usb usb5: We don't know the algorithms for LPM for this host, disabling LPM.
2020-08-13T13:33:12+02:00 2020-08-13T13:33:12+02:00 omnia kern info kernel[]: [    4.684014] usbcore: registered new interface driver usb-storage
2020-08-13T13:33:12+02:00 2020-08-13T13:33:12+02:00 omnia kern info kernel[]: [    5.014151] usb 3-1: new SuperSpeed USB device number 2 using xhci-hcd
2020-08-13T13:33:12+02:00 2020-08-13T13:33:12+02:00 omnia kern warning kernel[]: [    5.042464] usb 3-1: USB controller f10f0000.usb3 does not support streams, which are required by the UAS driver.
2020-08-13T13:33:12+02:00 2020-08-13T13:33:12+02:00 omnia kern warning kernel[]: [    5.052765] usb 3-1: Please try an other USB controller if you wish to use UAS.
2020-08-13T13:33:12+02:00 2020-08-13T13:33:12+02:00 omnia kern info kernel[]: [    5.060091] usb-storage 3-1:1.0: USB Mass Storage device detected
2020-08-13T13:33:12+02:00 2020-08-13T13:33:12+02:00 omnia kern info kernel[]: [    5.066286] usb-storage 3-1:1.0: Quirks match for vid 152d pid 9561: 4000000
2020-08-13T13:33:12+02:00 2020-08-13T13:33:12+02:00 omnia kern info kernel[]: [    5.073397] scsi host2: usb-storage 3-1:1.0
2020-08-13T13:33:12+02:00 2020-08-13T13:33:12+02:00 omnia kern info kernel[]: [    8.952864] usbcore: registered new interface driver uas
2020-08-13T13:33:12+02:00 2020-08-13T13:33:12+02:00 omnia kern info kernel[]: [   15.535682] usbcore: registered new interface driver ums-alauda
2020-08-13T13:33:12+02:00 2020-08-13T13:33:12+02:00 omnia kern info kernel[]: [   15.542004] usbcore: registered new interface driver ums-cypress
2020-08-13T13:33:12+02:00 2020-08-13T13:33:12+02:00 omnia kern info kernel[]: [   15.548418] usbcore: registered new interface driver ums-datafab
2020-08-13T13:33:12+02:00 2020-08-13T13:33:12+02:00 omnia kern info kernel[]: [   15.554837] usbcore: registered new interface driver ums-freecom
2020-08-13T13:33:12+02:00 2020-08-13T13:33:12+02:00 omnia kern info kernel[]: [   15.561269] usbcore: registered new interface driver ums-isd200
2020-08-13T13:33:12+02:00 2020-08-13T13:33:12+02:00 omnia kern info kernel[]: [   15.567731] usbcore: registered new interface driver ums-jumpshot
2020-08-13T13:33:12+02:00 2020-08-13T13:33:12+02:00 omnia kern info kernel[]: [   15.574258] usbcore: registered new interface driver ums-karma
2020-08-13T13:33:12+02:00 2020-08-13T13:33:12+02:00 omnia kern info kernel[]: [   15.580569] usbcore: registered new interface driver ums-sddr09
2020-08-13T13:33:12+02:00 2020-08-13T13:33:12+02:00 omnia kern info kernel[]: [   15.586988] usbcore: registered new interface driver ums-sddr55
2020-08-13T13:33:12+02:00 2020-08-13T13:33:12+02:00 omnia kern info kernel[]: [   15.593361] usbcore: registered new interface driver ums-usbat
2020-08-13T13:33:12+02:00 2020-08-13T13:33:12+02:00 omnia kern info kernel[]: [   15.600930] usbcore: registered new interface driver usbserial
2020-08-13T13:33:12+02:00 2020-08-13T13:33:12+02:00 omnia kern info kernel[]: [   15.606822] usbcore: registered new interface driver usbserial_generic
2020-08-13T13:33:12+02:00 2020-08-13T13:33:12+02:00 omnia kern info kernel[]: [   15.613409] usbserial: USB Serial support registered for generic
2020-08-13T13:33:12+02:00 2020-08-13T13:33:12+02:00 omnia kern info kernel[]: [   19.556793] usbcore: registered new interface driver qcserial
2020-08-13T13:33:12+02:00 2020-08-13T13:33:12+02:00 omnia kern info kernel[]: [   19.562629] usbserial: USB Serial support registered for Qualcomm USB modem
6

Try it although I do not give it much…

Also you should do grep for usb not after reboot but after problem occurs (I understood it as that it works for some time before it breaks).

7

I searched the logs (I’m keeping all the logs for some time on the internal disk which works fine) and as I mentioned above there are not usb related log entries when the disks are “remapped”

8

based on some observations it looks like that it might be related to hd-idle for one of the disks. I had the hd-idle configured for both disks in the external enclosure. I turned it off on one disks and I didn’t experience of remapping the disks, then I swapped them and experienced it then back and again no issue so far. It’s difficult to judge if it’s because the other disk didn’t get to “sleep”.

This is just a theory and difficult to debug as there isn’t anything in the logs. Any idea on how to prove / disprove if the hd-idle might be the root cause?

9

It could be a kernel bug considering this particular kconfig flag https://cateee.net/lkddb/web-lkddb/USB_DEFAULT_PERSIST.html

HD idle might amount to USB suspend and whilst with the kconfig flag is set the kernel supposedly should not re-enumerate /dev, causing any mounted filesystems to be lost.