TOS 4 - how to reset USB3 bus?

I’ve logged via ssh, send reboot command and now disk works again.
But this is not a first time I have this issue. It was there in TOS3, now is in TOS4. There is something wrong with driver on Omnia or on USB disk.

Also I don’t know, what exactly went wrong this night with Omnia. Was really slow at morning, internet connection very unstable. Have no clue yet.

Check what driver you disk uses (using lsusb -t). There are two which can be used - usb mass storage and uas. You might try the other one to see if it helps (if your disk support UAS, which should be somewhere in product manual or datasheet). You can probably force the driver using a hotplug script.

You should be able to reset the usb bus by sending unbind/bind command to the xhci driver like:

echo -n "<usb_controller_id>" | tee /sys/bus/pci/drivers/xhci_hcd/unbind
echo -n "<usb_controller_id>" | tee /sys/bus/pci/drivers/xhci_hcd/bind

Not sure how to get the id though as I do not see id with a lspci.

I’ve tried, but did not work :frowning:

root@turris:~# lsusb -t
/:  Bus 05.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/1p, 5000M
/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/1p, 480M
    |__ Port 1: Dev 2, If 0, Class=Vendor Specific Class, Driver=ftdi_sio, 12M
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/1p, 5000M
    |__ Port 1: Dev 2, If 0, Class=Mass Storage, Driver=usb-storage, 5000M
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/1p, 480M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=orion-ehci/1p, 480M

I have two WD Blue (EZRZ), 3,5" - 3TB disc in ICY BOX IB-RD3620SU3. I’m not sure about the UAS support as I’ve bought them in 2016.

Well, you may just try the UAS driver and see if it works :slight_smile: I’d suggest trying on TurrisOS 4.x because it has a newer kernel with some uas-targeted improvements.

1 Like

I’m on TOS4 and it does not work :frowning:

root@turris:~# dmesg |grep usb
[    5.070541] usb 3-1: USB controller f10f0000.usb3 does not support streams, which are required by the UAS driver.
[    5.080863] usb 3-1: Please try an other USB controller if you wish to use UAS.
[    5.088194] usb-storage 3-1:1.0: USB Mass Storage device detected

And it still happening. It was OK last week, but this week I had to restart router on Monday and currently it does not work again :frowning:

What’s next? TOS5? New USB3 HDD box? Return to special NAS BOX?

Try usbreset?

This package contains the small usbreset utility which can be used to send a USB port reset to a USB device - useful for debugging or to force re-detection of particular devices.

root@turris:~# usbreset
Usage:
  usbreset PPPP:VVVV - reset by product and vendor id
  usbreset BBB/DDD   - reset by bus and device number
  usbreset "Product" - reset by product name

Devices:
  Number 004/002  ID 0403:6015  
  Number 003/002  ID 152d:0569  ICY BOX IB-RD3620
root@turris:~# usbreset 003/002

and frozes there :frowning:

A little crazy idea - Turris routers have GPIO pins. You could add a GPIO-controlled relay/mosfet and solder it somehow to the USB port, e.g. killing one of the power lines and routing it through the relay/mosfet. If done properly, it could solve the problem. If something goes wrong, you risk the USB port (the better case) or the whole router (the worse case). But I’m no electrician, you should definitely consult this with someone with deep knowledge :slight_smile:

@turris-admin wouldn’t it be a nice idea for the next generation of Turris routers (if you decide to make a new one at some time). Having USB/mSATA/mPCIe ports switchable by GPIO would be a killer feature :slight_smile:

Or maybe a safer solution - you can try to take a short extension cable, extract the power line, and do the harakiri there. This would be way safer and you could test everything properly before going wild. But you’d have to take care about not introducing impedance or breaking the shielding, otherwise the line could drop to slower speeds.

The latest revision of Omnia is without GPIO.
And I don’t want to do any harakiri as working internet is important for my family.

The safest option so far: buy something like https://www.vova.com/cs/Usb-Hub-3-0-High-Speed-4-7-Port-Usb-3-0-Hub-Splitter-On-Off-Switch-With-Eu-Us-GSN1584342187846269482750732-g17013454-m18480510 and exchange the mechanical switch for a GPIO-operated relay.

That’s not exactly correct. Turris Omnia sold as 2019 and 2020 edition has GPIO again.
https://docs.turris.cz/hw/omnia/revisions/

and you can see it here https://www.turris.cz/en/omnia/specifications/

Sorry for sharing wrong info.

So this is a equivalent to physical unplug? This does not work. On TOS3 I was able reload usb modules and I was able to access HDD without restart. But this is not possible on TOS4 :frowning:

root@turris:~# lsmod |egrep -i "usb|xhci"
ums_usbat              20480  0 
usb_wwan               16384  1 qcserial
usbserial              24576  5 qcserial,usb_wwan,ftdi_sio
root@turris:~#

Did you try TOS5.x? If the issue is present there as well could the USB controller be malfunctioning? The log on this node reads

xhci-hcd f10f0000.usb3: xHCI Host Controller
xhci-hcd f10f0000.usb3: new USB bus registered, assigned bus number 2
xhci-hcd f10f0000.usb3: hcc params 0x0a000990 hci version 0x100 quirks 0x0000000000010010
xhci-hcd f10f0000.usb3: irq 46, io mem 0xf10f0000
xhci-hcd f10f0000.usb3: xHCI Host Controller
xhci-hcd f10f0000.usb3: new USB bus registered, assigned bus number 3
xhci-hcd f10f0000.usb3: Host supports USB 3.0  SuperSpeed
xhci-hcd f10f8000.usb3: xHCI Host Controller
xhci-hcd f10f8000.usb3: new USB bus registered, assigned bus number 4
xhci-hcd f10f8000.usb3: hcc params 0x0a000990 hci version 0x100 quirks 0x0000000000010010
xhci-hcd f10f8000.usb3: irq 47, io mem 0xf10f8000
xhci-hcd f10f8000.usb3: xHCI Host Controller
xhci-hcd f10f8000.usb3: new USB bus registered, assigned bus number 5
xhci-hcd f10f8000.usb3: Host supports USB 3.0  SuperSpeed

I didn’t. TOS5 is not stable yet. The worst thing is that this happen randomly. It starts resetting the USB in loop and later fails with:

May  6 07:07:03 turris kernel: [   99.578204] xhci-hcd f10f0000.usb3: Max number of devices this xHCI host supports is 32.
May  6 07:07:03 turris kernel: [   99.586326] usb usb3-port1: couldn't allocate usb_device

There is something wrong with the kernel.

Well,
it happen again. Could be this a reason?

[141205.044912] ath10k_pci 0000:02:00.0: SWBA overrun on vdev 0, skipped old beacon
[141205.147308] ath10k_pci 0000:02:00.0: SWBA overrun on vdev 0, skipped old beacon
[141205.249702] ath10k_pci 0000:02:00.0: SWBA overrun on vdev 0, skipped old beacon
[141205.352097] ath10k_pci 0000:02:00.0: SWBA overrun on vdev 0, skipped old beacon
[141205.454491] ath10k_pci 0000:02:00.0: SWBA overrun on vdev 0, skipped old beacon
[141205.556885] ath10k_pci 0000:02:00.0: SWBA overrun on vdev 0, skipped old beacon
[141205.659279] ath10k_pci 0000:02:00.0: SWBA overrun on vdev 0, skipped old beacon
[141205.761675] ath10k_pci 0000:02:00.0: SWBA overrun on vdev 0, skipped old beacon
[141205.864064] ath10k_pci 0000:02:00.0: SWBA overrun on vdev 0, skipped old beacon
[141205.966460] ath10k_pci 0000:02:00.0: SWBA overrun on vdev 0, skipped old beacon
[141206.068851] ath10k_pci 0000:02:00.0: SWBA overrun on vdev 0, skipped old beacon
[141206.171241] ath10k_pci 0000:02:00.0: SWBA overrun on vdev 0, skipped old beacon
[141206.273640] ath10k_pci 0000:02:00.0: SWBA overrun on vdev 0, skipped old beacon
[141206.376036] ath10k_pci 0000:02:00.0: SWBA overrun on vdev 0, skipped old beacon
[141206.478437] ath10k_pci 0000:02:00.0: SWBA overrun on vdev 0, skipped old beacon
[141206.580822] ath10k_pci 0000:02:00.0: SWBA overrun on vdev 0, skipped old beacon
[141206.683210] ath10k_pci 0000:02:00.0: SWBA overrun on vdev 0, skipped old beacon
[141206.785615] ath10k_pci 0000:02:00.0: SWBA overrun on vdev 0, skipped old beacon
[141206.888015] ath10k_pci 0000:02:00.0: SWBA overrun on vdev 0, skipped old beacon
[141206.990406] ath10k_pci 0000:02:00.0: SWBA overrun on vdev 0, skipped old beacon
[141207.092798] ath10k_pci 0000:02:00.0: SWBA overrun on vdev 0, skipped old beacon
[141207.195193] ath10k_pci 0000:02:00.0: SWBA overrun on vdev 0, skipped old beacon
[141207.297586] ath10k_pci 0000:02:00.0: SWBA overrun on vdev 0, skipped old beacon
[141207.399975] ath10k_pci 0000:02:00.0: SWBA overrun on vdev 0, skipped old beacon
[141207.502374] ath10k_pci 0000:02:00.0: SWBA overrun on vdev 0, skipped old beacon
[141207.604766] ath10k_pci 0000:02:00.0: SWBA overrun on vdev 0, skipped old beacon
[141207.707161] ath10k_pci 0000:02:00.0: SWBA overrun on vdev 0, skipped old beacon
[141207.809589] ath10k_pci 0000:02:00.0: SWBA overrun on vdev 0, skipped old beacon
[141207.911953] ath10k_pci 0000:02:00.0: SWBA overrun on vdev 0, skipped old beacon
[141208.008355] ath10k_pci 0000:02:00.0: Cannot communicate with firmware, attempting to fake crash and restart firmware.
[141208.019136] ath10k_pci 0000:02:00.0: firmware crashed! (uuid bddba077-b2c6-440f-9f7a-60b8ac38064e)
[141208.028227] ath10k_pci 0000:02:00.0: qca988x hw2.0 target 0x4100016c chip_id 0x043202ff sub 0000:0000
[141208.037567] ath10k_pci 0000:02:00.0: kconfig debug 0 debugfs 1 tracing 0 dfs 1 testmode 0
[141208.047003] ath10k_pci 0000:02:00.0: firmware ver 10.1-ct-8x-__fH-021-4fa9f30 api 2 features wmi-10.x,txstatus-noack,wmi-10.x-CT,ratemask-CT,regdump-CT,txrate-CT,flush-all-CT,pingpong-CT,ch-regs-CT,nop-CT,set-special-CT,get-temp-CT,tx-rc-CT,cust-stats-CT crc32 580c6146
[141208.071010] ath10k_pci 0000:02:00.0: board_file api 1 bmi_id N/A crc32 bebc7c08
[141208.078442] ath10k_pci 0000:02:00.0: htt-ver 2.2 wmi-op 2 htt-op 2 cal otp max-sta 128 raw 0 hwcrypto 1
[141208.089978] ath10k_pci 0000:02:00.0: firmware register dump:
[141208.095747] ath10k_pci 0000:02:00.0: [00]: 0x00940750 0x00400C00 0x00980000 0x009B6074
[141208.103784] ath10k_pci 0000:02:00.0: [04]: 0x009B60E4 0x009B6208 0x00941B6C 0x00941B20
[141208.111821] ath10k_pci 0000:02:00.0: [08]: 0x00941B00 0x009423A4 0x009422DC 0x009422C4
[141208.119862] ath10k_pci 0000:02:00.0: [12]: 0x00941B90 0x009423F4 0x009423D4 0x009423DC
[141208.127884] ath10k_pci 0000:02:00.0: [16]: 0x009423E4 0x009423EC 0x00942888 0x00942520
[141208.135925] ath10k_pci 0000:02:00.0: [20]: 0x009424FC 0x0094241C 0x00942540 0x00942498
[141208.143959] ath10k_pci 0000:02:00.0: [24]: 0x00942450 0x0094250C 0x009424D0 0x00942750
[141208.151992] ath10k_pci 0000:02:00.0: [28]: 0x00942798 0x00942844 0x009428C4 0x00942594
[141208.160028] ath10k_pci 0000:02:00.0: [32]: 0x0094258C 0x00942C28 0x00942F4C 0x00942F7C
[141208.168055] ath10k_pci 0000:02:00.0: [36]: 0x00942F90 0x00942FE4 0x00942FF8 0x00943040
[141208.176091] ath10k_pci 0000:02:00.0: [40]: 0x0094077C 0x00940790 0x00943084 0x00942F00
[141208.184132] ath10k_pci 0000:02:00.0: [44]: 0x009B6234 0x00942A80 0x00942D40 0x00942D64
[141208.192161] ath10k_pci 0000:02:00.0: [48]: 0x00942D78 0x00942D9C 0x00957E18 0x00957E28
[141208.200200] ath10k_pci 0000:02:00.0: [52]: 0x00957E20 0x00940788 0x009430EC 0x00000000
[141208.208237] ath10k_pci 0000:02:00.0: [56]: 0x00000000 0x00000000 0x00000000 0x00000000
[141208.216274] ath10k_pci 0000:02:00.0: Copy Engine register dump:
[141208.222314] ath10k_pci 0000:02:00.0: [00]: 0x00057400   2   2   3   3
[141208.228866] ath10k_pci 0000:02:00.0: [01]: 0x00057800  28  28 349 352
[141208.235426] ath10k_pci 0000:02:00.0: [02]: 0x00057c00  13  13   9  13
[141208.241979] ath10k_pci 0000:02:00.0: [03]: 0x00058000  30  30   0  30
[141208.248546] ath10k_pci 0000:02:00.0: [04]: 0x00058400 2367 2367 201 161
[141208.255267] ath10k_pci 0000:02:00.0: [05]: 0x00058800   3   3 386 387
[141208.261835] ath10k_pci 0000:02:00.0: [06]: 0x00058c00  29  29  29  29
[141208.268397] ath10k_pci 0000:02:00.0: [07]: 0x00059000   0   0   0   0
[141208.276995] ath10k_pci 0000:02:00.0: debug log header, dbuf: 0x4125fc  dropped: 0
[141208.285624] ath10k_pci 0000:02:00.0: [0] next: 0x4125e4 buf: 0x410a38 sz: 1500 len: 0 count: 0 free: 0
[141208.296052] ath10k_pci 0000:02:00.0: [1] next: 0x4125fc buf: 0x410448 sz: 1500 len: 0 count: 0 free: 0
[141208.329662] ath10k_pci 0000:02:00.0: SWBA overrun on vdev 0, skipped old beacon
[141208.337095] ath10k_pci 0000:02:00.0: SWBA overrun on vdev 0, skipped old beacon
[141208.344534] ath10k_pci 0000:02:00.0: SWBA overrun on vdev 0, skipped old beacon
[141208.351969] ath10k_pci 0000:02:00.0: SWBA overrun on vdev 0, skipped old beacon
[141208.359444] ath10k_pci 0000:02:00.0: failed to set preamble for vdev 0: -11
[141208.366676] ath10k_pci 0000:02:00.0: failed to send wmi nop: -108
[141208.373112] ath10k_pci 0000:02:00.0: failed to submit wmi echo: -108
[141208.379586] ath10k_pci 0000:02:00.0: failed to ping firmware: -108
[141208.385887] ath10k_pci 0000:02:00.0: failed to install key for vdev 0 peer 9c:da:3e:d4:b6:15: -108
[141208.394971] wlan0: failed to remove key (0, 9c:da:3e:d4:b6:15) from hardware (-108)
[141208.394978] ath10k_pci 0000:02:00.0: removing peer, cleanup-all, deleting: peer ea083400 vdev: 0 addr: 78:02:f8:f5:6f:0c 
[141208.413843] ath10k_pci 0000:02:00.0: removing peer, cleanup-all, deleting: peer c4372e00 vdev: 0 addr: 0c:7a:15:a3:8e:7d 
[141208.424933] ath10k_pci 0000:02:00.0: removing peer, cleanup-all, deleting: peer d9a0c200 vdev: 0 addr: 9c:da:3e:d4:b6:15 
[141208.436018] ath10k_pci 0000:02:00.0: removing peer, cleanup-all, deleting: peer e6b82c00 vdev: 0 addr: 40:4e:36:9b:a2:30 
[141208.447112] ath10k_pci 0000:02:00.0: removing peer, cleanup-all, deleting: peer ea082200 vdev: 0 addr: 38:78:62:e6:18:08 
[141208.458201] ath10k_pci 0000:02:00.0: removing peer, cleanup-all, deleting: peer e46d2000 vdev: 0 addr: 24:46:c8:4e:7d:90 
[141208.469288] ath10k_pci 0000:02:00.0: removing peer, cleanup-all, deleting: peer ed3aea00 vdev: 0 addr: 04:f0:21:32:3a:6b 
[141208.480442] ath10k_pci 0000:02:00.0: SWBA overrun on vdev 0, skipped old beacon
[141208.600269] ieee80211 phy0: Hardware restart was requested
[141209.629619] ath10k_pci 0000:02:00.0: 10.1 wmi init: vdevs: 16  peers: 127  tid: 256
[141209.646806] ath10k_pci 0000:02:00.0: wmi print 'P 128 V 8 T 410'
[141209.652958] ath10k_pci 0000:02:00.0: wmi print 'msdu-desc: 1424  sw-crypt: 0'
[141209.660231] ath10k_pci 0000:02:00.0: wmi print 'alloc rem: 26400 iram: 27140'
[141209.733512] ath10k_pci 0000:02:00.0: set-coverage-class, phyclk: 88  value: 0
[141210.628859] ath10k_pci 0000:02:00.0: dropping dbg buffer due to crash since read
[141210.744493] ath10k_pci 0000:02:00.0: dropping dbg buffer due to crash since read
[141210.753968] ath10k_pci 0000:02:00.0: device successfully recovered
[141210.761749] ath10k_pci 0000:02:00.0: dropping dbg buffer due to crash since read
[141211.628842] ath10k_pci 0000:02:00.0: dropping dbg buffer due to crash since read
[141214.628871] ath10k_pci 0000:02:00.0: dropping dbg buffer due to crash since read
[141215.628879] ath10k_pci 0000:02:00.0: dropping dbg buffer due to crash since read
[141216.628901] ath10k_pci 0000:02:00.0: dropping dbg buffer due to crash since read
[141228.629249] ath10k_pci 0000:02:00.0: dropping dbg buffer due to crash since read
[141230.629271] ath10k_pci 0000:02:00.0: dropping dbg buffer due to crash since read
[141274.629761] ath10k_pci 0000:02:00.0: dropping dbg buffer due to crash since read
[141503.631314] ath10k_pci 0000:02:00.0: dropping dbg buffer due to crash since read
[141506.631379] ath10k_pci 0000:02:00.0: dropping dbg buffer due to crash since read
[141507.631387] ath10k_pci 0000:02:00.0: dropping dbg buffer due to crash since read
[141663.634085] ath10k_pci 0000:02:00.0: dropping dbg buffer due to crash since read
[141695.634450] ath10k_pci 0000:02:00.0: dropping dbg buffer due to crash since read
[141741.635422] ath10k_pci 0000:02:00.0: dropping dbg buffer due to crash since read
[142112.639950] ath10k_pci 0000:02:00.0: dropping dbg buffer due to crash since read
[142563.646164] ath10k_pci 0000:02:00.0: dropping dbg buffer due to crash since read
[142932.648745] ath10k_pci 0000:02:00.0: dropping dbg buffer due to crash since read
[142970.649247] ath10k_pci 0000:02:00.0: dropping dbg buffer due to crash since read
[143779.657716] ath10k_pci 0000:02:00.0: dropping dbg buffer due to crash since read
[143783.657764] ath10k_pci 0000:02:00.0: dropping dbg buffer due to crash since read
[143922.659253] ath10k_pci 0000:02:00.0: dropping dbg buffer due to crash since read
[144032.661366] ath10k_pci 0000:02:00.0: dropping dbg buffer due to crash since read
[144719.659999] ath10k_pci 0000:02:00.0: dropping dbg buffer due to crash since read
[145161.664173] ath10k_pci 0000:02:00.0: dropping dbg buffer due to crash since read
[145462.666748] ath10k_pci 0000:02:00.0: dropping dbg buffer due to crash since read
[146218.602758] mv88e6085 f1072004.mdio-mii:10 lan4: Link is Down
[146218.608648] br-lan: port 5(lan4) entered disabled state
[146221.366718] mv88e6085 f1072004.mdio-mii:10 lan4: Link is Up - 10Mbps/Full - flow control rx/tx
[146221.375517] br-lan: port 5(lan4) entered blocking state
[146221.380861] br-lan: port 5(lan4) entered forwarding state
[146467.670864] ath10k_pci 0000:02:00.0: dropping dbg buffer due to crash since read
[147484.948464] mv88e6085 f1072004.mdio-mii:10 lan4: Link is Down
[147484.955683] br-lan: port 5(lan4) entered disabled state
[151221.683026] ath10k_pci 0000:02:00.0: dropping dbg buffer due to crash since read
[151223.683023] ath10k_pci 0000:02:00.0: dropping dbg buffer due to crash since read
[151526.681386] ath10k_pci 0000:02:00.0: dropping dbg buffer due to crash since read
[152096.680378] ath10k_pci 0000:02:00.0: dropping dbg buffer due to crash since read
[154667.682419] ath10k_pci 0000:02:00.0: dropping dbg buffer due to crash since read
[172894.799649] ath10k_pci 0000:02:00.0: dropping dbg buffer due to crash since read
[175706.831707] ath10k_pci 0000:02:00.0: dropping dbg buffer due to crash since read
[175707.831724] ath10k_pci 0000:02:00.0: dropping dbg buffer due to crash since read
[176497.838601] ath10k_pci 0000:02:00.0: dropping dbg buffer due to crash since read
[176498.838566] ath10k_pci 0000:02:00.0: dropping dbg buffer due to crash since read
[176499.838550] ath10k_pci 0000:02:00.0: dropping dbg buffer due to crash since read
[176543.469060] NFSD: Unable to create client record on stable storage: -110
[176836.838873] ath10k_pci 0000:02:00.0: dropping dbg buffer due to crash since read
[177155.837706] ath10k_pci 0000:02:00.0: dropping dbg buffer due to crash since read
[178690.834977] ath10k_pci 0000:02:00.0: dropping dbg buffer due to crash since read
[178694.834980] ath10k_pci 0000:02:00.0: dropping dbg buffer due to crash since read
[178875.835599] ath10k_pci 0000:02:00.0: dropping dbg buffer due to crash since read

Looks like too much errors for me.

Can you try running without the wifi card and observing if the USB bus problems occur again?