Turris Mox USB (from module A) not working any more (for mass storage) after TOS 6.0 upgrade

Mossroy · May 8, 2023, 3:58pm

@peci1 : @Hai mentioned in Turris Mox USB (from module A) not working any more (for mass storage) after TOS 6.0 upgrade - #2 by Hai the same issue with an externally powered disk.

Mossroy · May 12, 2023, 10:12am

I finally have some (good) news.

They managed to reproduce the issue, but only on Mox devices that have an old U-Boot version.

So they asked me which U-Boot version I have on my board. Mine is 2018.11. You can check yours with the following command:

root@turris:~# strings /dev/mtdblock1 | grep "U-Boot 20"
U-Boot 2018.11 (Dec 16 2018 - 12:50:19 +0000)
root@turris:~#

Their conclusion:

Dear Mossroy,

this means that you have old firmware, which is the cause of the issue.

Please install the nor-update package and run it, with commands:

opkg update
opkg install nor-update
nor-update

We will be sending you back your USB flash drive on monday.

I haven’t tested yet.

Do you have the same U-Boot version as I have?

@jada4p can you check you already have a more recent one? (which would explain why the issue was solved for you since TOS 6.2.2)

jada4p · May 12, 2023, 10:39am

Sure:

root@MOXjp:~# strings /dev/mtdblock1 | grep "U-Boot 20"
U-Boot 2021.10-rc3-00050-g7d3fea2c7f (Sep 07 2021 - 18:16:56 +0200)

Thus it is really newer one.

I can’t neither remember nor explain how it happened that there is newer U-boot version installed on my MOX… Sorry.

padam · May 12, 2023, 11:10am

@Mossroy I have the same version:

U-Boot 2018.11 (Dec 16 2018 - 12:50:19 +0000)

donny · May 12, 2023, 11:15am

Hello, the nor-update worked on my MOX machine even though the update seemed to fail at first attempt (unrequested run right after package installation). Also I haven’t found nor-update package, but found similar fix-nor-update that I installed.

# strings /dev/mtdblock1 | grep "U-Boot 20"
U-Boot 2018.11 (Dec 16 2018 - 12:50:19 +0000)

# opkg update
Downloading https://repo.turris.cz/hbk/mox/packages/core/Packages.gz
Updated list of available packages in /var/opkg-lists/turrisos_core
Downloading https://repo.turris.cz/hbk/mox/packages/core/Packages.sig
Signature check passed.
Downloading https://repo.turris.cz/hbk/mox/packages/base/Packages.gz
Updated list of available packages in /var/opkg-lists/turrisos_base
Downloading https://repo.turris.cz/hbk/mox/packages/base/Packages.sig
Signature check passed.
Downloading https://repo.turris.cz/hbk/mox/packages/cesnet/Packages.gz
Updated list of available packages in /var/opkg-lists/turrisos_cesnet
Downloading https://repo.turris.cz/hbk/mox/packages/cesnet/Packages.sig
Signature check passed.
Downloading https://repo.turris.cz/hbk/mox/packages/luci/Packages.gz
Updated list of available packages in /var/opkg-lists/turrisos_luci
Downloading https://repo.turris.cz/hbk/mox/packages/luci/Packages.sig
Signature check passed.
Downloading https://repo.turris.cz/hbk/mox/packages/node/Packages.gz
Updated list of available packages in /var/opkg-lists/turrisos_node
Downloading https://repo.turris.cz/hbk/mox/packages/node/Packages.sig
Signature check passed.
Downloading https://repo.turris.cz/hbk/mox/packages/packages/Packages.gz
Updated list of available packages in /var/opkg-lists/turrisos_packages
Downloading https://repo.turris.cz/hbk/mox/packages/packages/Packages.sig
Signature check passed.
Downloading https://repo.turris.cz/hbk/mox/packages/routing/Packages.gz
Updated list of available packages in /var/opkg-lists/turrisos_routing
Downloading https://repo.turris.cz/hbk/mox/packages/routing/Packages.sig
Signature check passed.
Downloading https://repo.turris.cz/hbk/mox/packages/telephony/Packages.gz
Updated list of available packages in /var/opkg-lists/turrisos_telephony
Downloading https://repo.turris.cz/hbk/mox/packages/telephony/Packages.sig
Signature check passed.
Downloading https://repo.turris.cz/hbk/mox/packages/turrispackages/Packages.gz
Updated list of available packages in /var/opkg-lists/turrisos_turrispackages
Downloading https://repo.turris.cz/hbk/mox/packages/turrispackages/Packages.sig
Signature check passed.

# opkg install fix-nor-update
Installing fix-nor-update (1-61) to root...
Downloading https://repo.turris.cz/hbk/mox/packages/turrispackages/fix-nor-update_1-61_aarch64_cortex-a53.ipk
Installing turris-mox-firmware (2.0-3) to root...
Downloading https://repo.turris.cz/hbk/mox/packages/turrispackages/turris-mox-firmware_2.0-3_aarch64_cortex-a53.ipk
Installing turris-nor-update (1.1.0-7) to root...
Downloading https://repo.turris.cz/hbk/mox/packages/turrispackages/turris-nor-update_1.1.0-7_aarch64_cortex-a53.ipk
Configuring turris-mox-firmware.
Configuring turris-nor-update.
Configuring fix-nor-update.
Mount is denied because the NTFS volume is already exclusively opened.
The volume may be already mounted, or another software may use it which
could be identified for example by the help of the 'fuser' command.
Verifying /dev/mtd0 against secure-firmware.bin ...
18a0c1bc58839cbbc7aef70db637d261 - /dev/mtd0
e12a263c63bd9860cff844763e81e56b - secure-firmware.bin
Failed
Unlocking /dev/mtd0 ...
Erasing /dev/mtd0 ...

Writing from secure-firmware.bin to /dev/mtd0 ...     
Verifying /dev/mtd1 against uboot ...
437cb920201336f556abd2b448248714 - /dev/mtd1
bd802eb0ec60fd89f983ac3cd4860fba - uboot
Failed
Cannot parse config file '/etc/fw_env.config': No such file or directory
Unlocking /dev/mtd1 ...
Erasing /dev/mtd1 ...

Writing from uboot to /dev/mtd1 ...     
Verifying /dev/mtd3 against rescue ...
e9b30ada843663c579829652847c7bb8 - /dev/mtd3
39f02d9fb36b4258bac37924bcdf570c - rescue
Failed
Unlocking /dev/mtd3 ...
Erasing /dev/mtd3 ...

Writing from rescue to /dev/mtd3 ...     
Can''t find anything to flash to 'dtb' partition

# reboot

...

 Warning: Changes performed using anything other than
 official web interface reForis are not covered by
 Turris support team unless instructed!
      ______                _         ____  _____
     /_  __/_  ____________(_)____   / __ \/ ___/
      / / / / / / ___/ ___/ / ___/  / / / /\__ 
     / / / /_/ / /  / /  / (__  )  / /_/ /___/ / 
    /_/  \__,_/_/  /_/  /_/____/   \____//____/  
                                             
 -----------------------------------------------------
 TurrisOS 6.3.2, Turris Mox
 -----------------------------------------------------
$ su
Password: 

# nor-update 
Mount is denied because the NTFS volume is already exclusively opened.
The volume may be already mounted, or another software may use it which
could be identified for example by the help of the 'fuser' command.
Verifying /dev/mtd0 against secure-firmware.bin ...
e12a263c63bd9860cff844763e81e56b - /dev/mtd0
e12a263c63bd9860cff844763e81e56b - secure-firmware.bin
Success
Verifying /dev/mtd1 against uboot ...
bd802eb0ec60fd89f983ac3cd4860fba - /dev/mtd1
bd802eb0ec60fd89f983ac3cd4860fba - uboot
Success
Verifying /dev/mtd3 against rescue ...
39f02d9fb36b4258bac37924bcdf570c - /dev/mtd3
39f02d9fb36b4258bac37924bcdf570c - rescue
Success
Can''t find anything to flash to 'dtb' partition

# strings /dev/mtdblock1 | grep "U-Boot 20"
U-Boot 2022.07 (Aug 15 2022 - 12:25:08 +0000)

# lsusb
Bus 003 Device 002: ID 0951:1656 Kingston DT Ultimate G2
Bus 002 Device 001: ID 1d6b:0002 Linux 5.15.107 xhci-hcd xHCI Host Controller
Bus 001 Device 001: ID 1d6b:0002 Linux 5.15.107 ehci_hcd EHCI Host Controller
Bus 003 Device 001: ID 1d6b:0003 Linux 5.15.107 xhci-hcd xHCI Host Controller

padam · May 12, 2023, 11:30am

For me also run right after package installation, second run was not necessary. After it was run strings showed new U-Boot installed:

root@turris:~# opkg install nor-update
Unknown package 'nor-update'.
Collected errors:
 * opkg_install_cmd: Cannot install package nor-update.
root@turris:~# opkg install fix-nor-update
Installing fix-nor-update (1-61) to root...
Downloading https://repo.turris.cz/hbs/mox/packages/turrispackages/fix-nor-update_1-61_aarch64_cortex-a53.ipk
Installing turris-mox-firmware (2.0-3) to root...
Downloading https://repo.turris.cz/hbs/mox/packages/turrispackages/turris-mox-firmware_2.0-3_aarch64_cortex-a53.ipk
Installing turris-nor-update (1.1.0-7) to root...
Downloading https://repo.turris.cz/hbs/mox/packages/turrispackages/turris-nor-update_1.1.0-7_aarch64_cortex-a53.ipk
Configuring turris-mox-firmware.
Configuring turris-nor-update.
Configuring fix-nor-update.
Verifying /dev/mtd0 against secure-firmware.bin ...
18a0c1bc58839cbbc7aef70db637d261 - /dev/mtd0
e12a263c63bd9860cff844763e81e56b - secure-firmware.bin
Failed
Unlocking /dev/mtd0 ...
Erasing /dev/mtd0 ...

Writing from secure-firmware.bin to /dev/mtd0 ...     
Verifying /dev/mtd1 against uboot ...
437cb920201336f556abd2b448248714 - /dev/mtd1
bd802eb0ec60fd89f983ac3cd4860fba - uboot
Failed
Unlocking /dev/mtd1 ...
Erasing /dev/mtd1 ...

Writing from uboot to /dev/mtd1 ...     
Verifying /dev/mtd3 against rescue ...
e9b30ada843663c579829652847c7bb8 - /dev/mtd3
39f02d9fb36b4258bac37924bcdf570c - rescue
Failed
Unlocking /dev/mtd3 ...
Erasing /dev/mtd3 ...

Writing from rescue to /dev/mtd3 ...     
Can't find anything to flash to 'dtb' partition

root@turris:~# strings /dev/mtdblock1 | grep "U-Boot 20"
U-Boot 2022.07 (Aug 15 2022 - 12:25:08 +0000)

I’ll upgrade my MOX to 6.x in the evening today.

jada4p · May 12, 2023, 12:43pm

It’s peculiar - I see from above that there is even more recent version of U-boat… I wonder if I should update to it… Could someone from team/support advice?

padam · May 12, 2023, 4:09pm

I can confirm that my USB3 HDD is working after the upgrade of U-Boot.

BusyBox v1.33.2 (2023-04-26 23:52:34 UTC) built-in shell (ash)

      ______                _         ____  _____
     /_  __/_  ____________(_)____   / __ \/ ___/
      / / / / / / ___/ ___/ / ___/  / / / /\__ 
     / / / /_/ / /  / /  / (__  )  / /_/ /___/ / 
    /_/  \__,_/_/  /_/  /_/____/   \____//____/  
                                             
 -----------------------------------------------------
 TurrisOS 6.3.2, Turris Mox
 -----------------------------------------------------
root@turris:~# lsusb
Bus 003 Device 002: ID 0bc2:3322 Seagate Expansion Desk
Bus 002 Device 001: ID 1d6b:0002 Linux 5.15.96 xhci-hcd xHCI Host Controller
Bus 001 Device 001: ID 1d6b:0002 Linux 5.15.96 ehci_hcd EHCI Host Controller
Bus 003 Device 001: ID 1d6b:0003 Linux 5.15.96 xhci-hcd xHCI Host Controller

csett86 · May 12, 2023, 7:57pm

I can also confirm that the USB3 SSD now continues to work with my MOX after the TOS 6.0 upgrade. Fix was the u-boot upgrade from U-Boot 2018.11 to U-Boot 2022.07 via

opkg install fix-nor-update

Now I have:

U-Boot 2022.07 (Aug 15 2022 - 12:25:08 +0000)
TurrisOS 6.3.2, Turris Mox

USB SSD listed (as it was before with TOS 5)

root@turris:~# lsusb
Bus 003 Device 002: ID 0781:55bb SanDisk Portable SSD
Bus 002 Device 001: ID 1d6b:0002 Linux 5.15.96 xhci-hcd xHCI Host Controller
Bus 001 Device 001: ID 1d6b:0002 Linux 5.15.96 ehci_hcd EHCI Host Controller
Bus 003 Device 001: ID 1d6b:0003 Linux 5.15.96 xhci-hcd xHCI Host Controller

csett86 · May 12, 2023, 7:58pm

Thanks a lot @Mossroy for taking the time and effort to get this solved via Turris Support and lending your hardware!

peci1 · May 12, 2023, 8:19pm

The Turris guys rock! This is the second super-weird low-level problem solved in the last month

Mossroy · May 14, 2023, 9:34am

It’s finally working!

What was needed was “only” a u-boot upgrade, that can be done by installing package fix-nor-update (see below)
After that, I upgraded to TOS 6.3.2 and my USB HDD is detected and working fine

BusyBox v1.30.1 () built-in shell (ash)

      ______                _         ____  _____
     /_  __/_  ____________(_)____   / __ \/ ___/
      / / / / / / ___/ ___/ / ___/  / / / /\__ 
     / / / /_/ / /  / /  / (__  )  / /_/ /___/ / 
    /_/  \__,_/_/  /_/  /_/____/   \____//____/  
                                             
 -----------------------------------------------------
 TurrisOS 5.4.4, Turris Mox
 -----------------------------------------------------
root@turris:~# opkg update
Downloading https://repo.turris.cz/hbs/mox/packages/core/Packages.gz
Updated list of available packages in /var/opkg-lists/turrisos_core
Downloading https://repo.turris.cz/hbs/mox/packages/core/Packages.sig
Signature check passed.
Downloading https://repo.turris.cz/hbs/mox/packages/base/Packages.gz
Updated list of available packages in /var/opkg-lists/turrisos_base
Downloading https://repo.turris.cz/hbs/mox/packages/base/Packages.sig
Signature check passed.
Downloading https://repo.turris.cz/hbs/mox/packages/cesnet/Packages.gz
Updated list of available packages in /var/opkg-lists/turrisos_cesnet
Downloading https://repo.turris.cz/hbs/mox/packages/cesnet/Packages.sig
Signature check passed.
Downloading https://repo.turris.cz/hbs/mox/packages/luci/Packages.gz
Updated list of available packages in /var/opkg-lists/turrisos_luci
Downloading https://repo.turris.cz/hbs/mox/packages/luci/Packages.sig
Signature check passed.
Downloading https://repo.turris.cz/hbs/mox/packages/node/Packages.gz
Updated list of available packages in /var/opkg-lists/turrisos_node
Downloading https://repo.turris.cz/hbs/mox/packages/node/Packages.sig
Signature check passed.
Downloading https://repo.turris.cz/hbs/mox/packages/packages/Packages.gz
Updated list of available packages in /var/opkg-lists/turrisos_packages
Downloading https://repo.turris.cz/hbs/mox/packages/packages/Packages.sig
Signature check passed.
Downloading https://repo.turris.cz/hbs/mox/packages/routing/Packages.gz
Updated list of available packages in /var/opkg-lists/turrisos_routing
Downloading https://repo.turris.cz/hbs/mox/packages/routing/Packages.sig
Signature check passed.
Downloading https://repo.turris.cz/hbs/mox/packages/sidn/Packages.gz
*** Failed to download the package list from https://repo.turris.cz/hbs/mox/packages/sidn/Packages.gz

Downloading https://repo.turris.cz/hbs/mox/packages/telephony/Packages.gz
Updated list of available packages in /var/opkg-lists/turrisos_telephony
Downloading https://repo.turris.cz/hbs/mox/packages/telephony/Packages.sig
Signature check passed.
Downloading https://repo.turris.cz/hbs/mox/packages/turrispackages/Packages.gz
Updated list of available packages in /var/opkg-lists/turrisos_turrispackages
Downloading https://repo.turris.cz/hbs/mox/packages/turrispackages/Packages.sig
Signature check passed.
Collected errors:
 * opkg_download: Failed to download https://repo.turris.cz/hbs/mox/packages/sidn/Packages.gz, wget returned 8.
root@turris:~# opkg install nor-update
Unknown package 'nor-update'.
Collected errors:
 * opkg_install_cmd: Cannot install package nor-update.
root@turris:~# opkg install fix-nor-update
Installing fix-nor-update (1-61) to root...
Downloading https://repo.turris.cz/hbs/mox/packages/turrispackages/fix-nor-update_1-61_aarch64_cortex-a53.ipk
Installing turris-mox-firmware (2.0-3) to root...
Downloading https://repo.turris.cz/hbs/mox/packages/turrispackages/turris-mox-firmware_2.0-3_aarch64_cortex-a53.ipk
Installing turris-nor-update (1.1.0-7) to root...
Downloading https://repo.turris.cz/hbs/mox/packages/turrispackages/turris-nor-update_1.1.0-7_aarch64_cortex-a53.ipk
Configuring turris-mox-firmware.
Configuring turris-nor-update.
Configuring fix-nor-update.
Verifying /dev/mtd0 against secure-firmware.bin ...
18a0c1bc58839cbbc7aef70db637d261 - /dev/mtd0
e12a263c63bd9860cff844763e81e56b - secure-firmware.bin
Failed
Unlocking /dev/mtd0 ...
Erasing /dev/mtd0 ...

Writing from secure-firmware.bin to /dev/mtd0 ...     
Verifying /dev/mtd1 against uboot ...
437cb920201336f556abd2b448248714 - /dev/mtd1
bd802eb0ec60fd89f983ac3cd4860fba - uboot
Failed
Unlocking /dev/mtd1 ...
Erasing /dev/mtd1 ...

Writing from uboot to /dev/mtd1 ...     
Verifying /dev/mtd3 against rescue ...
e9b30ada843663c579829652847c7bb8 - /dev/mtd3
39f02d9fb36b4258bac37924bcdf570c - rescue
Failed
Unlocking /dev/mtd3 ...
Erasing /dev/mtd3 ...

Writing from rescue to /dev/mtd3 ...     
Can't find anything to flash to 'dtb' partition
root@turris:~# strings /dev/mtdblock1 | grep "U-Boot 20"
U-Boot 2022.07 (Aug 15 2022 - 12:25:08 +0000)
root@turris:~# reboot

Mossroy · May 14, 2023, 10:44am

Yes, Marek did a great job.
I’m very grateful for that, and told him.

However, I’m also very disappointed that it took sooo long for us to be taken seriously. A serious regression lasting 7 months!

Now that we know the root cause, I understand that they did not reproduce it on newer Turris Mox boards (that already had a recent U-boot). But, hey, it was not only one person reporting this, we gave all details (including mentioning that we had boards from the indiegogo campaign), tried to follow all directions Turris team gave us etc.

It probably affects all early Turris Mox boards: not cool to let early supporters in the dark like that. I suggested Marek to communicate about that, or to automatically deploy this U-boot update.

NB: I’m wondering if we should keep the “fix-nor-update” package, or if it would be better to remove it now. I suppose that keeping it might provide us subsequent u-boot upgrades (that we would not have by default). Is it a good idea or not? Do we take a risk of having (in the future) a not-so-tested version of U-boot automatically installed with Turris OS upgrade?

peci1 · May 14, 2023, 11:10am

Oh, now imagine you’d solve this kind of regression with Netgear or TP-Link… 7 months would be just the time before you get to communicate with a real engineer (if ever).

Mossroy · May 14, 2023, 12:51pm

Well, you’re probably right, I don’t know.
It’s probably complicated to compare. Other “classical” commercial vendors probably have sold more devices, so might have been informed of the issue from more customers (or hopefully would have found it before shipping the upgrade).

But don’t get me wrong: I know why I chose Turris, and these reasons are still there: free software, transparency, unrestricted capabilities including root rights (that certainly make support more complicated for Turris) etc.
In this particular case, we’ve been “saved” by the ability to rollback to previous Turris OS version, which is probably not a common feature.
However, I still believe there is room for improvement in their processes.

But, anyway, let’s focus on the good news: I’m so glad to finally be able to upgrade my Mox!

jada4p · May 14, 2023, 6:28pm

As far as above entries were encouraging I updated u-boot as well… All seems OK.

Hai · May 15, 2023, 6:21am

I updated the U-Boot and my USB3 SATA adapter works now. Thank you @Mossroy!

peci1 · May 15, 2023, 8:35am

Just out of curiosity: can anyone explain how a bootloader can influence what does kernel see on the USB bus?

cocturr · May 15, 2023, 1:18pm

I have no knowledge about the details of hardware platforms of Turris products, but generally speaking, U-boot does hardware testing and initialization, so anything not done properly in those early stages will have a huge impact on the hardware capabilities as seen by the operating system kernel.

system · May 18, 2023, 1:19pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.