Controller never released inhibit bit(s)

Omnia hardware Omnia HW problems
1

It looks like my Omnia has died.
I’m writing this story in hope that my issue is not unique and there are known ways to solve it, and to remind others to keep back-ups of course.

The first symptoms started after upgrading to 3.11.5 on June 20. The upgrade didn’t went smooth and I experienced significant delays on disk operations (Turris OS 3.11.5 is released!). It did look like a software bug though with regards to btrfs.
Some time later I met some more btrfs issues which were mitigated (or that I thought) by removing old snapshots and rebalancing (Turris OS 3.11.5 is released!).
After that incident I experienced rare spontaneous reboots and could not yet find the root cause (Input power monitoring).
Until last night.
Yesterday evening the router stopped to forward internet traffic for the LAN clients though it was still accessible both from LAN an WAN. I saw a number of error messages in syslog regarding mmc (like the one in the title) and also noticed my rootfs was mounted read-only. Unfortunately I did not copy the logs and other information and just rebooted it in hope it would recover at least to a state when it would be able to provide internet.
It did not.
On restart, the WAN LED chaotically blinks, Power and PCI2 LEDs are on, and the PCI1 LED blinks every second. I don’t know what that means.
I will try to access the serial console tonight.

It all looks either like btrfs corruption (I can remember that fsck.btrfs indeed showed some errors) or eMMC failure.
I did run a mail server and Home Assistant on my Omnia, but I was sure that the flash used in it is strong enough.

So, what does it look like? Is there a chance to recover my data (at least) and the device itself?

2

Even if eMMC is unusable, some people on this forum switched to running Omnia with a drive instead. (But I’m certainly not a good person to advise around these things.)

3

It can be either and will become apparent once the console output is available.

Depends on the cause. There are some tools to rectify btrfs issues or recover/rescue btrfs data - best to consult man btrfs for the best approach/practise.

If the eMMC is gone/worn out chances of data rescue are slim. Would need to boot from a disk in the PCIe slot or interior USB and see if the mmcblk0p1 partition is mountable and then accessible/readable.

It has been cautioned repeatedly that the eMMC is prone to wear and tear when running services with frequent write operations to the eMMC.

If the eMMC is gone the OS can be run off a drive in the PCIe slot or interior USB drive instead (exterior USB does not seem supported). Best to consult the available documentation for such setup.

4

I also had to transfer (Had to undergo a memory failure ) the system to an SSD MSATA drive Chyby, které vedou k nefunkčnosti připojení - #48 by RomanHK - SW pomoc [CZ] - Turris forum
. . . .

5

Well, I will try to get a serial console and boot in a safe mode (the 5-LED reboot).
Then I will probably try to get an SSD, however if I remember correctly Omnia only has one free mini-pcie (rest are occupied by wifi boards), and I already use it for a zigbee modem.

6

For you … Booting from USB

https://doc.turris.cz/doc/en/howto/omnia_booting_from_external_storage

7

That “USB” also requires a mini-PCIe slot:)

8

By the way, what is the meaning of blinking LEDs? How to understand what error they indicate?

9

Sorry, for my oversights :frowning: Do not have Zigbee modems with a different interface than mini-PCIe? Via ethernet or USB ?

https://www.assured-systems.com/uk/product/mlbz4001-ethernet-to-zigbee-modem-industrial-m2m-module/

10

Not sure if TO patched u-boot in a way that only interior drives can be utilized as alternative to eMMC, least my understanding is that exterior USBs are supposed to be working in upstream OpenWrt.

However, booting with an UART console connected shows that USB drives connected to the exterior ports are not found/discovered - looks like that the pertaining bus is not exposed to u-boot. Subsequently one has to sacrifice one of the interior PCIe slots - either the right one (next to CPU) for a mSATA card or the left one (over the sim slot) for USB (adapter).

11

Here is the boot log:

U-Boot SPL 2015.10-rc2 (Aug 18 2016 - 20:43:35)
High speed PHY - Version: 2.0
SERDES0 card detect: PEX

Initialize Turris board topology
Detected Device ID 6820
board SerDes lanes topology details:
 | Lane #  | Speed |  Type       |
 --------------------------------
 |   0    |  5   |  PCIe0       |
 |   1    |  5   |  USB3 HOST0  |
 |   2    |  5   |  PCIe1       |
 |   3    |  5   |  USB3 HOST1  |
 |   4    |  5   |  PCIe2       |
 |   5    |  0   |  SGMII2      |
 --------------------------------
:** Link is Gen1, check the EP capability
PCIe, Idx 0: remains Gen1
:** Link is Gen1, check the EP capability
PCIe, Idx 1: remains Gen1
PCIe, Idx 2: detected no link
High speed PHY - Ended Successfully
DDR3 Training Sequence - Ver TIP-1.29.0
Memory config in EEPROM: 0x02
DDR3 Training Sequence - Switching XBAR Window to FastPath Window
DDR3 Training Sequence - Ended Successfully
   
   
U-Boot 2015.10-rc2 (Aug 18 2016 - 20:43:35 +0200), Build: jenkins-omnia-master-23
   
SoC:   MV88F6820-A0
       Watchdog enabled
I2C:   ready
SPI:   ready
DRAM:  2 GiB (ECC not enabled)
Enabling Armada 385 watchdog.
Disabling MCU startup watchdog.
Regdomain set to **
MMC:   mv_sdh: 0
SF: Detected S25FL164K with page size 256 Bytes, erase size 64 KiB, total 8 MiB
*** Warning - bad CRC, using default environment
   
PCI:
  00:01.0     - 168c:002e - Network controller
PCI:
  01:00.0     - 11ab:6820 - Memory controller
  01:01.0     - 168c:003c - Network controller
Model: Marvell Armada 385 GP
Board: Turris Omnia SN: 0000000B000128C9
Regdomain set to **
SCSI:  MVEBU SATA INIT
SATA link 0 timeout.
SATA link 1 timeout.
AHCI 0001.0000 32 slots 2 ports 6 Gbps 0x3 impl SATA mode
flags: 64bit ncq led only pmp fbss pio slum part sxs
Net:   neta2
Hit any key to stop autoboot:  0
Setting bus to 1
BOOT eMMC FS
btrfs probe failed
** Unrecognized filesystem type **
btrfs probe failed
** Unrecognized filesystem type **
Kernel image @ 0x1000000 [ 0x000000 - 0x689b30 ]
ERROR: Did not find a cmdline Flattened Device Tree
Could not find a valid device tree
=>
12

And this is the rescue shell.

U-Boot SPL 2015.10-rc2 (Aug 18 2016 - 20:43:35)
High speed PHY - Version: 2.0
SERDES0 card detect: PEX

Initialize Turris board topology
Detected Device ID 6820
board SerDes lanes topology details:
 | Lane #  | Speed |  Type       |
 --------------------------------
 |   0    |  5   |  PCIe0       |
 |   1    |  5   |  USB3 HOST0  |
 |   2    |  5   |  PCIe1       |
 |   3    |  5   |  USB3 HOST1  |
 |   4    |  5   |  PCIe2       |
 |   5    |  0   |  SGMII2      |
 --------------------------------
:** Link is Gen1, check the EP capability
PCIe, Idx 0: remains Gen1
:** Link is Gen1, check the EP capability
PCIe, Idx 1: remains Gen1
PCIe, Idx 2: detected no link
High speed PHY - Ended Successfully
DDR3 Training Sequence - Ver TIP-1.29.0
Memory config in EEPROM: 0x02
DDR3 Training Sequence - Switching XBAR Window to FastPath Window
DDR3 Training Sequence - Ended Successfully


U-Boot 2015.10-rc2 (Aug 18 2016 - 20:43:35 +0200), Build: jenkins-omnia-master-23

SoC:   MV88F6820-A0
       Watchdog enabled
I2C:   ready
SPI:   ready
DRAM:  2 GiB (ECC not enabled)
Enabling Armada 385 watchdog.
Disabling MCU startup watchdog.
Regdomain set to **
MMC:   mv_sdh: 0
SF: Detected S25FL164K with page size 256 Bytes, erase size 64 KiB, total 8 MiB
*** Warning - bad CRC, using default environment

PCI:
  00:01.0     - 168c:002e - Network controller
PCI:
  01:00.0     - 11ab:6820 - Memory controller
  01:01.0     - 168c:003c - Network controller
Model: Marvell Armada 385 GP
Board: Turris Omnia SN: 0000000B000128C9
Regdomain set to **
SCSI:  MVEBU SATA INIT
SATA link 0 timeout.
SATA link 1 timeout.
AHCI 0001.0000 32 slots 2 ports 6 Gbps 0x3 impl SATA mode
flags: 64bit ncq led only pmp fbss pio slum part sxs
Net:   neta2
Hit any key to stop autoboot:  0
Setting bus to 1
BOOT RESCUE
SF: Detected S25FL164K with page size 256 Bytes, erase size 64 KiB, total 8 MiB
device 0 offset 0x100000, size 0x700000
SF: 7340032 bytes @ 0x100000 Read: OK
Kernel image @ 0x1000000 [ 0x000000 - 0x689b30 ]

Starting kernel ...

[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 4.4.13-369554f06f5670aea91212be2c6d6e58-7 (beast@slave01) (gcc version 4.8.3 (OpenWrt/Linaro GCC 4.8-2014.04 r49274) ) #4 SMP Thu Aug 18 20:46:34 CEST 2016
[    0.000000] CPU: ARMv7 Processor [414fc091] revision 1 (ARMv7), cr=10c5387d
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[    0.000000] Machine model: Turris Omnia
[    0.000000] Memory policy: Data cache writealloc
[    0.000000] PERCPU: Embedded 11 pages/cpu @eefcb000 s13952 r8192 d22912 u45056
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 522752
[    0.000000] Kernel command line: earlyprintk console=ttyS0,115200 rootfstype=btrfs rootdelay=2 root=b301 rootflags=subvol=@,commit=5 rw omniarescue=4
[    0.000000] PID hash table entries: 4096 (order: 2, 16384 bytes)
[    0.000000] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
[    0.000000] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
[    0.000000] Memory: 2069096K/2097152K available (4730K kernel code, 144K rwdata, 676K rodata, 4912K init, 182K bss, 28056K reserved, 0K cma-reserved, 1310720K highmem)
[    0.000000] Virtual kernel memory layout:
[    0.000000]     vector  : 0xffff0000 - 0xffff1000   (   4 kB)
[    0.000000]     fixmap  : 0xffc00000 - 0xfff00000   (3072 kB)
[    0.000000]     vmalloc : 0xf0800000 - 0xff800000   ( 240 MB)
[    0.000000]     lowmem  : 0xc0000000 - 0xf0000000   ( 768 MB)
[    0.000000]     pkmap   : 0xbfe00000 - 0xc0000000   (   2 MB)
[    0.000000]     modules : 0xbf000000 - 0xbfe00000   (  14 MB)
[    0.000000]       .text : 0xc0008000 - 0xc054fe6c   (5408 kB)
[    0.000000]       .init : 0xc0550000 - 0xc0a1c000   (4912 kB)
[    0.000000]       .data : 0xc0a1c000 - 0xc0a40274   ( 145 kB)
[    0.000000]        .bss : 0xc0a40274 - 0xc0a6dc38   ( 183 kB)
[    0.000000] Hierarchical RCU implementation.
[    0.000000]  RCU restricting CPUs from NR_CPUS=4 to nr_cpu_ids=2.
[    0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=2
[    0.000000] NR_IRQS:16 nr_irqs:16 16
[    0.000000] mvebu_mbus: [Firmware Warn]: deprecated mbus-mvebu Device Tree, suspend/resume will not work
[    0.000000] L2C-310 enabling early BRESP for Cortex-A9
[    0.000000] L2C-310 full line of zeros enabled for Cortex-A9
[    0.000000] L2C-310 D prefetch enabled, offset 1 lines
[    0.000000] L2C-310 dynamic clock gating enabled, standby mode enabled
[    0.000000] L2C-310 Coherent cache controller enabled, 16 ways, 1024 kB
[    0.000000] L2C-310 Coherent: CACHE_ID 0x410054c9, AUX_CTRL 0x56070001
[    0.000000] Switching to timer-based delay loop, resolution 40ns
[    0.000003] sched_clock: 32 bits at 25MHz, resolution 40ns, wraps every 85899345900ns
[    0.000010] clocksource: armada_370_xp_clocksource: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 76450417870 ns
[    0.000149] Calibrating delay loop (skipped), value calculated using timer frequency.. 50.00 BogoMIPS (lpj=250000)
[    0.000156] pid_max: default: 32768 minimum: 301
[    0.000203] Mount-cache hash table entries: 2048 (order: 1, 8192 bytes)
[    0.000208] Mountpoint-cache hash table entries: 2048 (order: 1, 8192 bytes)
[    0.000423] CPU: Testing write buffer coherency: ok
[    0.000528] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
[    0.000541] Setting up static identity map for 0x8280 - 0x82d8
[    0.000619] mvebu-soc-id: MVEBU SoC ID=0x6820, Rev=0x4
[    0.000699] mvebu-pmsu: Initializing Power Management Service Unit
[    0.001122] Booting CPU 1
[    0.001293] CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
[    0.001322] Brought up 2 CPUs
[    0.001328] SMP: Total of 2 processors activated (100.00 BogoMIPS).
[    0.001332] CPU: All CPU(s) started in SVC mode.
[    0.003014] VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 4
[    0.003071] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.003089] xor: measuring software checksum speed
[    0.099896]    arm4regs  :  2786.800 MB/sec
[    0.199898]    8regs     :  1859.200 MB/sec
[    0.299896]    32regs    :  1669.200 MB/sec
[    0.299899] xor: using function: arm4regs (2786.800 MB/sec)
[    0.299905] pinctrl core: initialized pinctrl subsystem
[    0.300235] NET: Registered protocol family 16
[    0.300730] DMA: preallocated 256 KiB pool for atomic coherent allocations
[    0.329920] cpuidle: using governor ladder
[    0.332549] mvebu-pmsu: CPU hotplug support is currently broken on Armada 38x: disabling
[    0.332556] mvebu-pmsu: CPU idle is currently broken on Armada 38x: disabling
[    0.530191] raid6: int32x1  gen()   155 MB/s
[    0.699942] raid6: int32x1  xor()   202 MB/s
[    0.869929] raid6: int32x2  gen()   255 MB/s
[    1.039947] raid6: int32x2  xor()   229 MB/s
[    1.209975] raid6: int32x4  gen()   284 MB/s
[    1.379980] raid6: int32x4  xor()   190 MB/s
[    1.549919] raid6: int32x8  gen()   371 MB/s
[    1.719940] raid6: int32x8  xor()   214 MB/s
[    1.719943] raid6: using algorithm int32x8 gen() 371 MB/s
[    1.719947] raid6: .... xor() 214 MB/s, rmw enabled
[    1.719950] raid6: using intx1 recovery algorithm
[    1.720540] clocksource: Switched to clocksource armada_370_xp_clocksource
[    1.720893] NET: Registered protocol family 2
[    1.721131] TCP established hash table entries: 8192 (order: 3, 32768 bytes)
[    1.721169] TCP bind hash table entries: 8192 (order: 4, 65536 bytes)
[    1.721230] TCP: Hash tables configured (established 8192 bind 8192)
[    1.721266] UDP hash table entries: 512 (order: 2, 16384 bytes)
[    1.721290] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes)
[    1.721391] NET: Registered protocol family 1
[    2.857834] futex hash table entries: 512 (order: 3, 32768 bytes)
[    2.858153] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    2.858158] jffs2: version 2.2 (NAND) (SUMMARY) (LZMA) (RTIME) (CMODE_PRIORITY) (c) 2001-2006 Red Hat, Inc.
[    2.858473] bounce: pool size: 64 pages
[    2.858503] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252)
[    2.858508] io scheduler noop registered
[    2.858511] io scheduler deadline registered (default)
[    2.858881] armada-38x-pinctrl f1018000.pinctrl: registered pinctrl driver
[    2.859203] irq: Cannot allocate irq_descs @ IRQ48, assuming pre-allocated
[    2.859351] irq: Cannot allocate irq_descs @ IRQ80, assuming pre-allocated
[    2.859672] mvebu-pcie soc:pcie-controller: PCI host bridge to bus 0000:00
[    2.859681] pci_bus 0000:00: root bus resource [io  0x1000-0xfffff]
[    2.859686] pci_bus 0000:00: root bus resource [mem 0xe0000000-0xe7ffffff]
[    2.859692] pci_bus 0000:00: root bus resource [bus 00-ff]
[    2.859984] PCI: bus0: Fast back to back transfers disabled
[    2.859991] pci 0000:00:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    2.859997] pci 0000:00:02.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    2.860003] pci 0000:00:03.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    2.860265] PCI: bus1: Fast back to back transfers disabled
[    2.860557] PCI: bus2: Fast back to back transfers disabled
[    2.860617] PCI: bus3: Fast back to back transfers enabled
[    2.860681] pci 0000:00:01.0: BAR 8: assigned [mem 0xe0000000-0xe00fffff]
[    2.860688] pci 0000:00:02.0: BAR 8: assigned [mem 0xe0200000-0xe04fffff]
[    2.860695] pci 0000:01:00.0: BAR 0: assigned [mem 0xe0000000-0xe000ffff 64bit]
[    2.860706] pci 0000:00:01.0: PCI bridge to [bus 01]
[    2.860712] pci 0000:00:01.0:   bridge window [mem 0xe0000000-0xe00fffff]
[    2.860720] pci 0000:02:00.0: BAR 0: assigned [mem 0xe0200000-0xe03fffff 64bit]
[    2.860731] pci 0000:02:00.0: BAR 6: assigned [mem 0xe0400000-0xe040ffff pref]
[    2.860735] pci 0000:00:02.0: PCI bridge to [bus 02]
[    2.860741] pci 0000:00:02.0:   bridge window [mem 0xe0200000-0xe04fffff]
[    2.860746] pci 0000:00:03.0: PCI bridge to [bus 03]
[    2.860853] mv_xor f1060800.xor: Marvell shared XOR driver
[    2.900840] mv_xor f1060800.xor: Marvell XOR (Descriptor Mode): ( xor cpy intr )
[    2.900936] mv_xor f1060900.xor: Marvell shared XOR driver
[    2.940811] mv_xor f1060900.xor: Marvell XOR (Descriptor Mode): ( xor cpy intr )
[    2.940947] Serial: 8250/16550 driver, 2 ports, IRQ sharing disabled
[    2.941247] console [ttyS0] disabled
[    2.961313] f1012000.serial: ttyS0 at MMIO 0xf1012000 (irq = 22, base_baud = 15625000) is a 16550A
[    3.716215] console [ttyS0] enabled
[    3.739925] f1012100.serial: ttyS1 at MMIO 0xf1012100 (irq = 23, base_baud = 15625000) is a 16550A
[    3.749625] m25p80 spi0.0: s25fl164k (8192 Kbytes)
[    3.755921] 2 ofpart partitions found on MTD device spi0.0
[    3.761432] Creating 2 MTD partitions on "spi0.0":
[    3.766235] 0x000000000000-0x000000100000 : "U-Boot"
[    3.771569] 0x000000100000-0x000000800000 : "Rescue system"
[    3.777825] libphy: Fixed MDIO Bus: probed
[    3.782158] libphy: orion_mdio_bus: probed
[    3.788279] mvneta f1030000.ethernet eth0: Using hardware mac address d8:58:d7:00:6b:7f
[    3.797065] mvneta f1034000.ethernet eth1: Using hardware mac address d8:58:d7:00:6b:80
[    3.805880] mvneta f1070000.ethernet eth2: Using hardware mac address d8:58:d7:00:6b:81
[    3.814134] armada38x-rtc f10a3800.rtc: rtc core: registered f10a3800.rtc as rtc0
[    4.330643] i2c /dev entries driver
[    4.334886] at24 1-0054: 8192 byte 24c64 EEPROM, writable, 1 bytes/write
[    4.341659] i2c i2c-0: Added multiplexed i2c bus 1
[    4.346530] i2c i2c-0: Added multiplexed i2c bus 2
[    4.351410] i2c i2c-0: Added multiplexed i2c bus 3
[    4.356272] i2c i2c-0: Added multiplexed i2c bus 4
[    4.361145] i2c i2c-0: Added multiplexed i2c bus 5
[    4.366007] i2c i2c-0: Added multiplexed i2c bus 6
[    4.370877] i2c i2c-0: Added multiplexed i2c bus 7
[    4.375778] i2c i2c-0: Added multiplexed i2c bus 8
[    4.380588] pca954x 0-0070: registered 8 multiplexed busses for I2C mux pca9547
[    4.388050] sdhci: Secure Digital Host Controller Interface driver
[    4.394251] sdhci: Copyright(c) Pierre Ossman
[    4.398762] sdhci-pxav3 f10d8000.sdhci: No vmmc regulator found
[    4.404705] sdhci-pxav3 f10d8000.sdhci: No vqmmc regulator found
[    4.450536] mmc0: SDHCI controller on f10d8000.sdhci [f10d8000.sdhci] using ADMA
[    4.457977] sdhci-pltfm: SDHCI platform and OF driver helper
[    4.472936] NET: Registered protocol family 10
[    4.477860] NET: Registered protocol family 17
[    4.482337] bridge: automatic filtering via arp/ip/ip6tables has been deprecated. Update your scripts to load br_netfilter if you need this.
[    4.494976] 8021q: 802.1Q VLAN Support v1.8
[    4.499216] Registering SWP/SWPB emulation handler
[    4.504603] Btrfs loaded
[    4.507709] mvsw61xx 10.mvsw61xx: Found MV88E6176 at f1072004.mdio-mi:10
[    4.513811] mmc0: new high speed MMC card at address 0001
[    4.519855] mvsw61xx 10.mvsw61xx: Using indirect addressing
[    4.520612] mmcblk0: mmc0:0001 M MOR 1.83 GiB
[    4.520687] mmcblk0boot0: mmc0:0001 M MOR partition 1 2.00 MiB
[    4.520754] mmcblk0boot1: mmc0:0001 M MOR partition 2 2.00 MiB
[    4.520821] mmcblk0rpmb: mmc0:0001 M MOR partition 3 128 KiB
[    4.588858] armada38x-rtc f10a3800.rtc: setting system clock to 2019-09-19 19:28:45 UTC (1568921325)
[    4.600746] Freeing unused kernel memory: 4912K (c0550000 - c0a1c000)
modprobe grace
modprobe libata
[    4.673818] SCSI subsystem initialized
modprobe async_raid6_rec[    4.685834] async_tx: api initialized (async)
ov
modprobe nls_base
modprobe sha256_generic
modprobe mac8021[    4.695764] Loading modules backported from Linux version master-2015-12-03-0-g1677f62
1
[    4.704132] Backport generated by backports.git backports-20151120-0-g906a6b3
[    4.723469] cfg80211: World regulatory domain updated:
[    4.728622] cfg80211:  DFS Master region: unset
[    4.732999] cfg80211:   (start_freq - end_freq @ bandwidth), (max_antenna_gain, max_eirp), (dfs_cac_time)
[    4.742786] cfg80211:   (2402000 KHz - 2472000 KHz @ 40000 KHz), (N/A, 2000 mBm), (N/A)
[    4.750825] cfg80211:   (2457000 KHz - 2482000 KHz @ 20000 KHz, 92000 KHz AUTO), (N/A, 2000 mBm), (N/A)
[    4.760241] cfg80211:   (2474000 KHz - 2494000 KHz @ 20000 KHz), (N/A, 2000 mBm), (N/A)
[    4.768277] cfg80211:   (5170000 KHz - 5250000 KHz @ 80000 KHz, 160000 KHz AUTO), (N/A, 2000 mBm), (N/A)
[    4.777791] cfg80211:   (5250000 KHz - 5330000 KHz @ 80000 KHz, 160000 KHz AUTO), (N/A, 2000 mBm), (0 s)
[    4.787314] cfg80211:   (5490000 KHz - 5730000 KHz @ 160000 KHz), (N/A, 2000 mBm), (0 s)
[    4.795438] cfg80211:   (5735000 KHz - 5835000 KHz @ 80000 KHz), (N/A, 2000 mBm), (N/A)
[    4.803468] cfg80211:   (57240000 KHz - 63720000 KHz @ 2160000 KHz), (N/A, 0 mBm), (N/A)
modprobe fuse
[    4.815913] fuse init (API version 7.23)
modprobe dm-region-hash
[    4.825185] device-mapper: ioctl: 4.34.0-ioctl (2015-10-28) initialised: dm-devel@redhat.com
modprobe xfs
[    4.901417] SGI XFS with security attributes, no debug enabled
modprobe orion_wdt
[    4.909774] orion_wdt: Initial timeout 171 sec
modprobe md4
modprobe mvsdio
modprobe cifs
modprobe cryptodev
[    4.934865] cryptodev: driver 1.8 loaded.
modprobe async_xor
modprobe ahci_mvebu
[    4.944492] ahci-mvebu f10a8000.sata: AHCI 0001.0000 32 slots 2 ports 6 Gbps 0x3 impl platform mode
[    4.953580] ahci-mvebu f10a8000.sata: flags: 64bit ncq sntf led only pmp fbs pio slum part sxs
[    4.963050] scsi host0: ahci-mvebu
[    4.966625] scsi host1: ahci-mvebu
[    4.970128] ata1: SATA max UDMA/133 mmio [mem 0xf10a8000-0xf10a9fff] port 0x100 irq 44
[    4.978074] ata2: SATA max UDMA/133 mmio [mem 0xf10a8000-0xf10a9fff] port 0x180 irq 44
[    5.330533] ata2: SATA link down (SStatus 0 SControl 300)
[    5.335960] ata1: SATA link down (SStatus 0 SControl 300)
modprobe marvell-cesa
[    5.344698] marvell-cesa f1090000.crypto: CESA device successfully registered
modprobe libcrc32c
modprobe fat
modprobe dm-crypt
modprobe algif_hash
[    5.360744] NET: Registered protocol family 38
modprobe sunrpc
[    5.377543] RPC: Registered named UNIX socket transport module.
[    5.383509] RPC: Registered udp transport module.
[    5.388222] RPC: Registered tcp transport module.
[    5.392950] RPC: Registered tcp NFSv4.1 backchannel transport module.
modprobe nfs
modprobe af_alg
modprobe algif_skcipher
modprobe dm-mod
modprobe nls_utf8
modprobe vfat
modprobe nfsv3
modprobe raid1
[    5.426874] md: raid1 personality registered for level 1
modprobe sd_mod
modprobe loop
[    5.439425] loop: module loaded
modprobe ahci_platform
modprobe libahci_platform
modprobe md5
modprobe multipath
[    5.450652] md: multipath personality registered for level -4
modprobe usb-storage
[    5.466120] usbcore: registered new interface driver usbfs
[    5.471675] usbcore: registered new interface driver hub
[    5.477035] usbcore: registered new device driver usb
[    5.483307] usbcore: registered new interface driver usb-storage
modprobe nls_cp437
modprobe des_generic
modprobe lockd
modprobe usb-common
modprobe hwmon
modprobe seqiv
modprobe async_pq
modprobe armada_thermal
modprobe dm-mirror
modprobe ehci-orion
[    5.520954] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    5.527688] ehci-orion: EHCI orion driver
[    5.531861] orion-ehci f1058000.usb: EHCI Host Controller
[    5.537282] orion-ehci f1058000.usb: new USB bus registered, assigned bus number 1
[    5.544954] orion-ehci f1058000.usb: irq 39, io mem 0xf1058000
[    5.570546] orion-ehci f1058000.usb: USB 2.0 started, EHCI 1.00
[    5.576733] hub 1-0:1.0: USB hub found
[    5.580504] hub 1-0:1.0: 1 port detected
modprobe gf128mul
modprobe cbc
modprobe cfg80211
modprobe thermal_sys
modprobe xhci-pci
modprobe async_memcpy
modprobe hmac
modprobe authenc
modprobe md-mod
modprobe sha512_generic
modprobe dm-log
modprobe xts
modprobe ehci-hcd
modprobe usbcore
modprobe aead
modprobe crypto_null
modprobe ath
modprobe linear
[    5.620578] md: linear personality registered for level -1
modprobe raid10
[    5.628857] md: raid10 personality registered for level 10
modprobe libahci
modprobe raid0
[    5.638196] md: raid0 personality registered for level 0
modprobe compat
modprobe xhci-hcd
modprobe xhci-plat-hcd
[    5.649077] xhci-hcd f10f0000.usb3: xHCI Host Controller
[    5.654553] xhci-hcd f10f0000.usb3: new USB bus registered, assigned bus number 2
[    5.662174] xhci-hcd f10f0000.usb3: hcc params 0x0a000990 hci version 0x100 quirks 0x00010010
[    5.670757] xhci-hcd f10f0000.usb3: irq 46, io mem 0xf10f0000
[    5.676780] hub 2-0:1.0: USB hub found
[    5.680604] hub 2-0:1.0: 1 port detected
[    5.684636] xhci-hcd f10f0000.usb3: xHCI Host Controller
[    5.689965] xhci-hcd f10f0000.usb3: new USB bus registered, assigned bus number 3
[    5.697534] usb usb3: We don't know the algorithms for LPM for this host, disabling LPM.
[    5.705885] hub 3-0:1.0: USB hub found
[    5.709654] hub 3-0:1.0: 1 port detected
[    5.713777] xhci-hcd f10f8000.usb3: xHCI Host Controller
[    5.719109] xhci-hcd f10f8000.usb3: new USB bus registered, assigned bus number 4
[    5.726721] xhci-hcd f10f8000.usb3: hcc params 0x0a000990 hci version 0x100 quirks 0x00010010
[    5.735302] xhci-hcd f10f8000.usb3: irq 47, io mem 0xf10f8000
[    5.741308] hub 4-0:1.0: USB hub found
[    5.745079] hub 4-0:1.0: 1 port detected
[    5.749102] xhci-hcd f10f8000.usb3: xHCI Host Controller
[    5.754453] xhci-hcd f10f8000.usb3: new USB bus registered, assigned bus number 5
[    5.761999] usb usb5: We don't know the algorithms for LPM for this host, disabling LPM.
[    5.770319] hub 5-0:1.0: USB hub found
[    5.774110] hub 5-0:1.0: 1 port detected
modprobe ehci-platform
[    5.780422] ehci-platform: EHCI generic platform driver
modprobe exportfs
modprobe async_tx
modprobe block2mtd
modprobe ecb
modprobe exfat
[    5.797272] exFAT: Version 1.2.9
modprobe raid456
[    5.806461] md: raid6 personality registered for level 6
[    5.811835] md: raid5 personality registered for level 5
[    5.817157] md: raid4 personality registered for level 4
modprobe scsi_mod
MODE=4
Error: Invalid mode 4. Exit to shell.


BusyBox v1.23.2 (2016-08-18 20:38:45 CEST) built-in shell (ash)

/ # [    5.900550] usb 1-1: new full-speed USB device number 2 using orion-ehci

Is kernel image on the same eMMC chip?

13 
/ # fdisk /dev/mmcblk0

Welcome to fdisk (util-linux 2.25.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

[  260.691093] mmcblk0: error -110 sending stop command, original cmd response 0x80000900, card status 0x400900
[  260.700968] mmcblk0: r/w command failed, status = 0x80000900
[  260.706643] blk_update_request: I/O error, dev mmcblk0, sector 3848064
[  266.081993] mmcblk0: error -110 sending stop command, original cmd response 0x80000900, card status 0x400900
[  266.091866] mmcblk0: r/w command failed, status = 0x80000900
[  266.097541] blk_update_request: I/O error, dev mmcblk0, sector 3848064
[  266.104101] Buffer I/O error on dev mmcblk0, logical block 481008, async page read
Device does not [  266.111779] random: fdisk urandom read with 33 bits of entropy available
contain a recognized partition table.
14

You mean the one for the rescue shell? That resides in the SPI

Three potential causes:

  • the partition table got corrupted (probable but unlikely)
  • btrfs signature a/o metadata got corrupted (probable and not unheard of)
  • eMMC is shot (probable and most likely) because the rescue shell does not recognize/mount the mmcblk0p1 partition
15

that output from fdisk would seem indicative of the the eMMC being worn out

16

u-boot provides two cli utilities related to MMC

  • mmc
  • mmcinfo

I do not think either provides inspection for bad blocks but you might want to have a look anyway.

17

Thank you!

I am also trying to read and copy on a USB disk of what’s still possible to read with dd

18

Well, the dd run for more than 40 hours and only copied 140M with a number of errors. I will continue later. Will try to get a system running with an mSATA SSD then will continue attempting to restore data.

Example of dd errors:

[128501.163907] mmcblk0: error -110 sending stop command, original cmd response 0x80000900, card status 0x400900
[128501.173868] mmcblk0: r/w command failed, status = 0x80000900
[128501.179630] blk_update_request: I/O error, dev mmcblk0, sector 285880
[128501.186189] Buffer I/O error on dev mmcblk0, logical block 35735, async page read
dd: /dev/mmcblk0: I/O error
[128506.564174] mmcblk0: error -110 sending stop command, original cmd response 0x80000900, card status 0x400900
[128506.574135] mmcblk0: r/w command failed, status = 0x80000900
[128506.579895] blk_update_request: I/O error, dev mmcblk0, sector 285880
[128506.586455] Buffer I/O error on dev mmcblk0, logical block 35735, async page read
dd: /dev/mmcblk0: I/O error
[128511.964455] mmcblk0: error -110 sending stop command, original cmd response 0x80000900, card status 0x400900
[128511.974417] mmcblk0: r/w command failed, status = 0x80000900
[128511.980178] blk_update_request: I/O error, dev mmcblk0, sector 285880
[128511.986738] Buffer I/O error on dev mmcblk0, logical block 35735, async page read
dd: /dev/mmcblk0: I/O error
[128517.365292] mmcblk0: error -110 sending stop command, original cmd response 0x80000900, card status 0x400900
[128517.375254] mmcblk0: r/w command failed, status = 0x80000900
[128517.381030] blk_update_request: I/O error, dev mmcblk0, sector 285880
[128517.387574] Buffer I/O error on dev mmcblk0, logical block 35735, async page read
dd: /dev/mmcblk0: I/O error
[128522.765349] mmcblk0: error -110 sending stop command, original cmd response 0x80000900, card status 0x400900
[128522.775310] mmcblk0: r/w command failed, status = 0x80000900
[128522.781087] blk_update_request: I/O error, dev mmcblk0, sector 285888
[128522.787630] Buffer I/O error on dev mmcblk0, logical block 35736, async page read
dd: /dev/mmcblk0: I/O error
[128528.165744] mmcblk0: error -110 sending stop command, original cmd response 0x80000900, card status 0x400900
[128528.175706] mmcblk0: r/w command failed, status = 0x80000900
[128528.181484] blk_update_request: I/O error, dev mmcblk0, sector 285888
[128528.188028] Buffer I/O error on dev mmcblk0, logical block 35736, async page read
dd: /dev/mmcblk0: I/O error

This is the output from mmc u-boot command:

=> mmc
mmc - MMC sub system

Usage:
mmc info - display info of the current MMC device
mmc read addr blk# cnt
mmc write addr blk# cnt
mmc erase blk# cnt
mmc rescan
mmc part - lists available partition on current mmc device
mmc dev [dev] [part] - show or set current mmc device [partition]
mmc list - lists available devices
mmc hwpartition [args...] - does hardware partitioning
  arguments (sizes in 512-byte blocks):
    [user [enh start cnt] [wrrel {on|off}]] - sets user data area attributes
    [gp1|gp2|gp3|gp4 cnt [enh] [wrrel {on|off}]] - general purpose partition
    [check|set|complete] - mode, complete set partitioning completed
  WARNING: Partitioning is a write-once setting once it is set to complete.
  Power cycling is required to initialize partitions after set to complete.
mmc setdsr <value> - set DSR register value

=> mmc info
Device: mv_sdh
Manufacturer ID: 65
OEM: 646f
Name: M MOR 
Tran Speed: 52000000
Rd Block Len: 512
MMC version 4.4.1
High Capacity: Yes
Capacity: 2 TiB
Bus Width: 8-bit
Erase Group Size: 512 Bytes
HC WP Group Size: 512 KiB
User Capacity: 2 TiB
Boot Capacity: 2 MiB ENH
RPMB Capacity: 128 KiB ENH
=> mmc list
mv_sdh: 0 (eMMC)
=> mmc rescan
=> mmc list  
mv_sdh: 0 (eMMC)
=> mmc info  
Device: mv_sdh
Manufacturer ID: 65
OEM: 646f
Name: M MOR 
Tran Speed: 52000000
Rd Block Len: 512
MMC version 4.4.1
High Capacity: Yes
Capacity: 2 TiB
Bus Width: 8-bit
Erase Group Size: 512 Bytes
HC WP Group Size: 512 KiB
User Capacity: 2 TiB
Boot Capacity: 2 MiB ENH
RPMB Capacity: 128 KiB ENH
=> mmc part
## Unknown partition table

What does partition table on a normal device look like?

19

He have not. The missing support is for usb3 specific driver. The PCIe one is (if I am not mistaken) usb2 which works. On Mox this is already fixed and we have also u-boot for omnia with that support but with new version of u-boot we encountered some problems our hardware guyes are looking in to. This means that we have internal build that is not yet available that works on some boards and on others it does not.

1 Like
20

Thanks for the clarification.

Do you know whether the USB device port designation will be static or random, e.g. currently with the interior port only the env variable is set to usb 0:2 (0=device port and 2=partition on that device).

What I am trying to ask is whether the ports are static designated in a way like:

interior = port 0 (or 1 or 2)
exterior front = port 1 (or 0 or 2)
exterior back = port 2 (or 0 or 1)

On a 5x TO installation with u-boot-omnia 2019-07-4.0 the UART output however reads

> => version

U-Boot 2015.10-rc2 (Aug 18 2016 - 20:43:35 +0200)
arm-openwrt-linux-muslgnueabi-gcc (OpenWrt/Linaro GCC 4.8-2014.04 r49274) 4.8.3
GNU ld (GNU Binutils) 2.24.0

Should it not be expected to match the new version instead?