Omnia 2016 boot loop after upgrade to 6.0

maurer · October 20, 2022, 3:54pm

So my omnia 2016 1G nowifi (original indiegogo hw) is boot looping after upgrade to 6.0
easily reverted to 5.4.4 via 2 led reset button trick
I’d say this is the relevant part from the serial console log:

[    2.037329] BTRFS: device fsid 30605799-ecc5-4556-ac68-6a0b46c6ff76 devid 1 transid 1051161 /dev/root scanned by swapper/0 (1)
[    2.049338] BTRFS info (device mmcblk0p1): disk space caching is enabled
[    2.056059] BTRFS info (device mmcblk0p1): has skinny extents
[    2.066449] BTRFS critical (device mmcblk0p1): corrupt leaf: block=153014272 slot=26 extent bytenr=13107200 len=4096 previous extent [12988416 168 364544] overlaps current extent [13107200 169 0]
[    2.083896] BTRFS error (device mmcblk0p1): block=153014272 read time tree block corruption detected
[    2.086965] usb-storage 2-1:1.0: USB Mass Storage device detected
[    2.099209] BTRFS error (device mmcblk0p1): failed to read block groups: -5
[    2.106509] scsi host2: usb-storage 2-1:1.0
[    2.111127] BTRFS error (device mmcblk0p1): open_ctree failed
[    2.118114] squashfs: Unknown parameter 'commit'
[    2.122965] F2FS-fs (mmcblk0p1): Magic Mismatch, valid(0xf2f52010) - read(0x0)
[    2.130226] F2FS-fs (mmcblk0p1): Can't find valid F2FS filesystem in 1th superblock
[    2.138092] F2FS-fs (mmcblk0p1): Magic Mismatch, valid(0xf2f52010) - read(0x0)
[    2.145334] F2FS-fs (mmcblk0p1): Can't find valid F2FS filesystem in 2th superblock
[    2.153838] BTRFS info (device mmcblk0p1): disk space caching is enabled
[    2.160624] BTRFS info (device mmcblk0p1): has skinny extents
[    2.170933] BTRFS critical (device mmcblk0p1): corrupt leaf: block=153014272 slot=26 extent bytenr=13107200 len=4096 previous extent [12988416 168 364544] overlaps current extent [13107200 169 0]
[    2.188392] BTRFS error (device mmcblk0p1): block=153014272 read time tree block corruption detected
[    2.197573] BTRFS error (device mmcblk0p1): failed to read block groups: -5
[    2.204924] BTRFS error (device mmcblk0p1): open_ctree failed
[    2.210815] List of all partitions:
[    2.214321] 1f00             960 mtdblock0 
[    2.214326]  (driver?)
[    2.220889] 1f01            7168 mtdblock1 
[    2.220893]  (driver?)
[    2.227463] 1f02              64 mtdblock2 
[    2.227467]  (driver?)
[    2.234020] 0800        31266648 sda 
[    2.234023]  driver: sd
[    2.240145]   0801        31265607 sda1 75206d7c-69db-a541-8044-ef029c8fecc9
[    2.240150] 
[    2.248709] b300         7634944 mmcblk0 
[    2.248713]  driver: mmcblk
[    2.255525]   b301         7633920 mmcblk0p1 109ac059-01
[    2.255528] 
[    2.262348] No filesystem could mount root, tried: 
[    2.262350]  ext3
[    2.267240]  ext2
[    2.269167]  ext4
[    2.271094]  squashfs
[    2.273021]  f2fs
[    2.275295]  btrfs
[    2.277225] 
[    2.280730] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(179,1)
[    2.289187] CPU1: stopping
[    2.291901] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 5.15.74 #0
[    2.297925] Hardware name: Marvell Armada 380/385 (Device Tree)
[    2.303861] [<c010eab8>] (unwind_backtrace) from [<c010a814>] (show_stack+0x10/0x14)
[    2.311634] [<c010a814>] (show_stack) from [<c05e3e48>] (dump_stack_lvl+0x40/0x4c)
[    2.319226] [<c05e3e48>] (dump_stack_lvl) from [<c010cce8>] (do_handle_IPI+0xf8/0x12c)
[    2.327163] [<c010cce8>] (do_handle_IPI) from [<c010cd34>] (ipi_handler+0x18/0x20)
[    2.334753] [<c010cd34>] (ipi_handler) from [<c0184774>] (handle_percpu_devid_irq+0x78/0x12c)
[    2.343303] [<c0184774>] (handle_percpu_devid_irq) from [<c017ec70>] (handle_domain_irq+0x5c/0x78)
[    2.352290] [<c017ec70>] (handle_domain_irq) from [<c05fc200>] (gic_handle_irq+0x7c/0x90)
[    2.360494] [<c05fc200>] (gic_handle_irq) from [<c0100b7c>] (__irq_svc+0x5c/0x78)
[    2.367998] Exception stack(0xc1083f58 to 0xc1083fa0)
[    2.373061] 3f40:                                                       00001054 00000000
[    2.381259] 3f60: 00000001 c0117aa0 ffffe000 c0e04f50 c0e04f94 00000002 00000000 c0e04f08
[    2.389456] 3f80: c0d580a8 00000000 c107c03c c1083fa8 c010783c c0107840 60000013 ffffffff
[    2.397651] [<c0100b7c>] (__irq_svc) from [<c0107840>] (arch_cpu_idle+0x38/0x3c)
[    2.405071] [<c0107840>] (arch_cpu_idle) from [<c09e4960>] (default_idle_call+0x2c/0x3c)
[    2.413187] [<c09e4960>] (default_idle_call) from [<c015c3bc>] (do_idle+0x1d8/0x1e0)
[    2.420956] [<c015c3bc>] (do_idle) from [<c015c6ec>] (cpu_startup_entry+0x18/0x1c)
[    2.428548] [<c015c6ec>] (cpu_startup_entry) from [<00101550>] (0x101550)
[    2.435359] Rebooting in 1 seconds..

any pointers @Pepe ?
thanks for all the effort !

maurer · October 21, 2022, 1:04pm

looks like one of the disadvantages of using btrfs - it will fail

miska · October 21, 2022, 1:15pm

Or advantage - it has checksums so it knows it failed and wouldn’t even try to run corrupted data (could lead to all sort of errors and data loss). Either some directory structure is corrupted - so droping the snapshots and trying to update again might work. Or there is some deeper problem and in that case, I would export the router filesystem to a flash drive via schnapps and reflash the router from the flash drive. That will create new filesystem from scratch and thus should get you rid of all filesystem inconsistencies.

maurer · October 21, 2022, 1:20pm

@miska scrub output some corruption at 2016 factory level how to fix this ?

[76455.864654] BTRFS warning (device mmcblk0p1): checksum error at logical 5902336 on dev /dev/mmcblk0p1, sector 11528, root 258, inode 261, offset 1658880, length 4096, links 1 (path: boot/zImage-4.4.13-1-05df79f63527051ea0071350f86faf76-7)
[76455.900271] BTRFS warning (device mmcblk0p1): checksum error at logical 5931008 on dev /dev/mmcblk0p1, sector 11584, root 258, inode 261, offset 1687552, length 4096, links 1 (path: boot/zImage-4.4.13-1-05df79f63527051ea0071350f86faf76-7)
[76455.936673] BTRFS warning (device mmcblk0p1): checksum error at logical 5906432 on dev /dev/mmcblk0p1, sector 11536, root 258, inode 261, offset 1662976, length 4096, links 1 (path: boot/zImage-4.4.13-1-05df79f63527051ea0071350f86faf76-7)
[76455.960739] BTRFS warning (device mmcblk0p1): checksum error at logical 5947392 on dev /dev/mmcblk0p1, sector 11616, root 258, inode 261, offset 1703936, length 4096, links 1 (path: boot/zImage-4.4.13-1-05df79f63527051ea0071350f86faf76-7)
[76455.997664] BTRFS warning (device mmcblk0p1): checksum error at logical 5910528 on dev /dev/mmcblk0p1, sector 11544, root 258, inode 261, offset 1667072, length 4096, links 1 (path: boot/zImage-4.4.13-1-05df79f63527051ea0071350f86faf76-7)
[76456.057825] BTRFS warning (device mmcblk0p1): checksum error at logical 5914624 on dev /dev/mmcblk0p1, sector 11552, root 258, inode 261, offset 1671168, length 4096, links 1 (path: boot/zImage-4.4.13-1-05df79f63527051ea0071350f86faf76-7)
[76456.094466] BTRFS warning (device mmcblk0p1): checksum error at logical 5918720 on dev /dev/mmcblk0p1, sector 11560, root 258, inode 261, offset 1675264, length 4096, links 1 (path: boot/zImage-4.4.13-1-05df79f63527051ea0071350f86faf76-7)
[76456.128352] BTRFS warning (device mmcblk0p1): checksum error at logical 5922816 on dev /dev/mmcblk0p1, sector 11568, root 258, inode 261, offset 1679360, length 4096, links 1 (path: boot/zImage-4.4.13-1-05df79f63527051ea0071350f86faf76-7)
[76456.169919] BTRFS warning (device mmcblk0p1): checksum error at logical 5926912 on dev /dev/mmcblk0p1, sector 11576, root 258, inode 261, offset 1683456, length 4096, links 1 (path: boot/zImage-4.4.13-1-05df79f63527051ea0071350f86faf76-7)
[76460.913934] BTRFS warning (device mmcblk0p1): checksum error at logical 74870784 on dev /dev/mmcblk0p1, sector 146232, root 258, inode 1653, offset 4096, length 2472, links 1 (path: usr/lib/python2.7/lib-dynload/future_builtins.so)
[76460.924260] BTRFS warning (device mmcblk0p1): checksum error at logical 74924032 on dev /dev/mmcblk0p1, sector 146336, root 258, inode 1656, offset 12288, length 2696, links 1 (path: usr/lib/python2.7/lib-dynload/_hashlib.so)
[76460.997395] BTRFS warning (device mmcblk0p1): checksum error at logical 72466432 on dev /dev/mmcblk0p1, sector 141536, root 258, inode 1618, offset 8192, length 4096, links 1 (path: usr/lib/python2.7/lib-dynload/parser.so)
[76461.018206] BTRFS warning (device mmcblk0p1): checksum error at logical 74874880 on dev /dev/mmcblk0p1, sector 146240, root 258, inode 1654, offset 0, length 4096, links 1 (path: usr/lib/python2.7/lib-dynload/_lsprof.so)
[76461.517219] BTRFS warning (device mmcblk0p1): checksum error at logical 74936320 on dev /dev/mmcblk0p1, sector 146360, root 258, inode 1657, offset 8192, length 4096, links 1 (path: usr/lib/python2.7/lib-dynload/itertools.so)
[76461.552098] BTRFS warning (device mmcblk0p1): checksum error at logical 72470528 on dev /dev/mmcblk0p1, sector 141544, root 258, inode 1618, offset 12288, length 4096, links 1 (path: usr/lib/python2.7/lib-dynload/parser.so)
[76461.572967] BTRFS warning (device mmcblk0p1): checksum error at logical 74878976 on dev /dev/mmcblk0p1, sector 146248, root 258, inode 1654, offset 4096, length 4096, links 1 (path: usr/lib/python2.7/lib-dynload/_lsprof.so)
[76461.583317] BTRFS warning (device mmcblk0p1): checksum error at logical 74940416 on dev /dev/mmcblk0p1, sector 146368, root 258, inode 1657, offset 12288, length 4096, links 1 (path: usr/lib/python2.7/lib-dynload/itertools.so)
[76462.087933] BTRFS warning (device mmcblk0p1): checksum error at logical 72474624 on dev /dev/mmcblk0p1, sector 141552, root 258, inode 1618, offset 16384, length 4096, links 1 (path: usr/lib/python2.7/lib-dynload/parser.so)
[76462.108917] BTRFS warning (device mmcblk0p1): checksum error at logical 74883072 on dev /dev/mmcblk0p1, sector 146256, root 258, inode 1654, offset 8192, length 4096, links 1 (path: usr/lib/python2.7/lib-dynload/_lsprof.so)

miska · October 21, 2022, 2:05pm

Factory reset still uses the same filesystem. Try reflashing from the USB drive.

maurer · October 22, 2022, 4:57am

Wow schnapps export and 4 button reset (link above) really did it.
Now I’m on 6.0.1
Kudos @miska and Turris team

system · October 25, 2022, 4:57am

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.