Unable to flash firmware - medkit

Hello,
few days ago, after update, filesystem in TO got corrupted and was mounted read only. First I thought that there is HW problem on eMMC, but later I was ale to fix that by booting to rescue mode and run some btfrs checks/fix.

Aftrer reboot everything is working fine, I can create snapshots, filesystem is writeable, factory reset to first firmware or “going back” by schnapshots is working fine.

BUT.

If I try to flash firmware using medkit I always end in all leds reds, like it is flashing, but this takes very long - hours.

I tested it with older TO3 and also with TO4. USB flash disk is readable, formated to FAT32, image also looks fine. I tried with USB plugged to front and also rear slot, same result.

Any hints what might be reason? What to check etc?

Could you access serial port to get output of the flashing proccess?

Here are several last lines:

modprobe scsi_mod
MODE=4
Error: Invalid mode 4. Exit to shell.


BusyBox v1.23.2 (2016-08-18 20:38:45 CEST) built-in shell (ash)

/ # [    6.260554] usb 2-1: new high-speed USB device number 2 using xhci-hcd
[    6.467616] usb-storage 2-1:1.0: USB Mass Storage device detected
[    6.473883] scsi host2: usb-storage 2-1:1.0
[    7.472155] scsi 2:0:0:0: Direct-Access     Generic  Flash Disk       8.07 PQ: 0 ANSI: 4
[    7.481066] sd 2:0:0:0: [sdb] 31334400 512-byte logical blocks: (16.0 GB/14.9 GiB)
[    7.489291] sd 2:0:0:0: [sdb] Write Protect is off
[    7.494765] sd 2:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[    7.528572]  sdb: sdb1 sdb2
[    7.533314] sd 2:0:0:0: [sdb] Attached SCSI removable disk

And as it is now it will keep for hours.

Here is output from btrfs check

/ # btrfs check /dev/mmcblk0p1
Warning, could not drop caches
Warning, could not drop caches
Checking filesystem on /dev/mmcblk0p1
UUID: 79af27ae-8fc3-4262-97c0-500d46da169c
checking extents
ref mismatch on [512290816 32768] extent item 6, found 5
Incorrect local backref count on 512290816 root 514 owner 182014 offset 0 found 0 wanted 1 back 0x2296050
Backref disk bytenr does not match extent record, bytenr=512290816, ref bytenr=65536
Backref bytes do not match extent backref, bytenr=512290816, ref bytes=32768, backref bytes=4096
backpointer mismatch on [512290816 32768]
ref mismatch on [512323584 45056] extent item 6, found 5
Incorrect local backref count on 512323584 root 514 owner 182015 offset 0 found 0 wanted 1 back 0x22962e0
Backref disk bytenr does not match extent record, bytenr=512323584, ref bytenr=0
Backref bytes do not match extent backref, bytenr=512323584, ref bytes=45056, backref bytes=4096
backpointer mismatch on [512323584 45056]
ref mismatch on [512368640 40960] extent item 6, found 5
Incorrect local backref count on 512368640 root 514 owner 182017 offset 0 found 0 wanted 1 back 0x2298780
Backref disk bytenr does not match extent record, bytenr=512368640, ref bytenr=742759686981287936
Backref bytes do not match extent backref, bytenr=512368640, ref bytes=40960, backref bytes=4096
backpointer mismatch on [512368640 40960]
ref mismatch on [512610304 32768] extent item 6, found 5
Incorrect local backref count on 512610304 root 514 owner 182019 offset 0 found 0 wanted 1 back 0x2825c90
Backref disk bytenr does not match extent record, bytenr=512610304, ref bytenr=0
Backref bytes do not match extent backref, bytenr=512610304, ref bytes=32768, backref bytes=4096
backpointer mismatch on [512610304 32768]
ref mismatch on [533516288 12288] extent item 6, found 5
Incorrect local backref count on 533516288 root 514 owner 182016 offset 0 found 0 wanted 1 back 0x2e5a4d0
Backref disk bytenr does not match extent record, bytenr=533516288, ref bytenr=0
Backref bytes do not match extent backref, bytenr=533516288, ref bytes=12288, backref bytes=4096
backpointer mismatch on [533516288 12288]
ref mismatch on [533561344 12288] extent item 6, found 5
Incorrect local backref count on 533561344 root 514 owner 182018 offset 0 found 0 wanted 1 back 0x2e5ad10
Backref disk bytenr does not match extent record, bytenr=533561344, ref bytenr=14863849095159611392
Backref bytes do not match extent backref, bytenr=533561344, ref bytes=12288, backref bytes=4096
backpointer mismatch on [533561344 12288]
ref mismatch on [798326784 4096] extent item 2, found 1
Incorrect local backref count on 798326784 root 490 owner 170177 offset 0 found 0 wanted 1 back 0x14ce7a0
Backref disk bytenr does not match extent record, bytenr=798326784, ref bytenr=12826251738751919216
backpointer mismatch on [798326784 4096]
ref mismatch on [798396416 4096] extent item 2, found 1
Incorrect local backref count on 798396416 root 490 owner 170179 offset 0 found 0 wanted 1 back 0x2f49420
Backref disk bytenr does not match extent record, bytenr=798396416, ref bytenr=30786326348292
backpointer mismatch on [798396416 4096]
ref mismatch on [798556160 4096] extent item 2, found 1
Incorrect local backref count on 798556160 root 490 owner 170181 offset 0 found 0 wanted 1 back 0x2f49870
Backref disk bytenr does not match extent record, bytenr=798556160, ref bytenr=0
backpointer mismatch on [798556160 4096]
checking free space cache
checking fs roots
root 514 inode 182018 errors 4000, orphan file extent
The following data extent is lost in tree 514:
        inode: 182018, offset:0, disk_bytenr: 533561344, disk_len: 12288
found 1385763627 bytes used err is 1
total csum bytes: 1068884
total tree bytes: 314916864
total fs tree bytes: 291766272
total extent tree bytes: 21000192
btree space waste bytes: 89610909
file data blocks allocated: 5009432576
 referenced 4903497728
Warning, could not drop caches

from the rescue shell try fdisk /dev/mmcblk0 and see whether it produces any error.

Perhaps also see Controller never released inhibit bit(s) since the output/symptoms seem similar.

No errors from fdisk

/ # fdisk /dev/mmcblk0

Welcome to fdisk (util-linux 2.25.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.


Command (m for help): p
Disk /dev/mmcblk0: 7.3 GiB, 7818182656 bytes, 15269888 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x7c0b13ff

Device         Boot Start      End  Sectors  Size Id Type
/dev/mmcblk0p1       2048 15269887 15267840  7.3G 83 Linux

I saw that linked post, but it is a bit different for me - I can move between snapshots on disk, I can return to oldest one from 2017 or newest one from today, everything is working fine, router works as expected.

I also tried to retirn to that old one from 2017 and then update to latest TOS, again, no problem.

But flashing from USB does not work.

USB disk looks also OK if I am not wrong:

Device     Boot Start      End  Sectors Size Id Type
/dev/sdb1        2048 31332351 31330304  15G  c W95 FAT32 (LBA)

The output from btrfs check exhibits some errors with the BTRFS, not sure whether and eventually to which extent that comes into play.

Else you could try:

  • to format the USB drive/partition on the router with a different filesystem, mount the device and then fetch the medkit with wget directly and see how that works out.
  • if available try another USB device

Also the dev sdb showing two partitions - sdb1 & sdb2, not sure whether that works with medkit or whether it gets confused, try a single drive/partition instead.

Last but not least check that the correct medkit is being used and not eventually somehow/accidentally got mixed up - Turris, Omnia, Mox.

If your eMMC is not dying, then you may try rebalancing it.
However if it is in fact dying, that might be a very bad advice. Back up everything, now.

What exactly means by “rebalancing”?

Anyway, if it is dying its better if it die quickly:-)

There is nothing to backup, I am not using it currently as production one so I have free hands.

OK, so it means problem was in USB drive. Even after formatting it directly in Omnia and downloading image to it it did not work.

Then I found one very old 256 MB flash drive and its working like charm. Funny is that the not working USB drive was the one I got from NIC.CZ :slight_smile: