microSD card and corrupted FS

Something completely broken, tried pkgupdate several times and everytime got error on different package like this

INFO:Downloading packages
ERROR:(https://repo.turris.cz/hbs/turris1x/packages/core/libip4tc_1.8.3-1_powerpc_8540.ipk) Opening output file “/usr/share/updater/download/libip4tc-1.8.3-1.ipk” failed: No file descriptors available
line not found
line not found
line not found
WARN:Crash report of stack trace dump failed.
DIE:
[string “updater”]:114: Error while registering for download: https://repo.turris.cz/hbs/turris1x/packages/core/libip4tc_1.8.3-1_powerpc_8540.ipk: Unable to open output file for writing
Aborted

Reinstalled reforris downloaded diagnostics and sending to tech support by email as requested.

My problem with upgrade to 5.1.0 persist trying pkgupdate with disabled transmission and stopped LXC containers after reboot with no luck, pkgupdate fail on random package advice from technical to disable transmission useless so far.

We don’t have a magic ball and if it was useless, would you please wait? We received the response from you today at 4:42 that the proposed solution didn’t help you. It requires patience, because our support is not 24/7. I think our technical support including the developer team will find what’s wrong in your case and reply to you as soon as possible.

In the meantime, I think from what I know you might be running out of Inodes. I don’t know, which microSD card you are using on your router, but I suggest to run defragmentation to see if it is better.

You are right. I focused on BTRFS filesystem and there were some errors hence i run

  1. fstrim -v /
  2. btrfs check --repair --force /dev/mmcblk0p2
  3. btrfs quota rescan -w /

With this results

enabling repair mode
Opening filesystem to check...
WARNING: filesystem mounted, continuing because of --force
Checking filesystem on /dev/mmcblk0p2
UUID: 0f276d8f-aa1e-4011-bef7-3ea279374bf2
[1/7] checking root items
Fixed 0 roots.
[2/7] checking extents
No device size related problem found
[3/7] checking free space cache
cache and super generation don't match, space cache will be invalidated
[4/7] checking fs roots
root 1083 inode 6903081 errors 1000, some csum missing
root 1083 inode 6907323 errors 1000, some csum missing
root 1083 inode 6907392 errors 1000, some csum missing
root 1083 inode 6907681 errors 1000, some csum missing
root 1083 inode 11704913 errors 1000, some csum missing
root 1084 inode 345507 errors 1000, some csum missing
root 1084 inode 629857 errors 1000, some csum missing
root 1084 inode 634099 errors 1000, some csum missing
root 1084 inode 634168 errors 1000, some csum missing
root 1084 inode 634457 errors 1000, some csum missing
root 1099 inode 345507 errors 1000, some csum missing
root 1099 inode 629857 errors 1000, some csum missing
root 1099 inode 634099 errors 1000, some csum missing
root 1099 inode 634168 errors 1000, some csum missing
root 1099 inode 634457 errors 1000, some csum missing
root 1100 inode 345507 errors 1000, some csum missing
root 1100 inode 629857 errors 1000, some csum missing
root 1100 inode 634099 errors 1000, some csum missing
root 1100 inode 634168 errors 1000, some csum missing
root 1100 inode 634457 errors 1000, some csum missing
ERROR: errors found in fs roots
found 16692678656 bytes used, error(s) found
total csum bytes: 15120156
total tree bytes: 937656320
total fs tree bytes: 871170048
total extent tree bytes: 44515328
btree space waste bytes: 189722841
file data blocks allocated: 43554189312
 referenced 25683087360
quota rescan started

then i rebooted and df show there should be enough space

turris ~ # df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mmcblk0p2        31157248  15783560  14218088  53% /
devtmpfs                   512         0       512   0% /dev
tmpfs                      512         0       512   0% /sys/fs/cgroup
tmpfs                  1034464     55296    979168   5% /tmp
tmpfs                      512         0       512   0% /dev

turris ~ # btrfs fi us /
Overall:
    Device size:                  29.71GiB
    Device allocated:             20.04GiB
    Device unallocated:            9.67GiB
    Device missing:                  0.00B
    Used:                         14.99GiB
    Free (estimated):             13.56GiB      (min: 13.56GiB)
    Data ratio:                       1.00
    Metadata ratio:                   1.00
    Global reserve:               62.41MiB      (used: 0.00B)

Data,single: Size:18.01GiB, Used:14.12GiB (78.42%)
   /dev/mmcblk0p2         18.01GiB

Metadata,single: Size:2.00GiB, Used:890.06MiB (43.46%)
   /dev/mmcblk0p2          2.00GiB

System,single: Size:32.00MiB, Used:16.00KiB (0.05%)
   /dev/mmcblk0p2         32.00MiB

Unallocated:
   /dev/mmcblk0p2          9.67GiB
after reboot i tried pkgupdate again with same error on clean BTFS 


INFO:Queue removal of luci-app-ahcp
INFO:Queue removal of ahcpd
Press return to continue, CTRL+C to abort

INFO:Downloading packages
ERROR:(https://repo.turris.cz/hbs/turris1x/packages/packages/libgst1check_1.16.2-1_powerpc_8540.ipk) Opening output file "/usr/share/updater/download/libgst1check-1.16.2-1.ipk" failed: No file descriptors available
line not found
line not found
line not found
WARN:Crash report of stack trace dump failed.
DIE:
[string "updater"]:114: Error while registering for download: https://repo.turris.cz/hbs/turris1x/packages/packages/libgst1check_1.16.2-1_powerpc_8540.ipk: Unable to open output file for writing
Aborted

Thank you for pointing me to BTRFS but it is probably not the cause of problem. Any idea now ?

@Pepe Today I refreshed my SDCARD with image from July of TOS 5.0.3 and TOS 3.11.17 to make sure BTFS filesystem is clear. I made sure to check it is clear by btrfs check --repair --force /dev/mmcblk0p2 and get completely no errors. Then I upgraded snapshot of TO 3.11.17 to TOS 3.11.19 without any problem. After that tried update of TOS 5.0.3 to TOS 5.1.0 with same error as reported before with same error results as happened on TOS 5.0.4. Sending diagnostics to tech support by email.

INFO:Queue removal of python3-hyperlink
INFO:Queue removal of luci-i18n-ahcp-en
INFO:Queue removal of luci-app-ahcp
INFO:Queue removal of ahcpd
Press return to continue, CTRL+C to abort

INFO:Downloading packages
ERROR:(https://repo.turris.cz/hbs/turris1x/packages/base/libuci_2019-09-01-415f9e48-3_powerpc_8540.ipk) Opening output file “/usr/share/updater/download/libuci-2019-09-01-415f9e48-3.ipk” failed: No file descriptors available
line not found
line not found
line not found
WARN:Crash report of stack trace dump failed.
DIE:
[string “updater”]:114: Error while registering for download: https://repo.turris.cz/hbs/turris1x/packages/base/libuci_2019-09-01-415f9e48-3_powerpc_8540.ipk: Unable to open output file for writing
Aborted