It looks that after upgrading to TOS 5 I do miss free space on my sd card used as root FS.
Space usage:
# df -Ph
Filesystem Size Used Available Capacity Mounted on
/dev/mmcblk0p1 7.3G 6.5G 768.0M 90% /
devtmpfs 512.0K 0 512.0K 0% /dev
tmpfs 512.0K 0 512.0K 0% /sys/fs/cgroup
tmpfs 1009.8M 67.0M 942.8M 7% /tmp
tmpfs 512.0K 0 512.0K 0% /dev
/dev/sda1 9.0T 4.3T 4.2T 51% /mnt/nas
/dev/sdb1 9.0T 5.5T 3.1T 64% /mnt/backup
/dev/sdc1 28.1G 595.1M 26.0G 2% /mnt/sdc1
Snapshots:
# schnapps list
# | Type | Size | Date | Description
------+-----------+-------------+---------------------------+------------------------------------
398 | post | 13.53MiB | 2021-09-22 17:14:22 +0200 | Automatic post-update snapshot
407 | time | 13.48MiB | 2021-11-21 01:05:03 +0100 | Snapshot created by cron
408 | time | 13.48MiB | 2021-11-28 01:05:03 +0100 | Snapshot created by cron
409 | pre | 13.48MiB | 2021-12-01 17:56:45 +0100 | Automatic pre-update snapshot
410 | post | 13.49MiB | 2021-12-01 17:56:54 +0100 | Automatic post-update snapshot
411 | time | 13.49MiB | 2021-12-05 01:05:02 +0100 | Snapshot created by cron
412 | pre | 13.49MiB | 2021-12-08 17:31:00 +0100 | Automatic pre-update snapshot
414 | pre | 13.69MiB | 2021-12-08 20:17:24 +0100 | Automatic pre-update snapshot
415 | post | 16.88MiB | 2021-12-08 20:43:19 +0100 | Automatic post-update snapshot (TurrisOS 5.3.2)
416 | pre | 16.81MiB | 2021-12-09 00:22:50 +0100 | Automatic pre-update snapshot (TurrisOS 5.3.2)
417 | post | 16.77MiB | 2021-12-09 00:22:56 +0100 | Automatic post-update snapshot (TurrisOS 5.3.2)
418 | time | 16.88MiB | 2021-12-12 01:05:02 +0100 | Snapshot created by cron
419 | pre | 17.18MiB | 2021-12-15 13:40:10 +0100 | Automatic pre-update snapshot (TurrisOS 5.3.2)
420 | post | 16.89MiB | 2021-12-15 13:40:33 +0100 | Automatic post-update snapshot (TurrisOS 5.3.2)
421 | time | 748.33MiB | 2021-12-19 01:05:02 +0100 | Snapshot created by cron
And directories size in /
root:~# cd /
root:/# ls / | grep -v mnt | xargs du -sh | sort -h
0 dev
0 overlay
0 proc
0 sys
0 t-output
4.0K boot.scr
4.0K rom
4.0K tisk
4.0K var
8.0K run
20.0K srv
48.0K root
1.3M www
1.9M bin
3.3M boot
3.6M sbin
5.4M etc
11.5M lib
67.0M tmp
307.6M usr
It looks most of the SD card size is used, but I have no clue what causes the usage. Or is that expected and I shall migrate to a larger sd card?
Here is the output:
root@atom:~# du -kx / |sort -nr| head -30
344040 /
315008 /usr
240656 /usr/lib
80072 /usr/lib/python3.7
68800 /usr/lib/python3.7/site-packages
31196 /usr/bin
29236 /usr/lib/python3.7/site-packages/babel
28628 /usr/lib/python3.7/site-packages/babel/locale-data
25916 /usr/share
24036 /usr/lib/python2.7
20812 /usr/lib/opkg
20620 /usr/lib/opkg/info
16380 /usr/sbin
15972 /usr/lib/samba
13828 /usr/lib/python2.7/site-packages
11772 /lib
10992 /usr/lib/perl5/5.28
10992 /usr/lib/perl5
8936 /usr/share/nmap
8844 /lib/modules/4.14.254
8844 /lib/modules
8732 /usr/lib/python3.7/site-packages/reforis_static
7776 /usr/lib/python3.6
6492 /usr/lib/python2.7/site-packages/mx
5980 /usr/lib/python3.7/site-packages/twisted
5604 /usr/lib/python3.7/site-packages/reforis_static/reforis
5504 /etc
5340 /usr/lib/python3.7/site-packages/reforis_static/reforis/js
4804 /usr/share/postgresql
4568 /usr/lib/python3.6/site-packages
root@atom:~# df -Ph
Filesystem Size Used Available Capacity Mounted on
/dev/mmcblk0p1 7.3G 6.6G 758.5M 90% /
devtmpfs 512.0K 0 512.0K 0% /dev
tmpfs 512.0K 0 512.0K 0% /sys/fs/cgroup
tmpfs 1009.8M 72.1M 937.6M 7% /tmp
tmpfs 512.0K 0 512.0K 0% /dev
/dev/sda1 9.0T 4.3T 4.2T 51% /mnt/nas
/dev/sdb1 9.0T 5.5T 3.1T 64% /mnt/backup
/dev/sdc1 28.1G 595.2M 26.0G 2% /mnt/sdc1
this looks like a process has file opened on your root filesystem
lsof -n |grep DELETED
reboot should clear that up but I personally would like to know what process uses such file.
you can also remove some snapshots, I’d start with older time snapshots
1 Like
tac2
January 12, 2022, 8:51am
5
Alternativ you can run
lsof -nP +L1
That will list processes with open files with linkcount less than 1 link
But as mentioned - a reboot should clear that out
Thanks:
root:~# lsof -n |grep DELETED
root:~# lsof -nP +L1
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME
8 4176 root txt REG 0,5 45283 0 961 /memfd:liblxc (deleted)
root:~# df -Ph
Filesystem Size Used Available Capacity Mounted on
/dev/mmcblk0p1 7.3G 6.6G 758.5M 90% /
devtmpfs 512.0K 0 512.0K 0% /dev
tmpfs 512.0K 0 512.0K 0% /sys/fs/cgroup
tmpfs 1009.8M 72.2M 937.5M 7% /tmp
tmpfs 512.0K 0 512.0K 0% /dev
/dev/sda1 9.0T 4.3T 4.2T 51% /mnt/nas
/dev/sdb1 9.0T 5.5T 3.1T 64% /mnt/backup
/dev/sdc1 28.1G 595.2M 26.0G 2% /mnt/sdc1
reboot
root:~# df -Ph
Filesystem Size Used Available Capacity Mounted on
/dev/mmcblk0p1 7.3G 6.6G 758.4M 90% /
devtmpfs 512.0K 0 512.0K 0% /dev
tmpfs 512.0K 0 512.0K 0% /sys/fs/cgroup
tmpfs 1009.8M 65.1M 944.6M 6% /tmp
tmpfs 512.0K 0 512.0K 0% /dev
/dev/sdc1 28.1G 598.9M 26.0G 2% /mnt/sdc1
/dev/sdb1 9.0T 5.5T 3.1T 64% /mnt/backup
/dev/sda1 9.0T 4.3T 4.2T 51% /mnt/nas
I had to made a mistake while looking for the space usage, but can’t spot the fault
tac2
January 12, 2022, 3:50pm
7
Aaaa, you probably had some glitch some time on the external drives.
Try to unmount all drives and run du again(add h flagg for human readable)
1 Like
Good point, I’ll check it.
Sorry for late response…
Mounts dismounted:
root:/mnt# df -Ph
Filesystem Size Used Available Capacity Mounted on
/dev/mmcblk0p1 7.3G 5.9G 1.4G 81% /
devtmpfs 512.0K 0 512.0K 0% /dev
tmpfs 512.0K 0 512.0K 0% /sys/fs/cgroup
tmpfs 1009.8M 67.7M 942.0M 7% /tmp
tmpfs 512.0K 0 512.0K 0% /dev
And:
root:/mnt# du -sh /mnt
1.3M /mnt
Snapshots seem not to occupy too much space:
root:/mnt# schnapps list
# | Type | Size | Date | Description
------+-----------+-------------+---------------------------+------------------------------------
410 | post | 13.49MiB | 2021-12-01 17:56:54 +0100 | Automatic post-update snapshot
412 | pre | 13.49MiB | 2021-12-08 17:31:00 +0100 | Automatic pre-update snapshot
414 | pre | 13.69MiB | 2021-12-08 20:17:24 +0100 | Automatic pre-update snapshot
415 | post | 16.88MiB | 2021-12-08 20:43:19 +0100 | Automatic post-update snapshot (TurrisOS 5.3.2)
416 | pre | 16.81MiB | 2021-12-09 00:22:50 +0100 | Automatic pre-update snapshot (TurrisOS 5.3.2)
417 | post | 16.77MiB | 2021-12-09 00:22:56 +0100 | Automatic post-update snapshot (TurrisOS 5.3.2)
419 | pre | 17.18MiB | 2021-12-15 13:40:10 +0100 | Automatic pre-update snapshot (TurrisOS 5.3.2)
420 | post | 16.89MiB | 2021-12-15 13:40:33 +0100 | Automatic post-update snapshot (TurrisOS 5.3.2)
423 | time | 16.77MiB | 2022-01-02 01:05:02 +0100 | Snapshot created by cron
424 | time | 16.76MiB | 2022-01-09 01:05:02 +0100 | Snapshot created by cron
425 | time | 16.76MiB | 2022-01-16 01:05:02 +0100 | Snapshot created by cron
426 | time | 16.76MiB | 2022-01-23 01:05:02 +0100 | Snapshot created by cron
427 | pre | 33.51MiB | 2022-01-25 20:14:56 +0100 | Automatic pre-update snapshot (TurrisOS 5.3.2)
428 | post | 16.84MiB | 2022-01-25 20:16:47 +0100 | Automatic post-update snapshot (TurrisOS 5.3.4)
429 | time | 16.83MiB | 2022-01-30 01:05:02 +0100 | Snapshot created by cron
And btrfs info:
root:/mnt# btrfs filesystem usage /
Overall:
Device size: 7.28GiB
Device allocated: 7.28GiB
Device unallocated: 0.00B
Device missing: 0.00B
Used: 5.88GiB
Free (estimated): 1.37GiB (min: 1.37GiB)
Data ratio: 1.00
Metadata ratio: 1.00
Global reserve: 20.03MiB (used: 0.00B)
Multiple profiles: no
Data+Metadata,single: Size:7.28GiB, Used:5.88GiB (80.84%)
/dev/mmcblk0p1 7.28GiB
System,single: Size:4.00MiB, Used:4.00KiB (0.10%)
/dev/mmcblk0p1 4.00MiB
Unallocated:
/dev/mmcblk0p1 0.00B
Still no luck. Yeah, there is an option to disassembly the box and inspect SD card in another system, but I’d like to avoid it, if possible.
tac2
January 31, 2022, 9:49pm
10
Humm, strange.
While external drives unmounted.#du -hd 1
#du -hd 2
root:/mnt# du -hd 1
0 ./.snapshots
0 ./tmp
0 ./nas
0 ./backup
4.0K ./sda1
1.3M ./sdc1_rootfs
0 ./sdc1
1.3M .
root:/mnt# du -hd 2
0 ./.snapshots
0 ./tmp
0 ./nas
0 ./backup
4.0K ./sda1
640.0K ./sdc1_rootfs/rrd
400.0K ./sdc1_rootfs/majordomo_db
0 ./sdc1_rootfs/collectd
312.0K ./sdc1_rootfs/var
1.3M ./sdc1_rootfs
0 ./sdc1
1.3M .
tac2
January 31, 2022, 10:08pm
12
No dont run it from /mnt
Yup, sorry for the mistake:
root:/# du -hd 1
0 ./sys
0 ./overlay
3.3M ./boot
1.9M ./bin
20.0K ./srv
1.3M ./www
4.0K ./rom
3.6M ./sbin
67.8M ./tmp
0 ./dev
307.7M ./usr
0 ./proc
11.5M ./lib
1.3M ./mnt
48.0K ./root
5.4M ./etc
8.0K ./run
403.9M .
root:/# du -hd 2
0 ./sys/kernel
0 ./sys/devices
0 ./sys/class
0 ./sys/dev
0 ./sys/firmware
0 ./sys/fs
0 ./sys/bus
0 ./sys/module
0 ./sys/block
0 ./sys
0 ./overlay
3.3M ./boot
1.9M ./bin
12.0K ./srv/dev-detect
20.0K ./srv
1.1M ./www/luci-static
4.0K ./www/webcam
20.0K ./www/cgi-bin
24.0K ./www/webapps-icons
204.0K ./www/turris-webapps
1.3M ./www
4.0K ./rom
3.6M ./sbin
284.0K ./tmp/luci-modulecache
12.0K ./tmp/foris-sessions
0 ./tmp/.uci-foris-controller
136.0K ./tmp/.foris_workdir
4.0K ./tmp/update-state
0 ./tmp/user_notify
18.2M ./tmp/cache
0 ./tmp/suricata
0 ./tmp/pymp-ezt7m9bt
0 ./tmp/pymp-jecv0c6q
8.0K ./tmp/crypto_wrapper_root
20.0M ./tmp/kresd
0 ./tmp/empty
4.0K ./tmp/spool
0 ./tmp/dnsmasq.d
26.1M ./tmp/lib
4.0K ./tmp/hosts
32.0K ./tmp/etc
0 ./tmp/lvm
0 ./tmp/.uci
0 ./tmp/tmp
1.4M ./tmp/log
0 ./tmp/extroot
8.0K ./tmp/sysinfo
8.0K ./tmp/state
1.3M ./tmp/lock
180.0K ./tmp/run
0 ./tmp/shm
67.8M ./tmp
0 ./dev/net
0 ./dev/snd
0 ./dev/mapper
0 ./dev/bus
0 ./dev/pts
0 ./dev
30.5M ./usr/bin
16.0M ./usr/sbin
716.0K ./usr/libexec
235.1M ./usr/lib
25.3M ./usr/share
144.0K ./usr/man
307.7M ./usr
0 ./proc/fs
0 ./proc/bus
0 ./proc/cpu
0 ./proc/irq
0 ./proc/sys
0 ./proc/tty
0 ./proc/scsi
0 ./proc/asound
0 ./proc/driver
0 ./proc/sysvipc
0 ./proc/1
0 ./proc/2
0 ./proc/4
0 ./proc/6
0 ./proc/7
0 ./proc/8
0 ./proc/9
0 ./proc/10
0 ./proc/11
0 ./proc/12
0 ./proc/13
0 ./proc/14
0 ./proc/16
0 ./proc/17
0 ./proc/18
0 ./proc/224
0 ./proc/225
0 ./proc/227
0 ./proc/228
0 ./proc/230
0 ./proc/238
0 ./proc/264
0 ./proc/375
0 ./proc/455
0 ./proc/456
0 ./proc/561
0 ./proc/562
0 ./proc/565
0 ./proc/566
0 ./proc/579
0 ./proc/580
0 ./proc/583
0 ./proc/584
0 ./proc/599
0 ./proc/793
0 ./proc/799
0 ./proc/817
0 ./proc/818
0 ./proc/840
0 ./proc/842
0 ./proc/868
0 ./proc/869
0 ./proc/870
0 ./proc/896
0 ./proc/897
0 ./proc/898
0 ./proc/910
0 ./proc/911
0 ./proc/912
0 ./proc/925
0 ./proc/929
0 ./proc/933
0 ./proc/942
0 ./proc/946
0 ./proc/950
0 ./proc/963
0 ./proc/965
0 ./proc/966
0 ./proc/967
0 ./proc/968
0 ./proc/969
0 ./proc/970
0 ./proc/971
0 ./proc/972
0 ./proc/973
0 ./proc/974
0 ./proc/975
0 ./proc/976
0 ./proc/977
0 ./proc/978
0 ./proc/979
0 ./proc/980
0 ./proc/981
0 ./proc/982
0 ./proc/983
0 ./proc/984
0 ./proc/986
0 ./proc/997
0 ./proc/998
0 ./proc/1001
0 ./proc/1396
0 ./proc/1405
0 ./proc/1462
0 ./proc/1557
0 ./proc/1569
0 ./proc/1588
0 ./proc/1589
0 ./proc/1599
0 ./proc/1600
0 ./proc/1661
0 ./proc/1702
0 ./proc/1752
0 ./proc/1952
0 ./proc/1987
0 ./proc/1988
0 ./proc/2722
0 ./proc/4112
0 ./proc/4186
0 ./proc/4482
0 ./proc/4516
0 ./proc/4568
0 ./proc/4626
0 ./proc/4722
0 ./proc/4724
0 ./proc/4729
0 ./proc/4802
0 ./proc/4903
0 ./proc/5935
0 ./proc/6631
0 ./proc/7252
0 ./proc/7450
0 ./proc/7570
0 ./proc/7786
0 ./proc/7986
0 ./proc/8078
0 ./proc/8080
0 ./proc/8226
0 ./proc/8886
0 ./proc/9007
0 ./proc/9045
0 ./proc/9321
0 ./proc/9378
0 ./proc/9472
0 ./proc/9512
0 ./proc/9699
0 ./proc/9702
0 ./proc/10036
0 ./proc/10069
0 ./proc/10387
0 ./proc/10416
0 ./proc/10480
0 ./proc/10642
0 ./proc/10695
0 ./proc/10710
0 ./proc/10711
0 ./proc/10712
0 ./proc/10713
0 ./proc/11282
0 ./proc/11283
0 ./proc/11346
0 ./proc/11418
0 ./proc/11473
0 ./proc/11480
0 ./proc/11481
0 ./proc/11878
0 ./proc/16732
0 ./proc/17712
0 ./proc/20699
0 ./proc/21054
0 ./proc/24590
0 ./proc/26649
0 ./proc/26651
0 ./proc/27201
0 ./proc/29679
0 ./proc/31120
0 ./proc/32102
0 ./proc/32603
0 ./proc
8.0K ./lib/config
8.6M ./lib/modules
68.0K ./lib/preinit
1.4M ./lib/network
168.0K ./lib/upgrade
256.0K ./lib/firmware
76.0K ./lib/functions
4.0K ./lib/wifi
180.0K ./lib/netifd
11.5M ./lib
0 ./mnt/.snapshots
0 ./mnt/tmp
0 ./mnt/nas
0 ./mnt/backup
4.0K ./mnt/sda1
1.3M ./mnt/sdc1_rootfs
0 ./mnt/sdc1
1.3M ./mnt
16.0K ./root/.ssh
4.0K ./root/.config
48.0K ./root
8.0K ./etc/lxc
352.0K ./etc/rc.d
340.0K ./etc/config
604.0K ./etc/ssh
60.0K ./etc/cron.d
4.0K ./etc/davfs2
52.0K ./etc/lvm
96.0K ./etc/hotplug.d
24.0K ./etc/rc.button
0 ./etc/kresd
0 ./etc/openconnect
0 ./etc/openvpn
88.0K ./etc/updater
4.0K ./etc/ahcp
4.0K ./etc/firewall.d
396.0K ./etc/init.d
28.0K ./etc/gcom
20.0K ./etc/ppp
56.0K ./etc/resolver
40.0K ./etc/samba
132.0K ./etc/lighttpd
4.0K ./etc/vpnc
8.0K ./etc/chatscripts
28.0K ./etc/opkg
88.0K ./etc/modules-boot.d
0 ./etc/udev
0 ./etc/luci-uploads
652.0K ./etc/modules.d
12.0K ./etc/iproute2
1.5M ./etc/ssl
4.0K ./etc/sqm
4.0K ./etc/dhparam
0 ./etc/crontabs
16.0K ./etc/logrotate.d
0 ./etc/syslog-ng.d
20.0K ./etc/mitmkeys
0 ./etc/collectd
4.0K ./etc/schnapps
4.0K ./etc/uci-defaults
8.0K ./etc/sysstat
4.0K ./etc/cagen
80.0K ./etc/suricata
72.0K ./etc/suricata-pakon
44.0K ./etc/mosquitto
0 ./etc/patterndb.d
8.0K ./etc/fosquitto
20.0K ./etc/sentinel
8.0K ./etc/netdata
20.0K ./etc/sysctl.d
4.0K ./etc/knot-resolver
16.0K ./etc/board.d
4.0K ./etc/sysfs.d
4.0K ./etc/dbus-1
4.0K ./etc/avahi
24.0K ./etc/pam.d
36.0K ./etc/security
12.0K ./etc/adblock
68.0K ./etc/pki
12.0K ./etc/ipsec.d
5.4M ./etc
0 ./run/lxc
0 ./run/mount
8.0K ./run/blkid
0 ./run/mdadm
8.0K ./run
403.9M .
tac2
January 31, 2022, 10:20pm
14
And you don have lage files in /
ls -lha
Here is the listing:
root:/# ls -lha
drwxrwxr-x 1 root root 172 Jan 25 20:15 .
drwxrwxr-x 1 root root 172 Jan 25 20:15 ..
-rw------- 1 root root 1.0K Sep 16 23:03 .rnd
drwxr-xr-x 1 root root 1.4K Jan 25 20:16 bin
drwxr-xr-x 1 root root 158 Jan 25 20:15 boot
lrwxrwxrwx 1 root root 13 Jan 25 08:54 boot.scr -> boot/boot.scr
drwxr-xr-x 7 root root 3.1K Jan 31 15:12 dev
drwxr-xr-x 1 root root 2.6K Jan 25 20:15 etc
drwxr-xr-x 1 root root 610 Jan 25 20:15 lib
drwxr-xr-x 1 root root 82 Dec 20 12:08 mnt
drwxr-xr-x 1 root root 0 Sep 5 2016 overlay
dr-xr-xr-x 169 root root 0 Jan 1 1970 proc
drwxrwxr-x 1 root root 8 Jan 25 20:15 rom
drwxr-xr-x 1 root root 92 Dec 19 21:59 root
drwxr-xr-x 1 root root 60 Dec 12 2017 run
drwxr-xr-x 1 root root 1.7K Jan 25 20:16 sbin
drwxr-xr-x 1 root root 36 Dec 8 20:25 srv
dr-xr-xr-x 11 root root 0 Jan 28 03:30 sys
-rw-r--r-- 1 root root 0 Jul 30 2017 t-output
-rw-r--r-- 1 root root 588 Jan 2 2019 tisk
drwxrwxrwt 30 root root 1020 Jan 31 23:24 tmp
drwxr-xr-x 1 root root 50 Dec 8 20:27 usr
lrwxrwxrwx 1 root root 4 Jul 13 1998 var -> /tmp
drwxr-xr-x 1 root root 122 Jan 25 20:15 www
BTW, thanks for your patience, I highly appreciate that!
1 Like
tac2
February 1, 2022, 7:29am
16
clearly there is something I do not understand with du and df as when I check my own MOX the numbers do not add up either.
My number is in MB though so I have not payed attention to it before.
Is there any kind soul here who can shed some light over this, mostly for education purpose.
Is there a chance that df ”used” includes blocks of the sd card marked as worn out/broken?
Sorry, but I’m out of ideas.
EDIT: some kind of fs corruption or other strange state?
tac2
February 2, 2022, 3:55pm
17
needed to readup on df and du,
I would recomend btrfsck ( there is a number of options you can test)
And perhaps it’s wise to have a spare sd card at hand
