Space usage on the root fs

Software SW help
#1

It looks that after upgrading to TOS 5 I do miss free space on my sd card used as root FS.

Space usage:

# df -Ph
Filesystem                Size      Used Available Capacity Mounted on
/dev/mmcblk0p1            7.3G      6.5G    768.0M  90% /
devtmpfs                512.0K         0    512.0K   0% /dev
tmpfs                   512.0K         0    512.0K   0% /sys/fs/cgroup
tmpfs                  1009.8M     67.0M    942.8M   7% /tmp
tmpfs                   512.0K         0    512.0K   0% /dev
/dev/sda1                 9.0T      4.3T      4.2T  51% /mnt/nas
/dev/sdb1                 9.0T      5.5T      3.1T  64% /mnt/backup
/dev/sdc1                28.1G    595.1M     26.0G   2% /mnt/sdc1

Snapshots:

# schnapps list
    # | Type      | Size        | Date                      | Description
------+-----------+-------------+---------------------------+------------------------------------
  398 | post      |    13.53MiB | 2021-09-22 17:14:22 +0200 | Automatic post-update snapshot
  407 | time      |    13.48MiB | 2021-11-21 01:05:03 +0100 | Snapshot created by cron
  408 | time      |    13.48MiB | 2021-11-28 01:05:03 +0100 | Snapshot created by cron
  409 | pre       |    13.48MiB | 2021-12-01 17:56:45 +0100 | Automatic pre-update snapshot
  410 | post      |    13.49MiB | 2021-12-01 17:56:54 +0100 | Automatic post-update snapshot
  411 | time      |    13.49MiB | 2021-12-05 01:05:02 +0100 | Snapshot created by cron
  412 | pre       |    13.49MiB | 2021-12-08 17:31:00 +0100 | Automatic pre-update snapshot
  414 | pre       |    13.69MiB | 2021-12-08 20:17:24 +0100 | Automatic pre-update snapshot
  415 | post      |    16.88MiB | 2021-12-08 20:43:19 +0100 | Automatic post-update snapshot (TurrisOS 5.3.2)
  416 | pre       |    16.81MiB | 2021-12-09 00:22:50 +0100 | Automatic pre-update snapshot (TurrisOS 5.3.2)
  417 | post      |    16.77MiB | 2021-12-09 00:22:56 +0100 | Automatic post-update snapshot (TurrisOS 5.3.2)
  418 | time      |    16.88MiB | 2021-12-12 01:05:02 +0100 | Snapshot created by cron
  419 | pre       |    17.18MiB | 2021-12-15 13:40:10 +0100 | Automatic pre-update snapshot (TurrisOS 5.3.2)
  420 | post      |    16.89MiB | 2021-12-15 13:40:33 +0100 | Automatic post-update snapshot (TurrisOS 5.3.2)
  421 | time      |   748.33MiB | 2021-12-19 01:05:02 +0100 | Snapshot created by cron

And directories size in /

root:~# cd /
root:/# ls / | grep -v mnt | xargs du -sh | sort -h
0	dev
0	overlay
0	proc
0	sys
0	t-output
4.0K	boot.scr
4.0K	rom
4.0K	tisk
4.0K	var
8.0K	run
20.0K	srv
48.0K	root
1.3M	www
1.9M	bin
3.3M	boot
3.6M	sbin
5.4M	etc
11.5M	lib
67.0M	tmp
307.6M	usr

It looks most of the SD card size is used, but I have no clue what causes the usage. Or is that expected and I shall migrate to a larger sd card?
Thanks, ales

#2

try something like:

du -kx / |sort -nr|less

#3

Here is the output:

root@atom:~# du -kx / |sort -nr| head -30
344040	/
315008	/usr
240656	/usr/lib
80072	/usr/lib/python3.7
68800	/usr/lib/python3.7/site-packages
31196	/usr/bin
29236	/usr/lib/python3.7/site-packages/babel
28628	/usr/lib/python3.7/site-packages/babel/locale-data
25916	/usr/share
24036	/usr/lib/python2.7
20812	/usr/lib/opkg
20620	/usr/lib/opkg/info
16380	/usr/sbin
15972	/usr/lib/samba
13828	/usr/lib/python2.7/site-packages
11772	/lib
10992	/usr/lib/perl5/5.28
10992	/usr/lib/perl5
8936	/usr/share/nmap
8844	/lib/modules/4.14.254
8844	/lib/modules
8732	/usr/lib/python3.7/site-packages/reforis_static
7776	/usr/lib/python3.6
6492	/usr/lib/python2.7/site-packages/mx
5980	/usr/lib/python3.7/site-packages/twisted
5604	/usr/lib/python3.7/site-packages/reforis_static/reforis
5504	/etc
5340	/usr/lib/python3.7/site-packages/reforis_static/reforis/js
4804	/usr/share/postgresql
4568	/usr/lib/python3.6/site-packages

root@atom:~# df -Ph
Filesystem                Size      Used Available Capacity Mounted on
/dev/mmcblk0p1            7.3G      6.6G    758.5M  90% /
devtmpfs                512.0K         0    512.0K   0% /dev
tmpfs                   512.0K         0    512.0K   0% /sys/fs/cgroup
tmpfs                  1009.8M     72.1M    937.6M   7% /tmp
tmpfs                   512.0K         0    512.0K   0% /dev
/dev/sda1                 9.0T      4.3T      4.2T  51% /mnt/nas
/dev/sdb1                 9.0T      5.5T      3.1T  64% /mnt/backup
/dev/sdc1                28.1G    595.2M     26.0G   2% /mnt/sdc1
#4

this looks like a process has file opened on your root filesystem

lsof -n |grep DELETED

reboot should clear that up but I personally would like to know what process uses such file.

you can also remove some snapshots, I’d start with older time snapshots

1 Like
#5

Alternativ you can run

lsof -nP +L1

That will list processes with open files with linkcount less than 1 link
I.e. open a deleted file

But as mentioned - a reboot should clear that out

#6

Thanks:

root:~# lsof -n |grep DELETED
root:~# lsof -nP +L1
COMMAND  PID USER  FD   TYPE DEVICE SIZE/OFF NLINK NODE NAME
8       4176 root txt    REG    0,5    45283     0  961 /memfd:liblxc (deleted)

root:~# df -Ph
Filesystem                Size      Used Available Capacity Mounted on
/dev/mmcblk0p1            7.3G      6.6G    758.5M  90% /
devtmpfs                512.0K         0    512.0K   0% /dev
tmpfs                   512.0K         0    512.0K   0% /sys/fs/cgroup
tmpfs                  1009.8M     72.2M    937.5M   7% /tmp
tmpfs                   512.0K         0    512.0K   0% /dev
/dev/sda1                 9.0T      4.3T      4.2T  51% /mnt/nas
/dev/sdb1                 9.0T      5.5T      3.1T  64% /mnt/backup
/dev/sdc1                28.1G    595.2M     26.0G   2% /mnt/sdc1

reboot

root:~# df -Ph
Filesystem                Size      Used Available Capacity Mounted on
/dev/mmcblk0p1            7.3G      6.6G    758.4M  90% /
devtmpfs                512.0K         0    512.0K   0% /dev
tmpfs                   512.0K         0    512.0K   0% /sys/fs/cgroup
tmpfs                  1009.8M     65.1M    944.6M   6% /tmp
tmpfs                   512.0K         0    512.0K   0% /dev
/dev/sdc1                28.1G    598.9M     26.0G   2% /mnt/sdc1
/dev/sdb1                 9.0T      5.5T      3.1T  64% /mnt/backup
/dev/sda1                 9.0T      4.3T      4.2T  51% /mnt/nas

I had to made a mistake while looking for the space usage, but can’t spot the fault :frowning:

#7

Aaaa, you probably had some glitch some time on the external drives.
That way you have ended up written data to the mountpoints(then pointing at the sd card)

Try to unmount all drives and run du again(add h flagg for human readable)

1 Like
#8

Good point, I’ll check it.

#9

Sorry for late response…

Mounts dismounted:

root:/mnt# df -Ph
Filesystem                Size      Used Available Capacity Mounted on
/dev/mmcblk0p1            7.3G      5.9G      1.4G  81% /
devtmpfs                512.0K         0    512.0K   0% /dev
tmpfs                   512.0K         0    512.0K   0% /sys/fs/cgroup
tmpfs                  1009.8M     67.7M    942.0M   7% /tmp
tmpfs                   512.0K         0    512.0K   0% /dev

And:

root:/mnt# du -sh /mnt
1.3M	/mnt

Snapshots seem not to occupy too much space:

root:/mnt# schnapps list
    # | Type      | Size        | Date                      | Description
------+-----------+-------------+---------------------------+------------------------------------
  410 | post      |    13.49MiB | 2021-12-01 17:56:54 +0100 | Automatic post-update snapshot
  412 | pre       |    13.49MiB | 2021-12-08 17:31:00 +0100 | Automatic pre-update snapshot
  414 | pre       |    13.69MiB | 2021-12-08 20:17:24 +0100 | Automatic pre-update snapshot
  415 | post      |    16.88MiB | 2021-12-08 20:43:19 +0100 | Automatic post-update snapshot (TurrisOS 5.3.2)
  416 | pre       |    16.81MiB | 2021-12-09 00:22:50 +0100 | Automatic pre-update snapshot (TurrisOS 5.3.2)
  417 | post      |    16.77MiB | 2021-12-09 00:22:56 +0100 | Automatic post-update snapshot (TurrisOS 5.3.2)
  419 | pre       |    17.18MiB | 2021-12-15 13:40:10 +0100 | Automatic pre-update snapshot (TurrisOS 5.3.2)
  420 | post      |    16.89MiB | 2021-12-15 13:40:33 +0100 | Automatic post-update snapshot (TurrisOS 5.3.2)
  423 | time      |    16.77MiB | 2022-01-02 01:05:02 +0100 | Snapshot created by cron
  424 | time      |    16.76MiB | 2022-01-09 01:05:02 +0100 | Snapshot created by cron
  425 | time      |    16.76MiB | 2022-01-16 01:05:02 +0100 | Snapshot created by cron
  426 | time      |    16.76MiB | 2022-01-23 01:05:02 +0100 | Snapshot created by cron
  427 | pre       |    33.51MiB | 2022-01-25 20:14:56 +0100 | Automatic pre-update snapshot (TurrisOS 5.3.2)
  428 | post      |    16.84MiB | 2022-01-25 20:16:47 +0100 | Automatic post-update snapshot (TurrisOS 5.3.4)
  429 | time      |    16.83MiB | 2022-01-30 01:05:02 +0100 | Snapshot created by cron

And btrfs info:

 root:/mnt# btrfs filesystem usage /
Overall:
    Device size:		   7.28GiB
    Device allocated:		   7.28GiB
    Device unallocated:		     0.00B
    Device missing:		     0.00B
    Used:			   5.88GiB
    Free (estimated):		   1.37GiB	(min: 1.37GiB)
    Data ratio:			      1.00
    Metadata ratio:		      1.00
    Global reserve:		  20.03MiB	(used: 0.00B)
    Multiple profiles:		        no

Data+Metadata,single: Size:7.28GiB, Used:5.88GiB (80.84%)
   /dev/mmcblk0p1	   7.28GiB

System,single: Size:4.00MiB, Used:4.00KiB (0.10%)
   /dev/mmcblk0p1	   4.00MiB

Unallocated:
   /dev/mmcblk0p1	     0.00B

Still no luck. Yeah, there is an option to disassembly the box and inspect SD card in another system, but I’d like to avoid it, if possible.

#10

Humm, strange.

While external drives unmounted.
#du -hd 1

#du -hd 2

#11 
root:/mnt# du -hd 1
0	./.snapshots
0	./tmp
0	./nas
0	./backup
4.0K	./sda1
1.3M	./sdc1_rootfs
0	./sdc1
1.3M	.
root:/mnt# du -hd 2
0	./.snapshots
0	./tmp
0	./nas
0	./backup
4.0K	./sda1
640.0K	./sdc1_rootfs/rrd
400.0K	./sdc1_rootfs/majordomo_db
0	./sdc1_rootfs/collectd
312.0K	./sdc1_rootfs/var
1.3M	./sdc1_rootfs
0	./sdc1
1.3M	.
#12

No dont run it from /mnt
Runit on /

#13

Yup, sorry for the mistake:

root:/# du -hd 1
0	./sys
0	./overlay
3.3M	./boot
1.9M	./bin
20.0K	./srv
1.3M	./www
4.0K	./rom
3.6M	./sbin
67.8M	./tmp
0	./dev
307.7M	./usr
0	./proc
11.5M	./lib
1.3M	./mnt
48.0K	./root
5.4M	./etc
8.0K	./run
403.9M	.


root:/# du -hd 2
0	./sys/kernel
0	./sys/devices
0	./sys/class
0	./sys/dev
0	./sys/firmware
0	./sys/fs
0	./sys/bus
0	./sys/module
0	./sys/block
0	./sys
0	./overlay
3.3M	./boot
1.9M	./bin
12.0K	./srv/dev-detect
20.0K	./srv
1.1M	./www/luci-static
4.0K	./www/webcam
20.0K	./www/cgi-bin
24.0K	./www/webapps-icons
204.0K	./www/turris-webapps
1.3M	./www
4.0K	./rom
3.6M	./sbin
284.0K	./tmp/luci-modulecache
12.0K	./tmp/foris-sessions
0	./tmp/.uci-foris-controller
136.0K	./tmp/.foris_workdir
4.0K	./tmp/update-state
0	./tmp/user_notify
18.2M	./tmp/cache
0	./tmp/suricata
0	./tmp/pymp-ezt7m9bt
0	./tmp/pymp-jecv0c6q
8.0K	./tmp/crypto_wrapper_root
20.0M	./tmp/kresd
0	./tmp/empty
4.0K	./tmp/spool
0	./tmp/dnsmasq.d
26.1M	./tmp/lib
4.0K	./tmp/hosts
32.0K	./tmp/etc
0	./tmp/lvm
0	./tmp/.uci
0	./tmp/tmp
1.4M	./tmp/log
0	./tmp/extroot
8.0K	./tmp/sysinfo
8.0K	./tmp/state
1.3M	./tmp/lock
180.0K	./tmp/run
0	./tmp/shm
67.8M	./tmp
0	./dev/net
0	./dev/snd
0	./dev/mapper
0	./dev/bus
0	./dev/pts
0	./dev
30.5M	./usr/bin
16.0M	./usr/sbin
716.0K	./usr/libexec
235.1M	./usr/lib
25.3M	./usr/share
144.0K	./usr/man
307.7M	./usr
0	./proc/fs
0	./proc/bus
0	./proc/cpu
0	./proc/irq
0	./proc/sys
0	./proc/tty
0	./proc/scsi
0	./proc/asound
0	./proc/driver
0	./proc/sysvipc
0	./proc/1
0	./proc/2
0	./proc/4
0	./proc/6
0	./proc/7
0	./proc/8
0	./proc/9
0	./proc/10
0	./proc/11
0	./proc/12
0	./proc/13
0	./proc/14
0	./proc/16
0	./proc/17
0	./proc/18
0	./proc/224
0	./proc/225
0	./proc/227
0	./proc/228
0	./proc/230
0	./proc/238
0	./proc/264
0	./proc/375
0	./proc/455
0	./proc/456
0	./proc/561
0	./proc/562
0	./proc/565
0	./proc/566
0	./proc/579
0	./proc/580
0	./proc/583
0	./proc/584
0	./proc/599
0	./proc/793
0	./proc/799
0	./proc/817
0	./proc/818
0	./proc/840
0	./proc/842
0	./proc/868
0	./proc/869
0	./proc/870
0	./proc/896
0	./proc/897
0	./proc/898
0	./proc/910
0	./proc/911
0	./proc/912
0	./proc/925
0	./proc/929
0	./proc/933
0	./proc/942
0	./proc/946
0	./proc/950
0	./proc/963
0	./proc/965
0	./proc/966
0	./proc/967
0	./proc/968
0	./proc/969
0	./proc/970
0	./proc/971
0	./proc/972
0	./proc/973
0	./proc/974
0	./proc/975
0	./proc/976
0	./proc/977
0	./proc/978
0	./proc/979
0	./proc/980
0	./proc/981
0	./proc/982
0	./proc/983
0	./proc/984
0	./proc/986
0	./proc/997
0	./proc/998
0	./proc/1001
0	./proc/1396
0	./proc/1405
0	./proc/1462
0	./proc/1557
0	./proc/1569
0	./proc/1588
0	./proc/1589
0	./proc/1599
0	./proc/1600
0	./proc/1661
0	./proc/1702
0	./proc/1752
0	./proc/1952
0	./proc/1987
0	./proc/1988
0	./proc/2722
0	./proc/4112
0	./proc/4186
0	./proc/4482
0	./proc/4516
0	./proc/4568
0	./proc/4626
0	./proc/4722
0	./proc/4724
0	./proc/4729
0	./proc/4802
0	./proc/4903
0	./proc/5935
0	./proc/6631
0	./proc/7252
0	./proc/7450
0	./proc/7570
0	./proc/7786
0	./proc/7986
0	./proc/8078
0	./proc/8080
0	./proc/8226
0	./proc/8886
0	./proc/9007
0	./proc/9045
0	./proc/9321
0	./proc/9378
0	./proc/9472
0	./proc/9512
0	./proc/9699
0	./proc/9702
0	./proc/10036
0	./proc/10069
0	./proc/10387
0	./proc/10416
0	./proc/10480
0	./proc/10642
0	./proc/10695
0	./proc/10710
0	./proc/10711
0	./proc/10712
0	./proc/10713
0	./proc/11282
0	./proc/11283
0	./proc/11346
0	./proc/11418
0	./proc/11473
0	./proc/11480
0	./proc/11481
0	./proc/11878
0	./proc/16732
0	./proc/17712
0	./proc/20699
0	./proc/21054
0	./proc/24590
0	./proc/26649
0	./proc/26651
0	./proc/27201
0	./proc/29679
0	./proc/31120
0	./proc/32102
0	./proc/32603
0	./proc
8.0K	./lib/config
8.6M	./lib/modules
68.0K	./lib/preinit
1.4M	./lib/network
168.0K	./lib/upgrade
256.0K	./lib/firmware
76.0K	./lib/functions
4.0K	./lib/wifi
180.0K	./lib/netifd
11.5M	./lib
0	./mnt/.snapshots
0	./mnt/tmp
0	./mnt/nas
0	./mnt/backup
4.0K	./mnt/sda1
1.3M	./mnt/sdc1_rootfs
0	./mnt/sdc1
1.3M	./mnt
16.0K	./root/.ssh
4.0K	./root/.config
48.0K	./root
8.0K	./etc/lxc
352.0K	./etc/rc.d
340.0K	./etc/config
604.0K	./etc/ssh
60.0K	./etc/cron.d
4.0K	./etc/davfs2
52.0K	./etc/lvm
96.0K	./etc/hotplug.d
24.0K	./etc/rc.button
0	./etc/kresd
0	./etc/openconnect
0	./etc/openvpn
88.0K	./etc/updater
4.0K	./etc/ahcp
4.0K	./etc/firewall.d
396.0K	./etc/init.d
28.0K	./etc/gcom
20.0K	./etc/ppp
56.0K	./etc/resolver
40.0K	./etc/samba
132.0K	./etc/lighttpd
4.0K	./etc/vpnc
8.0K	./etc/chatscripts
28.0K	./etc/opkg
88.0K	./etc/modules-boot.d
0	./etc/udev
0	./etc/luci-uploads
652.0K	./etc/modules.d
12.0K	./etc/iproute2
1.5M	./etc/ssl
4.0K	./etc/sqm
4.0K	./etc/dhparam
0	./etc/crontabs
16.0K	./etc/logrotate.d
0	./etc/syslog-ng.d
20.0K	./etc/mitmkeys
0	./etc/collectd
4.0K	./etc/schnapps
4.0K	./etc/uci-defaults
8.0K	./etc/sysstat
4.0K	./etc/cagen
80.0K	./etc/suricata
72.0K	./etc/suricata-pakon
44.0K	./etc/mosquitto
0	./etc/patterndb.d
8.0K	./etc/fosquitto
20.0K	./etc/sentinel
8.0K	./etc/netdata
20.0K	./etc/sysctl.d
4.0K	./etc/knot-resolver
16.0K	./etc/board.d
4.0K	./etc/sysfs.d
4.0K	./etc/dbus-1
4.0K	./etc/avahi
24.0K	./etc/pam.d
36.0K	./etc/security
12.0K	./etc/adblock
68.0K	./etc/pki
12.0K	./etc/ipsec.d
5.4M	./etc
0	./run/lxc
0	./run/mount
8.0K	./run/blkid
0	./run/mdadm
8.0K	./run
403.9M	.
#14

And you don have lage files in /

ls -lha

#15

Here is the listing:

root:/# ls -lha
drwxrwxr-x    1 root     root         172 Jan 25 20:15 .
drwxrwxr-x    1 root     root         172 Jan 25 20:15 ..
-rw-------    1 root     root        1.0K Sep 16 23:03 .rnd
drwxr-xr-x    1 root     root        1.4K Jan 25 20:16 bin
drwxr-xr-x    1 root     root         158 Jan 25 20:15 boot
lrwxrwxrwx    1 root     root          13 Jan 25 08:54 boot.scr -> boot/boot.scr
drwxr-xr-x    7 root     root        3.1K Jan 31 15:12 dev
drwxr-xr-x    1 root     root        2.6K Jan 25 20:15 etc
drwxr-xr-x    1 root     root         610 Jan 25 20:15 lib
drwxr-xr-x    1 root     root          82 Dec 20 12:08 mnt
drwxr-xr-x    1 root     root           0 Sep  5  2016 overlay
dr-xr-xr-x  169 root     root           0 Jan  1  1970 proc
drwxrwxr-x    1 root     root           8 Jan 25 20:15 rom
drwxr-xr-x    1 root     root          92 Dec 19 21:59 root
drwxr-xr-x    1 root     root          60 Dec 12  2017 run
drwxr-xr-x    1 root     root        1.7K Jan 25 20:16 sbin
drwxr-xr-x    1 root     root          36 Dec  8 20:25 srv
dr-xr-xr-x   11 root     root           0 Jan 28 03:30 sys
-rw-r--r--    1 root     root           0 Jul 30  2017 t-output
-rw-r--r--    1 root     root         588 Jan  2  2019 tisk
drwxrwxrwt   30 root     root        1020 Jan 31 23:24 tmp
drwxr-xr-x    1 root     root          50 Dec  8 20:27 usr
lrwxrwxrwx    1 root     root           4 Jul 13  1998 var -> /tmp
drwxr-xr-x    1 root     root         122 Jan 25 20:15 www

BTW, thanks for your patience, I highly appreciate that!

1 Like
#16

clearly there is something I do not understand with du and df as when I check my own MOX the numbers do not add up either.

My number is in MB though so I have not payed attention to it before.

Is there any kind soul here who can shed some light over this, mostly for education purpose.

Is there a chance that df ”used” includes blocks of the sd card marked as worn out/broken?

Sorry, but I’m out of ideas.

EDIT: some kind of fs corruption or other strange state?

#17

needed to readup on df and du,
My difference in numbers can be related to how the different tools work, but not in you case as the difference is to large.

I would recomend btrfsck ( there is a number of options you can test)
Do not forget backing up you data/conf first on an external drive

And perhaps it’s wise to have a spare sd card at hand