Disk space gone

Software SW help
1

Hi all,

I have a problem with the space left on my turris. Since yesterday all the disk space is consumed but I am unable to track down the specific location.
Here’s some related info

Filesystem                Size      Used Available Use% Mounted on
/dev/mmcblk0p1            7.3G      7.1G    352.3M  95% /
tmpfs                  1011.2M      5.4M   1005.9M   1% /tmp
tmpfs                   512.0K      4.0K    508.0K   1% /dev
root@turris:~# du -hd 1 /
0       /sys
0       /overlay
2.3M    /boot
1.5M    /bin
0       /srv
324.0K  /www
4.0K    /rom
6.7M    /sbin
5.4M    /tmp
4.0K    /dev
484.4M  /usr
0       /proc
10.0M   /lib
0       /mnt
52.0K   /root
5.1M    /etc
8.0K    /run
32.0K   /home
515.8M  /
root@turris:~# lsof | grep deleted
root@turris:~#

I really have no idea what do to? I rebooted the device several times but this does not change anything.

Thanks for help.

1 Like
2

please post the output of schnapps list

3

Could you please tell me how I do that?

4

as you already have a root shell on your router simply enter schnapps list as a command.

5

Thanks, Here you go:

root@turris:~# schnapps list
    # | Type      | Date                      | Description
------+-----------+---------------------------+---------------------------------
    2 | rollback  | 2016-10-19 16:26:28 +0000 | Rollback to snapshot factory
    3 | rollback  | 2016-10-19 16:33:53 +0000 | Rollback to snapshot factory
    4 | rollback  | 2016-10-20 15:55:49 +0000 | Rollback to snapshot factory
    5 | pre       | 2016-10-20 18:11:12 +0200 | Automatic pre-update snapshot
    6 | pre       | 2016-10-21 02:56:08 +0200 | Automatic pre-update snapshot
    7 | time      | 2016-10-23 01:05:01 +0200 | Snapshot created by cron
    8 | pre       | 2016-10-28 14:56:17 +0200 | Automatic pre-update snapshot
    9 | time      | 2016-10-30 01:05:01 +0200 | Snapshot created by cron
   10 | time      | 2016-11-06 01:05:01 +0100 | Snapshot created by cron
   11 | pre       | 2016-11-07 02:56:19 +0100 | Automatic pre-update snapshot
   12 | pre       | 2016-11-07 14:56:07 +0100 | Automatic pre-update snapshot
   13 | time      | 2016-11-13 01:05:03 +0100 | Snapshot created by cron

Seeing that list, I guess that schnapps made a snapshot of some mad lxc consuming all the disk space? Is this possible?

6

Ok, after deleting the latest snapshot, I got back 4.3 GB of space. Thanks for the hint!

7

Yes, snapshops cover anything on the internal filesystem. LXC is included.

Solution part one: delete snapshots using schnapps delete 2 3 4 5 6 7 8 9 10 11 to remove all snapshots except the last 2.

This should give back space.

Part two: do you want LXC in the snapshots?

1 Like
8

I do not require LXC in the snapshots but it would be nice to have it included.

However, is it possible to put snapshots on another device such as an sshfs mount? I have more than enough space on a remote machine.

9

Could you check what exactly takes 500 MB in /usr? This is really big.

This snapshots are a feature of the btrfs file system. They work a bit like Shadow Copy in Windows. If you delete a big file that is already in a snapshot it won’t give you any free space. If you copy it again on your turris it will take new space. This is something you should take into account.

They won’t help you with rollback using the reset button if you have them on an external storage.

If the config in the repo is the same as on the device then you may end with up to 16 automatic snapshots. This means they have to be under 500 MB in average.

10

The 500mb file in usr is my snapraid content file, which is there on purpose. I’m using 2 parity so I have to keep 3 different content files on 3 physically different devices (one internal, one on the disk array, one on an sshfs mount).

Yes I though that having them on an external device wont work very well together with the reset button. Is it possible to exclude the internal lxc folder (srv/lxc) from the snapshots?

11

Yes but this is nothing simple. Snapraid images are kinda bad for this. I would use an usb stick for this. Maybe the no-cow flag helps on this file. chattr +C snapraidimage But i would have to read some docs to give an answer.

This won’t help with the snapshot size problem.

12

From my understanding, this could be possible if you made that folder as a subvolume of the very root of the Btrfs filesystem, not the subvolume of current TurrisOS root:

# mount /dev/mmcblk0p1 /mnt/
# ls /mnt/
1.info    3.info    5.info    @         @2        @4        @6
2.info    4.info    6.info    @1        @3        @5        @factory
# btrfs subvolume create /mnt/@lxc
Create subvolume '/mnt/@lxc'
# umount /mnt
# mount /dev/mmcblk0p1 /srv/lxc -o subvol=@lxc

The last command have to be repeated after every reboot. I haven’t figured out yet, how to automate mounting of subvolumes properly, since the native OpenWRT block mounter refuses to mount the same device more than once. I guess you can always put it into rc.local, there may be, however, some race conditions with containers starting up earlier.

13

Just a clarification: Btrfs does not support recursive snapshots. So even creating a subvolume nested under a regular volume does solve this issue. It should be therefore sufficient to create a subvolume for LXC containers like this:

btrfs subvolume create /srv/lxc
1 Like
14

Just a small thanks to all you guys… you all rawk!