Lxc-snapshot resulting in an rsync error?

  • Could anyone from cz.nic maybe shed some light on what to expect here (time-frame-wise)?

P.S. Not that I am in a great hurry (@Ondrej_Caletka’s btrfs-level shapshotting works for me at the moment), but I am curious whether this is on the scope of being addressed by cz.nic (or at least the one maintaining Omnia’s package repositories) and how such things “work” proceduraly.

Any help with lxc-snapshot (ubuntu name of container for example) ? no work (on stoped lxc container, on runnig also):
lxc-info -n ubuntu :say STOPPED

lxc / srv/ubuntu/ on mSata disk…

now do :

lxc-snapshot -n ubunturoot@turris:~# lxc-snapshot -n ubuntu

lxc-snapshot: lxccontainer.c: do_lxcapi_snapshot: 3209 Snapshot of directory-bac ked container requested.
lxc-snapshot: lxccontainer.c: do_lxcapi_snapshot: 3210 Making a copy-clone. If you do want snapshots, then
lxc-snapshot: lxccontainer.c: do_lxcapi_snapshot: 3211 please create an aufs or overlayfs clone first, snapshot that
lxc-snapshot: lxccontainer.c: do_lxcapi_snapshot: 3212 and keep the original con tainer pristine.
rsync: extended attributes are not supported on this client
rsync error: syntax or usage error (code 1) at main.c(1567) [client=3.1.2]
lxc-snapshot: bdev.c: rsync_rootfs: 3367 rsyncing /srv/lxc/ubuntu/rootfs to /srv /lxc/ubuntu/snaps/snap0/rootfs
lxc-snapshot: lxccontainer.c: copy_storage: 2610 Error copying storage
lxc-snapshot: lxccontainer.c: do_lxcapi_snapshot: 3217 clone of /srv/lxc:ubuntu failed
lxc-snapshot: lxc_snapshot.c: do_snapshot: 55 Error creating a snapshot

Any help with snapshooting lxc for backup ?

It seems that lxc-snapshot is giving rsync command a preserve extended attributes (xattrs) option that is not supported by Omnia’s rsync or filesystem or Omnia’s kernel.

Fine - but it is no solution, etc. why command exist when no work… so better than no exist, than no work. I can not repair this.

Any updates (been a while)?

I’m still having issues with using lxc-snapshot so I’m guessing no progress has been made by OpenWRT/LEDE or the Turris folk on the new rsync either.

I know that one alternative is to use btrfs but I’d rather not reformat all over again. Plus ext4 is better than btrfs for SSDs apparently?

Could I not ‘make’ rsync myself and install the ipk myself?

Alternatively, could I not create snapshots manually by just creating copies of the /rootfs/ directory or is there more to lxc-snapshot than just creating a copy of the /rootfs/ directory?

Is it really that bad? You have me worried (as I have many system-critical stuff running in my LXC-containers (on top of BTRFS due to the aforementioned)!

That is actually a good idea (although I am far from an expert… it sounds legit). Should you try, would you please share your results? I’m very interested in this (I may re-format to ext4 if 'make’ing one’s own rsync leads to a working lxc-snapshot!

Indeed you could I guess… (I even wrote [a small how-to on Github](https://github.com/woosting/turris-notes/blob/master/howtos/lxc/container-handling/backup.md for that preserving all the user rights etc. which is not necessarily the case with a simple copy), but that is more of a backup (not really a “layered”) approach. Snapshots are near-instant as they only store the incremental differences (-ish).

You are furthermore welcome to use my backup script (and or help improving by forking it!) if you are confident enough with bash-scripting.

But - as said - these have a more archive/storage/backup/migration character. I have not yet found a better snapshot method than using BTRFS (which I am reluctant now to use… reading up on BTRFS being bad for SSD).

Personally I now wonder:

  • In what cases BTRFS indeed generates so many write-cycles?
  • Is embedding the rsync solution on the scope (of cz.nic)?

Having had a look at the OpenWRT documentation for building packages, it’s not difficult per-se. But to build an individual package (i.e. rsync), you have to setup your whole environment for building OpenWRT as a whole and a lot of that stuff looks well beyond my capabilities unfortunately.

Having had a look at your backup script, it seems ideal! And although neither my copy-paste approach or your script is as good as native lxc-snapshot, your script seems like the best alternative IMO, especially with my reservations about BTRFS.

Good news everyone!

I contacted the Turris team by email yesterday about whether fixing the rsync error was something on their radar and this is the response I received this morning:

Hi {{me}},

thanks for the message. We know about this and have already fixed it, it should be part of the next bigger release or maybe even part of a smaller one inside 3.8.

Have a great day!

Granted we’ll have to wait for a future release (hopefully not too long) but we now know for certain about the status of this issue which is great.

1 Like

Thanks for the headsup of the good news!

This is why I love the Turris Omnia, shit gets fixed (try such a feature request at Linksys…).

I’m already happy to know it is on the scope (even “fixed” in theary)!

I’m not sure if you could use ext4 (without using LVM or some other FS abstraction layer). I’m starting to think that lxc-snapshot is depending on the filesystem’s “layer functionality” (as discussed in the Ubuntu documentation about LXC):

Snapshots are supported for btrfs, lvm, zfs, and overlayfs containers. If lxc-snapshot is called on a directory-backed container, an error will be logged and the snapshot will be created as a copy-clone.

we now have 3.8.1 and lxc-snapshot is not fixed :frowning:

I remain hopeful it will be soon!

@thencein Seeing that you got the info from an official support e-mail from cz.nic my hopes are still up too!

@maurer We’ll just have to be a little more patient I guess .

Can you please once again debug what is happening? I just checked and request reported to us to enable extended attributes for rsync was deployed (I just tested that I can use -X switch without rsync failing that it’s not supported). It seems to me that there is some other problem in play.

I am storing my containers on an ext4 file system and as @woosting has pointed out, lxc-snapshot is unable to create ‘true’ snapshots on ext4 because it’s not supported by the file system.

That said, I was able to create a copy-clone successfully. (If the filesystem doesn’t support snapshots, it reverts to copy cloning instead).

root@turris:~# echo "Fully working Guacamole with 2FA, Let's Encrypt SSL using NGINX" > snap-comm && lxc-snapshot -n guac-srv -c snap-comm && rm snap-comm

lxc-snapshot: lxccontainer.c: do_lxcapi_snapshot: 3209 Snapshot of directory-backed container requested.
lxc-snapshot: lxccontainer.c: do_lxcapi_snapshot: 3210 Making a copy-clone.  If you do want snapshots, then
lxc-snapshot: lxccontainer.c: do_lxcapi_snapshot: 3211 please create an aufs or overlayfs clone first, snapshot that
lxc-snapshot: lxccontainer.c: do_lxcapi_snapshot: 3212 and keep the original container pristine. 

For my current setup, lxc-snapshot is working as intended seeing as I’m using an FS that doesn’t support snapshots anyway.

Is anyone here using a non-ext4 file system that would be willing to try running lxc-snapshot to see if they’re able to get true snapshots working (i.e. not just copy-clone)?

I just tried. At least I think I did… My /srv/ is mounted to my mSATA disk which is BTRFS formatted (how I currently make manual BTRFS subvolume snapshots form my containers on FS level).

When I simply lxc-snapshot -n <containername> it gives the same output as you posted (despite it being a BTRFS subvolume). So I wanted to make an lxc-clone instead (telling it to be a (btrfs)layered clone) to, in-turn take snapshots from, which is what the instructions tell us to do, and which is what I did before on other (“regular”) GNU/Linux boxes but lxc-clone is missing from the Turris’ lxc toolkit (or can I now - seeing rsync got some attention from the devs - simply install/use lxc-clone now?):

I am more than willing to test stuff, if someone can point me into the right direction and tell me how to do it. Clearly I am missing some experience/knowledge here (maybe I should create a test container “telling” lxc to make use of the BTRFS possibilities?), or maybe the FS MUST be aufs / overlayfs?

Let’s “put our houlders under this one” together, shall we? :slight_smile:

I haven’t actually tried this out but are you not able to just opkg install lxc-clone?

I’m happy to inform you guys that I successfully installed lxc-clone (via LuCi) and:

  1. Created a clone of an existing container (explicitly making it use a BTRFS backing store):

    lxc-clone -B btrfs <original-name> <clone-name>

  2. After that I was able to make (near-instant) snapshots using regular LXC tooling:

    lxc-snapshot -n <clone-name>

The second command refers to the “Clone-name” because I had to clone the original to a BTRFS backed one in step one, if the container was created using a BTRFS backing store to begin with then step one can be omitted of course).

So THANKS fly out to @cynerd for making rsync with extended attributes available on out systems! Which (amongst other things) resulted in the option for near-instant snapshots of near-zero size!

I do have a remaining/follow-up question though.

2 Likes

for me it fails:
lxc-clone -B btrfs deb8-arm deb8-arm-clone
clone failed
and I see in dmesg:
[688374.474605] BTRFS warning (device mmcblk0p1): csum failed ino 466 off 118784 csum 545085046 expected csum 1064283543