Lxc-snapshot resulting in an rsync error?

So; I have container running fine and dandy… time to experiment!

But first… lxc-snapshot (to have a milestone to return to when things go awry)!

But then I am presented with the following:

rsync: extended attributes are not supported on this client
rsync error: syntax or usage error (code 1) at main.c(1567) [client=3.1.2]
lxc-snapshot: bdev.c: rsync_rootfs: 3367 rsyncing /srv/lxc/base/rootfs to /srv/lxc/base/snaps/snap0/rootfs
lxc-snapshot: lxccontainer.c: copy_storage: 2610 Error copying storage
lxc-snapshot: lxccontainer.c: do_lxcapi_snapshot: 3217 clone of /srv/lxc:base failed
lxc-snapshot: lxc_snapshot.c: do_snapshot: 55 Error creating a snapshot

Anyone else getting (or know how to solve / workaround) this?

It appears that OpenWrt’s rsync is built without that feature.

Ah! Thanks for confirming!

Hmm, where does that leave us, what are our options here? :thinking:

Can we rebuild rsync, for instance (and/or configure LXC not to require extended attributes to be reserved)?

Maybe manually copy the container (but how to make that one apparent to LXC then) as a workaround?

In the interest of keeping things maintainable I would seek to have every feature/bug/etc implemented upstream in OpenWrt/LEDE, but in this case it looks like it may be on the way:

1 Like

I know it’s just a workaround, but perhaps could you use native btrfs snapshot instead of lxc snapshot? I mean make each VM a btrfs subvolume and then just mount it again.

[quote=“tomnia, post:4, topic:1849”]
In the interest of keeping things maintainable I would seek to have every feature/bug/etc implemented upstream in OpenWrt/LEDE[/quote]

I wholeheartedly agree (one of the reasons why I love the Turris Omnia; it adds to open source development - in this case of openWRT).

I see! YAY for open source development! :slight_smile:

Do I interpret that correctly when I say that a rudimentary fix has landed in an x86_64 build of the OpenWRT rsync package? If so; (when/if the pull lrequest is approved by the maintainer ACL/XATTR it) will it “automatically” seep through to the armhf binary too?

What is needed for this process to commence to have it land on our routers (what time time-frame are we talking on average for such processes)? Will cz.nic have to be involved (to have the package adapted/put into Turrus Omnia’s repos) or will it simply land automatically in an update of our routers when OpenWRT updates the package?

Anything I personally can do to help?

Thanks @Ondrej_Caletka for this fine suggestion for a workaround in the mean time!

Legend has it that my friends and I formatted our drive BRTFS already! I will try your suggestion tonight once home from work!

Do you have any newby-friendly url’s my friends and I can use to read-up on BRTFS’s subvolume/snapshot functionality (or maybe even a small instruction in the specific context of the Omnia by your hand)? I am not yet that familiar with filesystems (let alone layered filesystems), but I am willing (wanting!) to learn / test!

Me neither. Look here for some basic introduction. The only thing that is Omnia-specific is the frontend called schnapps, which can be used for working with root file system snapshots. But for custom snapshots, you can use the btrfs tool directly.

The other OpenWRT specific is the automatic mounting, which I didn’t figured out yet properly. The problem is that block binary refuses to mount a device more than once, which is necessary for proper work with snapshots.

1 Like

The pull request says it was tested on that platform by the submitter, but it should apply to all platforms.

This can vary widely. Once OpenWrt implements the change, it should find its way into Turris eventually.

1 Like

@Ondrej_Caletka I swiftly looked into it but this did not look like something I will understand in the middle of the night. :sleeping:

Quite literally… it is 1am at my end now… < off-topic> Sadly some *!#&( hit a pet that crossed the road and drove away - leaving me and another bystander with the animal bleeding to death to look at on my way home commuting… So I had to deal with that first - both practically and emotionally…< /off-topic>

Going to look into it this weekend (its bigger than expected, but worth while in the long run I think). [quote=“Ondrej_Caletka, post:8, topic:1849”]
The problem is that block binary refuses to mount a device more than once, which is necessary for proper work with snapshots.
You reckon the block issue is solvable (with some web-searching / trial and error)? Or is that a technically blocking limitation and should put my spare time in another Omnia project while I “wait” on the new rsync package that @tomnia pointed out (although that may take quite a while to commence)?

I think it would require some patchwork for the block binary. But you are free to use plain UNIX mount command. The only problem is the persistence over reboots, but I guess this could be solved by putting few mount commands into /etc/rc.local.

:sweat: I’m having troubles descending to FS level and make (BRTFS) snapshots (see my next post)… So for now I’m investing a lot of time (learning though), but it is giving me a real hard time - and this is one of my main use-cases (run virtual servers that are snapshottable).

How would I go about monitoring this bug-fix? What are the stations that has to be passed, how can I recognize it having landed in openWRT and - most importantly - Turris repositories?

Sadly I Had some personal set-backs this weekend; so no time for me to put into this yet… but I am working on it. However the article you are referring to @Ondrej_Caletka is not for the faint-hearted TBH. It is going to take me some time before I understand! I am determined though!

I also found an article on www.linux.com that I would like to share here for those interested (slightly more newbie friendly - but I still have to read it completely though).

I also looked at schnaps but the topic about schnapps (root level snapshotting) is not yet ready yet (at the moment of this posting).

If anyone is able to help / please do not hesitate to respond here and/or possibly join us on Teamspeak (during the evenings and/or weekends) if you are willing to help out (send me a PM and I’ll update you on the server and credentials).

I’m starting to slowly grasp the BTRFS basics (albeit I still need to delve a bit deeper into BRTFS snapshotting itself though) but I got the principle down (really good article, in the sense that it that it is a very newbie-friendly system-level explanation).

Chances are I am getting my hands dirty this weekend (the USB drive that I got mounted on /srv/ is already BTRFS formatted and used - so should be fairly easy to start experimenting).

One question up-front though;

  • When I mount my BTRFS formatted USB drive to /srv/ (and manually mkdir lxc in it for my containers to reside in), will it be included in the default Turris schnaps too?

I would not want that to happen - on first thought (MY EEPROM SPACE :scream:) - as my containers may get big (hence the external USB drive). I only want the router defaults (and prototypical router stuff) to be included in those schapps. Or… on second thought - Is schnapps that smart that it will store the /usr/ (mounted usb BTRFS stick) delta’s as layers on the USB disk itself (seeing it is BTRFS in and of itself)? The latter would be great as well, but may be a bit wishful-thinking…

Otherwise I do want to snapshot my LXC containers separately (on the same USB drive as where the root FS resides). Of course preferably by using LXC native lxc-snapshot functionality (which is bugged a.t.m.), but once I learn more about BTRFS snapshotting that would be my third best (right behind my aforementioned wishfull thinking ;)).

At any rate thanks @Ondrej_Caletka for your excellent pointers!

On default schnapps script works only with /dev/mmcblk0p1 device, also known as Omnia’s root device.

1 Like

Yah… I figured as much (was just looking at @miska’s schnapps repo hopefully I may be confident enough one day to make a pull-request myself :sunglasses:).

Which leaves me with 2 options (for now):

  • Wait for the fix to land on my box via the Turris default updates.
  • Learn about BTRFS myself and use its snapshot functionality.

Hmmm… wait or learn… wait or learn…

Decision: I am more impatient than I am lazy… I proclaim the upcoming weekend to be BTRFS weekend! :smile:

Plus BTRFS snapshots are never recursive, which means they never contain any nested subvolumes. I think it is not an issue for LXC containers as those are not touched by automatic updates, so there is no point in snapshotting them. You should snapshot them manually before making a manual update/configuration change of the container itself.

I have just one big drive with btrfs and I chhose this way:
mounting this drive as /mnt/nas - this was set in LuCI
created forlder /mnt/nas/srv/lxc and link it to /srv/lxc
for each container create subvolume example for my nextcloud :
btrfs subvolume create /mnt/nas/srv/lxc/nc
and create subvolume for part I do not want to have in snapshot:
btrfs subvolume create /mnt/nas/srv/lxc/nc/rootfs/mnt/ncdata
now if I want to create snapshot (and I do it before any config trial) just type :
btrfs subvolume snapshot /mnt/nas/srv/lxc/nc /mnt/nas/srv/lxc/nc_snapshot_2016-12-16

this way snapshot appear as new container and I can start it from LuCI or shell lxc-start -n nc_snapshot_2016-12-16. (of course I have to stop original one lxc-stop -n nc as I have same mac address for both containers)

To move snapshot to original subvolume i delete created subvolume -
btrfs subvolume delete /mnt/nas/srv/lxc/nc
and move snapshot simply by
mv /mnt/nas/srv/lxc/nc_snapshot_2016-12-16 /mnt/nas/srv/lxc/nc

This way I do not need any mounting except main btrfs volume and easy create testing environment in seconds.
Thank you @Ondrej_Caletka to propose this solution. I love it :slight_smile:

1 Like

I’m happy to inform you guys (esp. @Ondrej_Caletka) than I am happily snapshotting my containers using btrfs file-system-level snapshots too!

I simply mounted my BTRFS formatted USB drive onto: /srv/lxc and create a (btrfs-snapshottable) subvolume per container. That way I can make snapshots of individual containers at-will!

Thanks all for the pointers (very nice workaround)! Now let’s see how long it will take for Rsync’s defaults to be fixed in Turris’ repository (good example of what to expect for future/other bug-fixes)!

1 Like
  • Could anyone from cz.nic maybe shed some light on what to expect here (time-frame-wise)?

P.S. Not that I am in a great hurry (@Ondrej_Caletka’s btrfs-level shapshotting works for me at the moment), but I am curious whether this is on the scope of being addressed by cz.nic (or at least the one maintaining Omnia’s package repositories) and how such things “work” proceduraly.