Multiple virtual servers (LXC containers) possible?

[nerdpunk]

@adminX

ArchLinuxARM and use the dsa driver for the switch chip. This will end in patching the kernel.

i tested the dsa driver on my wrt1200 on linux-4.1, and it was working quite well

[woosting]

@woosting

• You guys reckon the Omnia will be stable / fast enough to run a LAMP+mail-server?

Will it run fast enough from an external USB3 HD, or do I really need to run the containers it from an internal mSATA SSD?

• Will the Omnia be stable / fast enough to run a LAMP+mail-server?

And regarding to the file-system location, seeing @bernstein’s related reply in another topic, I guess in order of preference (not taking cost into account):

1. mSATA (FAST + relieving the int.flash of the Omnia)
2. external-USB-HDD (slower but still relieving the int.flash + possibly LARGE storage-wise)
3. internal flash (relatively small / sensitive to “wearing” by reads/writes to it).

[woosting]

So now a 4rth option is added to the list:

1. mSATA (FAST + relieving the int.flash of the Omnia)
2. Conventional SATA HDs using the NAS perk (normal/fast speeds + relieving the int. flash).
3. external-USB-HDD (slower but still relieving the int.flash + possibly LARGE storage-wise)
4. internal flash (relatively small / sensitive to “wearing” by reads/writes to it).

Still I am thinking of going for option 1 (for the high performance and to keep the housing small). But I do have one worry that may push me to buy the NAS/SATA upgrade:

• Does an mSATA_ SSD fit into the normal (non-NAS) housing of the Turris Omnia?

EDIT: According to @adminX’s post in another topic it should fit. Any second opinions? And/or @adminX how do how do you know for sure?

[adminX]

@woosting

EDIT: According to @adminX’s post in another topic it should fit. Any second opinions? And/or @adminX how do how do you know for sure?

mSATA SSDs are the same form factor as mini-PCIe cards. Their only difference is the pinout of the connector.

There is another option 1+2: mSATA SSD + up to 2 internal SATA HDD/SSD. But this one will only allow one WiFi card and no LTE.
Maybe option 2 with a SSD and a HDD is a better version of this. Still allows 2 WiFi or 1 WiFi + UMTS/LTE.

I would still go for option 1+3(b): minimized access time on the internal SSD, fast enough large space and 1 WiFi + UMTS/LTE. Option 3b are network connected drives (SAN using iSCSI build from an old NAS).

[woosting]

mSata (for the root LXC root FSs) + external HD (for volume/storage) is my current go-to solution. Still speedy (mSATA SSD) and enough “breathing room” for data (external disk).

Second choice: run the LXC containers entirely from the external (USB3) disk. Cheap.

Would running it all from an external (USB3) disk be really that slow (how would that compare - for instance - to running a RaspberrypiB+ from an class10 SD card / comparible / slower / quicker)?

[miska]

@woosting

Would running it all from an external (USB3) disk be really that slow (how would that compare - for instance - to running a RaspberrypiB+ from an class10 SD card / comparible / slower / quicker)?

I would say faster in general, but there is one trick - with classic HDD I prefer to let it spin down when it is not doing anything and in this setup it will be faster if HDD is spinning but could take some time to spin it up. Got fifth option for you - USB3 flash drive - easily replaceable, nowadays quite large and fast and not spinning.

And regarding other question you posted long time ago, you can get LXC container from ARM and run it on your x86 PC using qemu, with qemu-user and binfmt it should work quite fine, would be slower than native x86 container but from my experience it is quite usable unless you need real performance - so as a temporally backup solution you can use even your PC.

[woosting]

@miska

Got fifth option for you - USB3 flash drive - easily replaceable, nowadays quite large and fast and not spinning.

Yah, I thought of that (buying a 32Gb thumbdrive and using it for my LXC-rootfs/snapshots). Do you reckon it is reliable enough to run, say, a web + mail server (what about it wearing)?

@miska

you can get LXC container from ARM and run it on your x86 PC using qemu, with qemu-user and binfmt it should work quite fine

Ooooh, thanks! I’ll look into that (so much to learn in this area…)!

[woosting]

Also, in follow-up of @miska’s suggestion to use a flash-disk as the LXC rootfs host:

Thumbdrives have increased in size lately and are relatively “big”, but not that big. Nowadays they are about what size; 16GB, 32GB, or maybe 64GB tops (to stay affordable / as otherwise the internal sata-SDD would be better choice for sure, if not already)?

Hence my question: does anyone have good knowledge (or reference) to apply ‘overlayfs’ (on the Omnia)? So LXC-snapshots would require little to no space (only storing the deltas)?

I wonder how LXC is applied in general on the Omnia b.t.w. I asked cz.nic to release documentation in advance. But cz.nic officials seem fairly inactive on this forum (probably too busy gathering/testing for release etc.). So I fear I’ll have to wait on the actual shipment date for that to be released too. But any (3rd party) information about the LXC implementation and appliance on the Turris Omnia is more then welcome (it being my main appliance, next to securely routing of course)!

[miska]

@woosting

Also, in follow-up of @miska’s suggestion to use a flash-disk as the LXC rootfs host:

Thumbdrives have increased in size lately and are relatively “big”, but not that big. Nowadays they are about what size; 16GB, 32GB, or maybe 64GB tops (to stay affordable / as otherwise the internal sata-SDD would be better choice for sure, if not already)?

Internal SATA is much better choice for sure, USBs are just much more affordable. Yep, they will die, but if you set them up right, they can last long and you can do RAID out of them. Internal mSATA would warn you before dying.

@woosting

Hence my question: does anyone have good knowledge (or reference) to apply ‘overlayfs’ (on the Omnia)? So LXC-snapshots would require little to no space (only storing the deltas)?

You can use overlayfs, but AFAIK only by hand - so if you want to use squashfs with overlayfs to save some space, you would probably have to script it.

@woosting

I wonder how LXC is applied in general on the Omnia b.t.w. I asked cz.nic to release documentation in advance. But cz.nic officials seem fairly inactive on this forum (probably too busy gathering/testing for release etc.). So I fear I’ll have to wait on the actual shipment date for that to be released too. But any (3rd party) information about the LXC implementation and appliance on the Turris Omnia is more then welcome (it being my main appliance, next to securely routing of course)!

Same as everywhere else - lxc-start, lxc-stop, lxc-console and such

[woosting]

@miska

You can use overlayfs, but AFAIK only by hand

I tried to Google on how to do this (I have a general understanding of a layered-fs, but no practical experience whatsoever). But to no avail.

@miska could you summarize the general process steps for me using the most simple route (I’d say using overlayfs as that is in the linux kernel by default), including actually making the LXC-snapshot? Ofc. I’ll google the details; but just to push me into the general direction (now I seem to drown in the conceptual info.)?

@miska

Same as everywhere else - lxc-start, lxc-stop, lxc-console and such

Ah so that is good! That does assume an SSH connection (or at least a CLI). I just read that that is indeed possible, I just did not think of SSH-ing into my router at a first thought. Usually router-administration is done (primarily) via a web-interface. I guess much is different in the Omnia camp!

Would that all work out-of-the-box b.t.w. (I think that was promised by cz.nic)?

[white]

@woosting

Ah so that is good! That does assume an SSH connection (or at least a CLI). I just read that that is indeed possible, I just did not think of SSH-ing into my router at a first thought. Usually router-administration is done (primarily) via a web-interface. I guess much is different in the Omnia camp!

Router administration is done normally via SSH/CLI for enterprise devices and via web UI for basic consumer or small office devices

[nerdpunk]

@woosting

Hence my question: does anyone have good knowledge (or reference) to apply ‘overlayfs’ (on the Omnia)? So LXC-snapshots would require little to no space (only storing the deltas)?

lxc also supports btrfs, didn’t try it yet, but maybe that’s the better way to go

[miska]

@woosting

@miska

You can use overlayfs, but AFAIK only by hand

I tried to Google on how to do this (I have a general understanding of a layered-fs, but no practical experience whatsoever). But to no avail.

@miska could you summarize the general process steps for me using the most simple route (I’d say using overlayfs as that is in the linux kernel by default), including actually making the LXC-snapshot? Ofc. I’ll google the details; but just to push me into the general direction (now I seem to drown in the conceptual info.)?

Depends what do you want to achieve. I thought you had some idea what you want to use overlayfs as you were asking for it

You can create for example squashfs[see mksquashfs], use overlayfs[mount -t overlay …] to attach writeable overlay on top of it and use that in LXC. So you would save some space. You can create squahfs periodically from the resulting dir and remount everything and backup that squashfs.

Or you could have one squashfs (just base OS) with multiple writeable overlays used by different containers.

LXC uses just directories on your disk and it doesn’t care what fs is underneath and AFAIK lxc-snapshot just creates copy of it. So you can make whatever you want out of it.

@woosting

@miska

Same as everywhere else - lxc-start, lxc-stop, lxc-console and such

Ah so that is good! That does assume an SSH connection (or at least a CLI). I just read that that is indeed possible, I just did not think of SSH-ing into my router at a first thought. Usually router-administration is done (primarily) via a web-interface. I guess much is different in the Omnia camp!

Would that all work out-of-the-box b.t.w. (I think that was promised by cz.nic)?

There is a web wizard to guide you through simple setup and web UI, but ssh will work out of the box (maybe once you are done with wizard). I think there is even web page in OpenWRT where you can enter command, it will run them and tell you the output. So think about Omnia more like a computer that has more network cards

[woosting]

Thanks Miska!

@miska

I thought you had some idea what you want to use overlayfs as you were asking for it

I do; I want to run system crytical services on my Omnia (such as my personal e-mail server etc.). I do not want them to be down (for a long time) and possibly even be transportable (sadly real migration may require HW virtualisation - which is too demanding - even for an Omnia).

As I am slowly getting adapt at Linux I am still fairly inexperienced. I know enough to make terrible mistakes when tinkering with the system. So once I have a reasonable stable configuration going; I want to snapshot that so when I mess up during later “experimentation” I can relatively easily switch back to the situation that was stable.

To minimize the snapshots’ overhead (and hence keep disk space at bay e.g. if I indeed use an 64Gb Thumb-drive for instance) I thought a layered approach would be best (only storing the delta’s additionally with each snapshot).

@man page

lxc-clone Creates a new container as a clone of an existing
container. Two types of clones are supported: copy and snapshot. A copy
clone copies the root filessytem from the original container to the new. A
snapshot filesystem uses the backing store’s snapshot functionality to create
a very small copy-on-write snapshot of the original container. Snapshot
clones require the new container backing store to support snapshotting. Currently
this includes only aufs, btrfs, lvm, overlayfs and zfs. LVM devices do not support
snapshots of snapshots.

The backing store of the new container will be the same type as the
original container, with one exception, overlay containers.
aufs and overlayfs snapshots can be created of directory backed
containers. This can be requested by using (for overlayfs) the
-B overlayfs arguments.

So I think it should be very easy if the original container’s “backing store” (I must read up on what that is exactly (I understand that it is where the lxc rootfs is stored)) is a layered-filesystem to begin with. But the text quoted above also suggest that the -B argument may help make an overlayfs when cloning / taking a snapshot from a directory-backed container.

I’ll experiment a little with that!

@nerdpunk

lxc also supports btrfs, didn’t try it yet, but maybe that’s the better way to go

@nerdpunk I’ll look into that specific one (as it seems to surface in about every how-to/tutorial I read!

The thing is so much to do so little time… can’t the Omnia be delayed a little more?

[woosting]

apparently it works as such;
lxc-clone -s -B overlayfs -o foo -n bar

It should create a clone (bar) of the base container (foo), as an overlay on top of it. That one (the clone “bar”) will have an overlayfs and is hence snapshot-able in a layered fashion from there onwards:

I was confused as at my end it did not work due to a bug - kernel rename of ‘overlayfs’ to ‘overlay’ not taken into account - in the Debian package I was/am using (anyone able to verify that it does work in another distro)? Luckily (for me) it is marked as fixed in versions lxc/1:1.1.5-1 though!

B.t.w.:
lxc-clone -s -B aufs -o foo -n bar
does work for me at this point!

[woosting]

@nerdpunk

lxc also supports btrfs, didn’t try it yet, but maybe that’s the better way to go

@nerdpunk I read up on it and it seems a nice alternative (not sure whether I am ready to replace ext4 by it though as it still fairly young / not picked up a lot yet). But it sure looks promising!

[woosting]

I guess the remaining question (regarding layered snapshotting of LXC containers) then is:

• Does openWRT kernel correctly support ‘overlayfs’ and/or ‘aufs’?

and/or:

• how to create a LXC-container in an overlayfs (or btrfs) backing store to begin with on openWRT?

[miska]

@woosting

I guess the remaining question (regarding layered snapshotting of LXC containers) then is:

• Does openWRT kernel correctly support ‘overlayfs’ and/or ‘aufs’?

As ‘overlayfs’ is in mainline it is supported and work. AUFS AFAIK is not there (neither mainline nor Omnia).

@woosting

and/or:

• how to create a LXC-container in an overlayfs (or btrfs) backing store to begin with on openWRT?

In theory same as everywhere else - LXC and kernel and overlayfs is pretty much the same as on any Linux distribution. But how exactly I can’t tell as I wouldn’t use it via lxc tools if I would use it at all. I would use overlayfs to create rootfs directories I want managing overlays by hand and run lxc on top of those.

[woosting]

@miska

I would use overlayfs to create rootfs directories I want managing overlays by hand and run lxc on top of those.

Yah (and that should be an option as - like you said - the Omnia can be regarded a small form-factor computer with linux). The reason I am leaning towards using LXC specific tooling is that that seems simpler / delicately build for LXC usage.

I thought to build a directory backed root-fs and clone that one into a new (base) container having an overlayfs (to be used as the base to make layered lxc-snapshots from):

lxc-clone -s -B overlayfs -o dirBasedFsCont -n overlayBasedFsContClone

Maybe doing it manually as you are planning on is better though. Or even simply the “normal/typical” way to do it, as I can not seem to find any other way. I should read up on how to do that (or could you conceptually sum up the basic steps one should perform to do so?).

[nerdpunk]

for all of you, who want to use unpriviledged LXC containers (with TurrisOS or plain openWrt/LEDE), i got kind of a howto (not a beginners guide, that one is located here), and i hope the devs will pick it up and integrate this into TurrisOS

basically it boils down to:

1. apply this patch to the packages feed, select and install shadow-newgidmap and shadow-newuidmap

2. add appropiate values to /etc/subgid and /etc/subuid files

3. applying this patch (quick and dirty, sry ;)) to an openwrt build root, will enable you to build and install uidmapshift, a utility to shift the uid:gid of an existing container

4. the last issue gave me a headache for some time, until i digged it up in the kernel sources: procd mounts proc and sys with the noatime flag (which is quite unusual, at least never saw that on another distro), and if lxc mounts proc and sys into another namespace, noatime needs to be set there, too, at least if it’s not mounted as root (which is the case for unpriviledged containers, of cause)
   long story short: starting an unpriviledged container will fail on openWrt/LEDE, unless you add “noatime” to the proc+sys mount options in your containers config… or… imo the better way (because you can stick to default configs, like “proc:mixed” and such) -> use this patch to change procd’s mount flags for proc+sys to “relatime”, which actually works

that’s it