Unprivileged LXC container on Turris OS 5.0.0

The latest release of Turris OS 5.0.0 finally allows to execute unprivileged LXC containers. While LUCI support would be nice, currently the sole option is the command line on the Turris itself. Let’s get started by setting up an unprivileged container called “test”:

As multiple tutorials [1][2] describe it already, first we need to set up the LXC environment:

# append idmaps to lxc default config
$ echo "lxc.idmap = u 0 100000 65536" >> /etc/lxc/default.conf
$ echo "lxc.idmap = g 0 100000 65536" >> /etc/lxc/default.conf

# set subuid if not present
$ echo "root:100000:65536" > /etc/subuid

# set subgid if not present
$ echo "root:100000:65536" > /etc/subgid

Even if “default” LXC packages are already installed, we still require the ones that enable us to support unprivileged containers:

$ opkg update
$ opkg install lxc-unprivileged
$ opkg install lxc-usernsexec

In theory, everything should be in place now. But issuing lxc-create fails:

$ lxc-create -t download -n test
Setting up the GPG keyring
Downloading the image index
ERROR: Failed to download http://repo.turris.cz/lxc//meta/1.0/index-user
lxc-create: test: lxccontainer.c: create_run_template: 1617 Failed to create container from template
lxc-create: test: tools/lxc_create.c: main: 327 Failed to create container test

Seems to be as if the official Turris repository does not support it yet. Consequently, let us use the repository of images.linuxcontainers.org:

$ lxc-create --quiet \
             --name test \
             --bdev best \
             --template download \
             -- \
             --dist archlinux \
             --release current \
             --arch armhf \
             --server images.linuxcontainers.org \
             --no-validate

After the command succeeds, there should be a container without root:root permissions:

$ ls -l /srv/lxc/ | grep test
drwxrwx---    3 100000   100000        4096 Mar 22 11:47 test

Let us execute it:

$ lxc-start -n test
lxc-start: test: lxccontainer.c: wait_on_daemonized_start: 842 Received container state "ABORTING" instead of "RUNNING"
lxc-start: test: tools/lxc_start.c: main: 330 The container failed to start
lxc-start: test: tools/lxc_start.c: main: 333 To get more details, run the container in foreground mode
lxc-start: test: tools/lxc_start.c: main: 336 Additional information can be obtained by setting the --logfile and --logpriority options
root@turris:/mnt# lxc-start -n test -F
lxc-start: test: utils.c: safe_mount: 1179 Operation not permitted - Failed to mount "proc" onto "/usr/lib/lxc/rootfs/proc"
                                                                                                                           lxc-start: test: conf.c: lxc_mount_auto_mounts: 724 Operation not permitted - Failed to mount "proc" on "/usr/lib/lxc/rootfs/proc" with flags 14
                             lxc-start: test: conf.c: lxc_setup: 3539 Failed to setup first automatic mounts
                                                                                                            lxc-start: test: start.c: do_start: 1263 Failed to setup container "test"
                                                                                                                                                                                     lxc-start: test: sync.c: __sync_wait: 62 An error occurred in another process (expected sequence number 5)
                                                 lxc-start: test: start.c: __lxc_start: 1939 Failed to spawn container "test"
                                                                                                                             lxc-start: test: tools/lxc_start.c: main: 330 The container failed to start
lxc-start: test: tools/lxc_start.c: main: 336 Additional information can be obtained by setting the --logfile and --logpriority options

Quite a mess. As mentioned [3], the reason on Turris OS 5.0.0 seems to be:

Failure to mount proc is usually a sign of the kernel’s overmounting protection preventing the mount.

So let us fix it until Turris OS is fixed:

$ mkdir -p /mnt/lxc-unpriv-fix/proc
$ mount -t proc none /mnt/lxc-unpriv-fix/proc
$ mkdir -p /mnt/lxc-unpriv-fix/sys
$ mount -t sysfs none /mnt/lxc-unpriv-fix/sys

Issue left: auto-start
Make above sys/proc issue permanent

$ cat < EOF > cat /etc/rc.local 
# Put your custom commands here that should be executed once
# the system init finished. By default this file does nothing.

mount -t proc none /mnt/lxc-unpriv-fix/proc
mount -t sysfs none /mnt/lxc-unpriv-fix/sys

exit 0
EOF

/etc/init.d/lxc-auto does sadly not function for me, as the container exits after successful creation, any ideas? I suspect the init.d system…

8 Likes

thanks! A LOT!
can’t believe the fix is as simple as mounting proc & sys anywhere…

added your solution to the bug report : https://gitlab.labs.nic.cz/turris/turris-build/issues/28#note_156351

2 Likes

sadly i couldn’t get docker to work inside an unprivileged lxc container. i couldn’t get overlay2 to work (docker create fails when mounting) and neither vfs (container won’t start)

very likely due to not being allowed to mount anything inside the unprivileged container…

Any ideas how to debug autostart process? Unprivilidged containers work for me but its anoyying to start containers after update with reboot.

EDIT: simply adding lxc-start container to /etc/rc.local solves the problem.

1 Like

Thanks @psiegl for this.

I have been running a couple unprivileged containers on my TO for a few days using this method.

It was flawless for a while, surviving a few reboots and even the odd TOS update (5.2.7, I think?). Suddenly though, I started seeing the error below and containers fail to boot:

Failed to look up module alias 'autofs4': Function not implemented
Failed to mount cgroup at /sys/fs/cgroup/systemd: Operation not permitted
[!!!!!!] Failed to mount API filesystems.
Exiting PID 1...

I have tested Debian, Ubuntu, Fedora and Alpine containers. Of those, only Alpine starts up properly.

A bit of Googling and it suggests it’s something to do with systemd (which Alpine doesn’t use so it could well be), and/or outdated LXC. I tried a few of the workarounds suggested in different forums but nothing worked so far.

Has anyone seen this and found a workaround?

This one, perhaps?

Huh, interesting. Looks like I didn’t get the fix because my /etc/lxc/default.conf was changed to add those lxc.idmap settings.

I diffed it with the -opkg one and added the workarounds to the config and now it seems to work.

I have no idea why something that was working suddenly stopped though.

Thanks for the pointer, I’ll monitor it for now but hopefully that will do the job!