[TOS 6.4.1] Summary on how to run unprivileged LXC containers (04/08/2023)

"I am going to write summary howto on running LXC containers with the current state of software in unprivileged way.

Stay tuned…"

  1. In order to run LXC containers use Reforis to install LXC packages.
  2. Then install additional packages to make containers unprivileged.
    And set subuid and subgid Mentioned here:

Follow it until creating container using lxc-create.

I figured out that fixing mount points make containers run with lxc-auto.
And make containers more secure when using nosuid,nodev,noexec on system mount points.

and we don’t have to make hacks in /etc/rc.local to start container anymore.

So the correct mount points should be:

mkdir -p /usr/lib/lxc/rootfs/proc
mount -o rw,nosuid,nodev,noexec,relatime -t proc proc /usr/lib/lxc/rootfs/proc
mkdir -p /usr/lib/lxc/rootfs/sys
mount -o rw,nosuid,nodev,noexec,relatime -t sysfs sysfs /usr/lib/lxc/rootfs/sys

Put that in your /etc/rc.local. The most important part here is relatime. If we use noatime here the container won’t start because LXC is vulnerable to kernel overmounting with different options. It is fixed already in lxc 5.0.1 but TOS 6.4.1 is still using lxc 4.0.12. But 4.0.12 also brings a fix for:

That I encountered in the past.

Ok now another problem creating a container… So linuxcontainers.org stopped releasing containers for armv7 (armhf) architecture.

So we have to use Turris repo to get containers. There still a bug in the repo itself:

So we cannot simply use template download to get images but have to use workaround figured out by aspadini:
https://www.reddit.com/r/Turris/comments/14qpfq8/lxc_images_download
We have to get a rootfs and metadata somewhere on /srv by simply downloading files from repo example container to disk and then use command mentioned on Reddit to create the container.

For workaround see: [TOS 6.4.1] Summary on how to run unprivileged LXC containers (04/08/2023) - #4 by AreYouLoco - SW tweaks - Turris forum

And voila!

That should be all thats necessary…

7 Likes

Great stuff @AreYouLoco, if interested I also have a workaround to continue to manage LXC containers through LuCI and the Turris repo, it requires some extra work but still… You can reach out to me directly if interested.

1 Like

@aspadini Maybe you could share it here so more interested people can benefit from this knowledge :grin:

5 Likes

I believe there are some modifications needed in /usr/share/lxc/templates/lxc-download. Could you share the mystery?

Edit: I figured it out: Simply change DOWNLOAD_MODE="user" to DOWNLOAD_MODE="system" at lines 202 and 205 and you can use template download also for unprivileged containers and Turris repo.

3 Likes

Apologies for the delay with the replies.

Before providing workarounds about the Turris LXC images repository usage for unprivileged containers in LuCI, let’s first understand why this is not working.

In brief:

  1. LXC templates such as lxc-download or lxc-local are shell scripts used by lxc-create to generate containers root filesystem, notably lxc-download uses a specified download server (LXC images repository) to get the necessary archives (e.g. rootfs.tar.xz, meta.tar.xz)
  2. As per the lxc-download template specifications, a download server can have two different indexes (or images lists), one for privileged containers (index-system file) the other for unprivileged containers (index-user file), the correct index-list is automatically selected based on LXC configuration in use
  3. The Turris LXC images repository only contains an index for privileged containers (index-system file), this causes the lxc-download template to fail when trying to create an unprivileged container as the required index file is not available on the specified server

With that being said, the best solution would be to have the Turris team to publish an appropriate index for unprivileged containers, awaiting the permanent fix there are several workarounds that can be applied, here below you can find just some proposals.

1 Create your own repository

The instructions to achieve this goes beyond the scope of this post

2 Create a custom copy of the Turris repository

wget --mirror --convert-links --adjust-extension --page-requisites --reject .html -e robots=off --random-wait --no-verbose --include-directories="/lxc" --user-agent="lxc/4.0.12 compat:7" https://repo.turris.cz/lxc/ -P /your/preferred/path/

You can then create your own index file for unprivileged containers, for example, assuming we want to keep the same list for both privileged and unprivileged containers

cp /your/preferred/path/repo.turris.cz/lxc/meta/1.0/index-system /your/preferred/path/repo.turris.cz/lxc/meta/1.0/index-user

Once you’ve done this you can create a specific lighttpd configuration to host your custom copy of the Turris repository, for instance you can create a /etc/lighttpd/conf.d/99-custom-lxc-repo.conf file as follow:

# lighttpd include file for LXC images repository
 
 $HTTP["url"] =~ "^/lxc" {
 	alias.url = ( "/lxc/" => "/your/preferred/path/repo.turris.cz/lxc/" )
 }

Once the new configuration file is created you’ll have to restart lighttpd (service lighttpd restart) and point the Containers URL under LuCI - Services - LXC Containers to your localhost, your /etc/config/lxc file should then look similar to the following:

config lxc 'lxc'
	option min_space '100000'
	option min_temp '100000'
	option url 'localhost/lxc'
	option ssl_enabled '1'

Please note this requires a proper certificate file configured on the lighttpd web server, wget must be able to download the required files upon server’s certificate verification. This configuration is beyond the scope of this post.

3 Redirect requests

This workaround is similar to the previous one with the exception that in this scenario we’ll not create a copy of the Turris repository but simply leverage on lighttpd to redirect the requests to the Turris repository, notably we’ll intercept requests for unprivileged containers (index-user file) and redirect those towards another index file, in this case the one for privileged containers (index-system file). This is a task more suited for a proxy server rather than lighttpd but still it’s just another workaround.

In order to do so, you can create a /etc/lighttpd/conf.d/99-custom-lxc-repo.conf file as follow:

# lighttpd include file for Turris LXC images repository

$HTTP["url"] =~ "^/lxc" {
  url.redirect = ( ".*/lxc/(.*)" => "https://repo.turris.cz/lxc/$1" )
}

$HTTP["url"] =~ "^/lxc/meta/1.0/index-user" {
  url.redirect = ( ".*" => "https://repo.turris.cz/lxc/meta/1.0/index-system" )
}

Once the new configuration file is created you’ll have to restart lighttpd (service lighttpd restart) and point the Containers URL under LuCI - Services - LXC Containers to your localhost, your /etc/config/lxc file should then look similar to the following:

config lxc 'lxc'
	option min_space '100000'
	option min_temp '100000'
	option url 'localhost/lxc'
	option ssl_enabled '1'

Please note this requires a proper certificate file configured on the lighttpd web server, wget must be able to download the required files upon server’s certificate verification. This configuration is beyond the scope of this post.

I hope this post provided a better understanding of the issue and how this should be addressed, please note the workarounds proposed are just examples on how to bypass the issue, feel free to reach out in case of any doubt or questions.

1 Like

@aspadini out of curiosity. Which workaround are you using?

For the time being I’m using all the workarounds proposed giving the priority to #3 Redirect requests

Yeah you were using regular expressions. Respect. I once went on the route to learn that properly but well. Not yet!

Seems I get some permission errors when trying to download an image:

# lxc-create --name innen --bdev best --template download -- --dist Ubuntu --release 23.04 --arch armv7l

Downloading the image index
Downloading the rootfs
Downloading the metadata
The image cache is now ready
Unpacking the rootfs
tar: dev/console: Cannot mknod: Operation not permitted
tar: dev/full: Cannot mknod: Operation not permitted
tar: dev/null: Cannot mknod: Operation not permitted
tar: dev/ptmx: Cannot mknod: Operation not permitted
tar: dev/random: Cannot mknod: Operation not permitted
tar: dev/tty: Cannot mknod: Operation not permitted
tar: dev/urandom: Cannot mknod: Operation not permitted
tar: dev/zero: Cannot mknod: Operation not permitted
tar: Exiting with failure status due to previous errors
lxc-create: innen: lxccontainer.c: create_run_template: 1627 Failed to create container from template
lxc-create: innen: tools/lxc_create.c: main: 317 Failed to create container innen

Did it work for you out of the box? I freshly formatted my USB-Storage and afterwards followed your instructions.

I guess your USB storage is mounted with “nodev”. Did you set it up with Reforis plugin?

Yes, I used reForis to set up the drive (format + mount). Screenshot from luci:

/etc/config/fstab

config global
	option anon_swap '0'
	option anon_mount '0'
	option auto_swap '1'
	option auto_mount '1'
	option delay_root '5'
	option check_fs '0'

config mount
	option target '/srv'
	option uuid 'f07250c4-e1bc-42e7-988e-13f7d5a39f57'
	option enabled '0'

Post what you get from mount | grep sda

# mount | grep sda
/dev/sda on /srv type btrfs (rw,relatime,space_cache,subvolid=5,subvol=/)

Disable automounting the filesystems in LuCI and reboot and post output of that command again. What I think is happening is that LuCI is mounting the FS before/after Storage plugin with default options. Also ommit the --bdev in lxc-create and try again or try --bdev btrfs

You were right regarding the system automount.

Output after reboot:

# mount | grep sda
/dev/sda on /srv type btrfs (rw,noatime,space_cache,subvolid=256,subvol=/@)

Result is the same when leaving out --bdev or trying with --bdev btrfs.

I created an issue on gitlab for this (Standard storage plugin mount options don't work with lxc (#17) · Issues · Turris / reForis / Storage Plugin · GitLab) - I don’t like standard functionality (reForis storage plugin) not working with advertised functionality (lxc for Turris Omnia). Its more than overdue to have an official (and for each version tested) how-to on lxc-handling (creation, maintenance).

btw: If repos fail we can also go with local installation (which fails with the same erros in my case):

lxc-create --name innen --template local -- --fstree /srv/lxc/rootfs.tar.xz --metadata /srv/lxc/meta.tar.xz

@ssdnvv paste config of your container. Maybe you are missing mknod capability

~# cat /etc/lxc/default.conf
lxc.net.0.type = veth
lxc.net.0.link = br-lan
lxc.net.0.flags = up
lxc.net.0.name = eth0
# # Some workarounds
lxc.include = /usr/share/lxc/config/common.conf
lxc.hook.start-host = /usr/share/lxc/hooks/systemd-workaround
# Template to generate fixed MAC address
lxc.net.0.hwaddr = x2:xx:xx:xx:xx:xx
lxc.idmap = u 0 100000 65536
lxc.idmap = g 0 100000 65536
~# cat /etc/lxc/lxc.conf
lxc.lxcpath = /srv/lxc
~# cat /usr/share/lxc/config/common.conf
# Default configuration shared by all containers

# Setup the LXC devices in /dev/lxc/
lxc.tty.dir = lxc

# Allow for 1024 pseudo terminals
lxc.pty.max = 1024

# Setup 4 tty devices
lxc.tty.max = 4

# Drop some harmful capabilities
lxc.cap.drop = mac_admin mac_override sys_time sys_module sys_rawio

# Ensure hostname is changed on clone
lxc.hook.clone = /usr/share/lxc/hooks/clonehostname

# Default unified cgroup configuration
#
# CGroup allowlist
lxc.cgroup2.devices.deny = a
## Allow any mknod (but not reading/writing the node)
lxc.cgroup2.devices.allow = c *:* m
lxc.cgroup2.devices.allow = b *:* m
## Allow specific devices
### /dev/null
lxc.cgroup2.devices.allow = c 1:3 rwm
### /dev/zero
lxc.cgroup2.devices.allow = c 1:5 rwm
### /dev/full
lxc.cgroup2.devices.allow = c 1:7 rwm
### /dev/tty
lxc.cgroup2.devices.allow = c 5:0 rwm
### /dev/console
lxc.cgroup2.devices.allow = c 5:1 rwm
### /dev/ptmx
lxc.cgroup2.devices.allow = c 5:2 rwm
### /dev/random
lxc.cgroup2.devices.allow = c 1:8 rwm
### /dev/urandom
lxc.cgroup2.devices.allow = c 1:9 rwm
### /dev/pts/*
lxc.cgroup2.devices.allow = c 136:* rwm
### fuse
lxc.cgroup2.devices.allow = c 10:229 rwm

# Setup the default mounts
lxc.mount.auto = cgroup:mixed proc:mixed sys:mixed
lxc.mount.entry = /sys/fs/fuse/connections sys/fs/fuse/connections none bind,optional 0 0

# Block some syscalls which are not safe in privileged
# containers
lxc.seccomp.profile = /usr/share/lxc/config/common.seccomp

# Lastly, include all the configs from /usr/share/lxc/config/common.conf.d/
lxc.include = /usr/share/lxc/config/common.conf.d/

Edit: this is the standard config except the lines for subuid and subgid.

This works for me. Weird…

You have the very same mounting and lxc options?
Any further ideas?