Turris 1.x and LXC on Turris OS 4.0+

Software Turris OS 4.x
1

This is related to Turris Omnia as there is arm architecture. I am on blue Turris 1.0 with powerpcspe architecture and under current TOS 5.0 HBK it is unable to start LXC containers that had been running under TOS 3 as every attempt to start it end with this:

root@turris:~# lxc-start -n pihole -F
Failed to lookup module alias 'autofs4': Function not implemented
Failed to lookup module alias 'unix': Function not implemented
Failed to mount cgroup at /sys/fs/cgroup/systemd: Operation not permitted
[!!!!!!] Failed to mount API filesystems.
Exiting PID 1...
root@turris:~#

My updated config is 

root@turris:/srv/lxc/pihole# cat config
lxc.uts.name = pihole
lxc.rootfs.path = /srv/lxc/pihole/rootfs
lxc.start.auto=1
lxc.start.order=8
lxc.start.delay=60
# Apparently, cgroup for `/dev/snd`
lxc.cgroup.devices.allow = c 116:* rwm
lxc.mount.entry=devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000
lxc.mount.entry = /dev/snd dev/snd none bind,optional,create=dir
lxc.mount.entry = /dev/net dev/net none bind,optional,create=dir
lxc.mount.entry = /dev/input dev/input none bind,optional,create=dir
lxc.mount.entry=proc /proc proc nodev,noexec,nosuid 0 0
lxc.mount.entry=tmpfs /dev/shm tmpfs  defaults 0 0
lxc.mount.entry= /data data none bind.ro 0.0
lxc.mount.entry= /mnt/flash mnt/flash none bind.ro 0.0
lxc.mount.entry= /lib/modules lib/modules none bind.ro 0.0
lxc.pty.max=1024
lxc.apparmor.allow_incomplete = 1

# Set up /dev for systemd
lxc.autodev=1

# networking
lxc.net.0.type = veth
lxc.net.0.flags = up
lxc.net.0.link = br-lan
lxc.net.0.hwaddr = 00:11:22:00:01:02
lxc.net.0.ipv4.address = 192.168.0.2/24
lxc.net.0.ipv4.gateway = 192.168.0.1
lxc.seccomp.profile =

No one replied my question so far if it is possible to run LXC on Turris 1.0 with powerpcspe architecture so I don’t know it it is possible or not or what has to be done even all LXC packages are installed. But repository for popwepcspe been dropped so even to create new cotainer it probably will not be possible or I don’t know how to do it

2

Your post is identical to which one you posted to Turris OS 4.0.5 is released out!. Which makes threads longer. Use [code][/code].

3

Thank you at least for this reply, I updated both posts with [code] as you recommended. But so far I still seek answer as my previous post had been unanswered for 16 days.

Maybe no one tried LXC under old blue Turris 1.0 but under situation all packages in place I would expect it to start after updated config which I had done and as noted I get same results on TOS4 as well as on TOS5 which leads me to doubts if it is possible at all.

If LXC on TOS5 on powerpcspe is not supported so far I would just deal with it some other way and stop to try but so far I don’t know. What I had done is that just copied the working rootfs from previous TOS3 snapshot and updated config. Another option is to try create new container but as powerpcspe debian has already dropped support I am not sure that it will be possible to create new debian container under TOS5 at all or not. I am aware that TOS5 is still experimental but probably more people will interested in answer if it is possible or not. If not then I or other people may give up effort to migrate to TOS5 altogether or choose different approach to upgrade.

4 
Failed to mount cgroup at /sys/fs/cgroup/systemd: Operation not permitted
[!!!!!!] Failed to mount API filesystems.

I saw this message already several times. It occures in case you let LXC mount automatically the cgroup:mixed i.e. like:

lxc.mount.auto = proc:mixed sys:ro cgroup:mixed

Somehow LXC is too late in this scenario to mount it, which leads to the above API issue.
As a consequence you need the workaround that you can find also in the internet:

lxc.mount.auto = proc:mixed sys:ro
lxc.mount.entry = tmpfs dev/shm tmpfs rw,nosuid,nodev,create=dir 0 0
lxc.mount.entry = tmpfs run tmpfs rw,nosuid,nodev,mode=755,create=dir 0 0
lxc.mount.entry = tmpfs run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,create=dir 0 0
lxc.mount.entry = tmpfs run/user tmpfs rw,nosuid,nodev,mode=755,size=50m,create=dir 0 0
lxc.mount.entry = tmpfs sys/fs/cgroup tmpfs rw,nosuid,nodev,create=dir 0 
lxc.hook.mount = /srv/lxc/patine/hook-cgroups.sh # FIXME

while the hook-cgroups.sh looks as follows:

#!/bin/sh
mkdir -p ${LXC_ROOTFS_MOUNT}/sys/fs/cgroup/systemd
mount cgroup ${LXC_ROOTFS_MOUNT}/sys/fs/cgroup/systemd \
  -t cgroup \
  -o rw,nosuid,nodev,noexec,relatime,xattr,name=systemd

EDIT: If you search within the internet, search for LXC CAP_SYS_ADMIN. That is where I encountered the issue, while trying to enable it.

5

Thank you very much, I will try it as soon as possible and report the results if I succeed.

After modify config this way as you recommended:

root@turris:/srv/lxc/pihole# cat config
lxc.uts.name = pihole
lxc.rootfs.path = /srv/lxc/pihole/rootfs
lxc.start.auto=1
lxc.start.order=8
lxc.start.delay=60
# Apparently, cgroup for `/dev/snd`
lxc.mount.auto = proc:mixed sys:ro
lxc.mount.entry = tmpfs dev/shm tmpfs rw,nosuid,nodev,create=dir 0 0
lxc.mount.entry = tmpfs run tmpfs rw,nosuid,nodev,mode=755,create=dir 0 0
lxc.mount.entry = tmpfs run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,create=dir 0 0
lxc.mount.entry = tmpfs run/user tmpfs rw,nosuid,nodev,mode=755,size=50m,create=dir 0 0
lxc.mount.entry = tmpfs sys/fs/cgroup tmpfs rw,nosuid,nodev,create=dir 0
lxc.hook.mount = /srv/lxc/pihole/hook-cgroups.sh #FIXME
lxc.cgroup.devices.allow = c 116:* rwm
lxc.mount.entry=devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000
lxc.mount.entry = /dev/snd dev/snd none bind,optional,create=dir
lxc.mount.entry = /dev/net dev/net none bind,optional,create=dir
lxc.mount.entry = /dev/input dev/input none bind,optional,create=dir
lxc.mount.entry=proc /proc proc nodev,noexec,nosuid 0 0
lxc.mount.entry= /data data none bind.ro 0.0
lxc.mount.entry= /mnt/flash mnt/flash none bind.ro 0.0
lxc.mount.entry= /lib/modules lib/modules none bind.ro 0.0
lxc.pty.max=1024
lxc.apparmor.allow_incomplete = 1

# Set up /dev for systemd
lxc.autodev=1

# networking
lxc.net.0.type = veth
lxc.net.0.flags = up
lxc.net.0.link = br-lan
lxc.net.0.hwaddr = 00:11:22:00:01:02
lxc.net.0.ipv4.address = 192.168.0.2/24
lxc.net.0.ipv4.gateway = 192.168.0.1
lxc.seccomp.profile =
root@turris:/srv/lxc/pihole#

The error is different:

root@turris:~# lxc-start -n pihole -F
lxc-start: pihole: utils.c: safe_mount: 1179 No such file or directory - Failed to mount "/usr/lib/lxc/rootfs/proc/tty" onto "/usr/lib/lxc/rootfs/proc/sys/net"
                                                                                                                                                               lxc-start: pihole: conf.c: run_buffer: 335 Script exited with status 32
                                     lxc-start: pihole: conf.c: lxc_setup: 3589 Failed to run mount hooks
                                                                                                         lxc-start: pihole: start.c: do_start: 1263 Failed to setup container "pihole"
                                                                                                                                                                                      lxc-start: pihole: sync.c: __sync_wait: 62 An error occurred in another process (expected sequence number 5)
                                                                                                 lxc-start: pihole: start.c: __lxc_start: 1939 Failed to spawn container "pihole"
                                                                                                                                                                                 lxc-start: pihole: tools/lxc_start.c: main: 330 The container failed to start
lxc-start: pihole: tools/lxc_start.c: main: 336 Additional information can be obtained by setting the --logfile and --logpriority options
root@turris:~#

I set the log file but not much other information in it

lxc-start pihole 20200328104719.758 ERROR    utils - utils.c:safe_mount:1179 - No such file or directory - Failed to mount "/usr/lib/lxc/rootfs/proc/tty" onto "/usr/lib/lxc/rootfs/proc/sys/net"
lxc-start pihole 20200328104719.775 ERROR    conf - conf.c:run_buffer:335 - Script exited with status 32
lxc-start pihole 20200328104719.775 ERROR    conf - conf.c:lxc_setup:3589 - Failed to run mount hooks
lxc-start pihole 20200328104719.775 ERROR    start - start.c:do_start:1263 - Failed to setup container "pihole"
lxc-start pihole 20200328104719.776 ERROR    sync - sync.c:__sync_wait:62 - An error occurred in another process (expected sequence number 5)
lxc-start pihole 20200328104719.777 ERROR    start - start.c:__lxc_start:1939 - Failed to spawn container "pihole"
lxc-start pihole 20200328104719.810 ERROR    lxc_start - tools/lxc_start.c:main:330 - The container failed to start
lxc-start pihole 20200328104719.810 ERROR    lxc_start - tools/lxc_start.c:main:336 - Additional information can be obtained by setting the --logfile and --logpriority options

also tried with

lxc.mount.auto = proc:mixed sys:ro cgroup:mixed

with same results

6

I managed to get it working with configuration form here: Debian LXC - CGroupV2.
I added this line to the config file

 lxc.init.cmd = /sbin/init systemd.unified_cgroup_hierarchy=1

and it seems to be working.
My config is now

lxc.uts.name = debian1
lxc.rootfs.path  = /srv/lxc/debian1/rootfs
lxc.mount.entry=proc /proc proc nodev,noexec,nosuid 0 0
lxc.mount.entry=tmpfs /dev/shm tmpfs  defaults 0 0
lxc.pty.max=1024
 
# networking
lxc.net.0.type = veth
lxc.net.0.flags = up
lxc.net.0.link = br-lan
lxc.net.0.hwaddr = 00:11:22:33:44:55
lxc.net.0.ipv4.address = 192.168.1.10/24
lxc.net.0.ipv4.gateway = 192.168.1.1

lxc.init.cmd = /sbin/init systemd.unified_cgroup_hierarchy=1

Edit: I’m running TurrisOS 5.0 on Turris 1.1

7

Thank you very much will try it under TOS5 and TOS4 eventually, I would probably not find this solution myself and turris team is quiet about LXC on PowerPCSPE blue turris.

It will be great if I could make LXC working under TOS5 again as for me it is big obstacle from going forward with migration from TOS3

8

I can now confirm that I successfully started PIHOLE lxc container on blue Turris 1.0 with this config:

lxc.uts.name = pihole
lxc.rootfs.path = /srv/lxc/pihole/rootfs
lxc.start.auto=1
lxc.start.order=8
lxc.start.delay=60
lxc.mount.auto = proc:mixed sys:ro cgroup:mixed
lxc.cgroup.devices.allow = c 116:* rwm
lxc.mount.entry = proc /proc proc nodev,noexec,nosuid 0 0
lxc.mount.entry = tmpfs dev/shm tmpfs rw,nosuid,nodev,create=dir 0 0
lxc.mount.entry = tmpfs run tmpfs rw,nosuid,nodev,mode=755,create=dir 0 0
lxc.mount.entry = tmpfs run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,create=dir 0 0
lxc.mount.entry = tmpfs run/user tmpfs rw,nosuid,nodev,mode=755,size=50m,create=dir 0 0
lxc.mount.entry = tmpfs sys/fs/cgroup tmpfs rw,nosuid,nodev,create=dir 0
lxc.mount.entry = devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000
lxc.mount.entry = /dev/snd dev/snd none bind,optional,create=dir
lxc.mount.entry = /dev/net dev/net none bind,optional,create=dir
lxc.mount.entry = /dev/input dev/input none bind,optional,create=dir
lxc.mount.entry= /data data none bind.ro 0.0
lxc.mount.entry= /mnt/flash mnt/flash none bind.ro 0.0
lxc.mount.entry= /lib/modules lib/modules none bind.ro 0.0
lxc.pty.max=1024
lxc.apparmor.allow_incomplete = 1
lxc.autodev=1
lxc.net.0.type = veth
lxc.net.0.flags = up
lxc.net.0.link = br-lan
lxc.net.0.hwaddr = 00:11:22:00:01:02
lxc.net.0.ipv4.address = 192.168.0.2/24
lxc.net.0.ipv4.gateway = 192.168.0.1
lxc.seccomp.profile =
lxc.init.cmd = /sbin/init systemd.unified_cgroup_hierarchy=1

the problem was caused by line in my config that was fix and solution needed to start lxc container on master or x86 device:

#lxc.hook.mount = /srv/lxc/pihole/hook-cgroups.sh #FIXME

that was necessary to start LXC container on my another x86 device and lead to this script hook-cgroups.sh but this script is obviously not necessary for powerpcspe device like turris:

#!/bin/sh
mkdir -p ${LXC_ROOTFS_MOUNT}/sys/fs/cgroup/systemd
mount cgroup ${LXC_ROOTFS_MOUNT}/sys/fs/cgroup/systemd
-t cgroup
-o rw,nosuid,nodev,noexec,relatime,xattr,name=systemd

With this config LXC pihole container that I copied rootfs from TOS3 was able to boot on foreground even with some complaints like this:

root@turris:~# lxc-start -n pihole -F
lxc-start: pihole: utils.c: safe_mount: 1179 No such file or directory - Failed to mount “/usr/lib/lxc/rootfs/proc/tty” onto “/usr/lib/lxc/rootfs/proc/sys/net”
systemd 241 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid)
Detected virtualization lxc.
Detected architecture ppc.

Welcome to Debian GNU/Linux 10 (buster)!

Set hostname to .
Couldn’t move remaining userspace processes, ignoring: Input/output error
File /lib/systemd/system/systemd-journald.service:12 configures an IP firewall (IPAddressDeny=any), but the local system does not support BPF/cgroup based firewalling.
Proceeding WITHOUT firewalling in effect! (This warning is only shown for the first loaded unit using IP firewalling.)
/lib/systemd/system/lighttpd.service:6: PIDFile= references path below legacy directory /var/run/, updating /var/run/lighttpd.pid â /run/lighttpd.pid; please update the unit file accordingly.
[ OK ] Listening on initctl Compatibility Named Pipe.
[ OK ] Created slice system-getty.slice.
[ OK ] Reached target Swap.
[ OK ] Listening on Journal Socket (/dev/log).
[ OK ] Created slice User and Session Slice.
[ OK ] Reached target Slices.
[ OK ] Started Forward Password Requests to Wall Directory Watch.
[ OK ] Listening on Syslog Socket.
[ OK ] Started Dispatch Password Requests to Console Directory Watch.
[ OK ] Reached target Local Encrypted Volumes.
[ OK ] Reached target Paths.
[ OK ] Reached target Remote File Systems.
[ OK ] Listening on Journal Socket.
Starting Load Kernel Modules…
Starting Remount Root and Kernel File Systems…
Starting Set the console keyboard layout…
Starting Helper to synchronize boot up for ifupdown…
Starting Nameserver information manager…
Mounting POSIX Message Queue File System…
Mounting Kernel Debug File System…
Starting Journal Service…
[ OK ] Started Helper to synchronize boot up for ifupdown.
[ OK ] Started Load Kernel Modules.
[ OK ] Started Remount Root and Kernel File Systems.
[ OK ] Mounted POSIX Message Queue File System.
[ OK ] Mounted Kernel Debug File System.
[ OK ] Started Nameserver information manager.
Starting Create System Users…
Mounting FUSE Control File System…
Mounting Kernel Configuration File System…
[ OK ] Started Journal Service.
Starting Flush Journal to Persistent Storage…
[ OK ] Mounted FUSE Control File System.
[ OK ] Mounted Kernel Configuration File System.
[ OK ] Started Create System Users.
Starting Create Static Device Nodes in /dev…
[ OK ] Started Set the console keyboard layout.
[ OK ] Started Flush Journal to Persistent Storage.
[ OK ] Started Create Static Device Nodes in /dev.
[ OK ] Reached target Local File Systems (Pre).
[ OK ] Reached target Local File Systems.
Starting Create Volatile Files and Directories…
Starting Raise network interfaces…
Starting Set console font and keymap…
[ OK ] Started Set console font and keymap.
[ OK ] Started Create Volatile Files and Directories.
Starting Update UTMP about System Boot/Shutdown…
[ OK ] Reached target System Time Synchronized.
[ OK ] Started Raise network interfaces.
[ OK ] Reached target Network.
[ OK ] Reached target Network is Online.
[ OK ] Started Update UTMP about System Boot/Shutdown.
[ OK ] Reached target System Initialization.
[ OK ] Started Daily Cleanup of Temporary Directories.
[ OK ] Started Daily man-db regeneration.
[ OK ] Started Periodic ext4 Online Metadata Check for All Filesystems.
[ OK ] Started Daily rotation of log files.
[ OK ] Listening on D-Bus System Message Bus Socket.
[ OK ] Started Clean PHP session files every 30 mins.
[ OK ] Listening on Avahi mDNS/DNS-SD Stack Activation Socket.
[ OK ] Reached target Sockets.
[ OK ] Reached target Basic System.
Starting Permit User Sessions…
Starting LSB: pihole-FTL daemon…
Starting System Logging Service…
Starting OpenBSD Secure Shell server…
Starting Login Service…
Starting DHCP Client Daemon…
Starting Unbound DNS server…
Starting Remove Stale Online ext4 Metadata Check Snapshots…
Starting LSB: Start or stop stunnel 4.x (TLS tunnel for network daemons)…
Starting LSB: Start daemon at boot time…
[ OK ] Started Regular background program processing daemon.
Starting LSB: saslauthd startup script…
Starting LSB: service and resource monitoring daemon…
Starting Avahi mDNS/DNS-SD Stack…
Starting LSB: GNU inetutils syslog daemon…
[ OK ] Started D-Bus System Message Bus.
Starting Lighttpd Daemon…
[ OK ] Started Daily apt download activities.
[ OK ] Started Daily apt upgrade and clean activities.
[ OK ] Reached target Timers.
Starting LSB: LVM2 poll daemon…
[ OK ] Started System Logging Service.
[ OK ] Started Permit User Sessions.
[ OK ] Started DHCP Client Daemon.
[ OK ] Started LSB: GNU inetutils syslog daemon.
[ OK ] Started OpenBSD Secure Shell server.
[ OK ] Started LSB: saslauthd startup script.
[ OK ] Started LSB: service and resource monitoring daemon.
[ OK ] Started LSB: LVM2 poll daemon.
[ OK ] Started Console Getty.
[ OK ] Reached target Login Prompts.
[ OK ] Started Lighttpd Daemon.
[ OK ] Started Avahi mDNS/DNS-SD Stack.
[ OK ] Started Login Service.
[ OK ] Started Unattended Upgrades Shutdown.
[ OK ] Started Unbound DNS server.
[ OK ] Started Unbound DNS server via resolvconf.
[ OK ] Reached target Host and Network Name Lookups.
Starting LSB: exim Mail Transport Agent…
[ OK ] Started Remove Stale Online ext4 Metadata Check Snapshots.
[ OK ] Started LSB: exim Mail Transport Agent.

Debian GNU/Linux 10 pihole console

pihole login:

I think that this information will be usefull and handy for other blue Turries 1.0 users that would also like to migrate from TOS3 to TOS5 and run LXC containers.

It is bit sad that no one from the team replied to my original message saying that LXC container works under TOS5 and copied the working config like @kerby for more than two months. It will be important information for me that to run LXC on TOS5 is possible and I would expect some help from the team. Even now knowing it was kind of my fault using hack for x86 openwrt master to run lxc. But just knowing it works I may put more effort to investigate such issue but that’s life. Lucklily we have at least this forum with other users willing to help us with migration issues.

1 Like
9

I’m glad to hear that it’s working for you @Twinkie.
What sources are you using in /etc/apt/sources.list? I also tried to install pihole in Debian, but didn’t managed to get it to work, so I would be grateful if you could give me some tips or a link to a guide.
Also, could you please put the your output to CODE block - the post is quite long now.

10

As debian dropped support for powerpcspe there are only archive sources available now and it is those

deb https://snapshot.debian.org/archive/debian-ports/20190518T205337Z/ unstable main
deb https://snapshot.debian.org/archive/debian-ports/20190518T205337Z/ unreleased main
deb https://snapshot.debian.org/archive/debian-ports/20190518T205337Z/ experimental main

There is updated guide how to create container on this forum with this sources and also official or unofficial guide can be used for creating lxc debian container at first that with diffrerence that debian packages are downloaded from those snapshot.

I installed pihole then from guide here on turris doc webpage with small difference that i compile binary pihole-FTL from sources as from pihole website it is downloaded binaries for powerpc 32bit but not powerpcspe. Even that makes not much difference I compile it as when compiled by native toolchain it is more stable.

For this purpose I set up toolchain for openwrt in another lxc container as it is annoying to compile it on computer.

I can tar.gz. my pihole rootfs and put it somewhere for download if it helps you and also got several freshly created LXC rootfs for debian also that can be used for any purpose.

11

Thank you for the tips ant the list, with the updated list apt is finally working.
I will try to build the pihole-FTL myself, so that I can update the app when new versions come out.
If I fail with the compiling, then I would appreciate the archived rootfs. Thank you again for the help.

12

Well after create of pihole debian LXC container you can install pihole following guide here from curl part https://doc.turris.cz/doc/cs/public/pihole

It is better to dedicate separate LXC container for pihole and then use it as DNS server from pihole container IP address. I tweaked it to local dns server as well filling /etc/pihole/lan.list with local ip addreses of my nas, computers, phones and desktops

To make pihole-FTL binary from source you need to create toolchain first for TOS3 guide is here https://doc.turris.cz/doc/cs/howto/turris_os_build but it. I am not sure if it will create runable binary for TOS5 or if you need other toolchain.
It should be possible to use toolchain from openwrt master after you configure right powerpc target. It should produce working binary and you will not need some outdated obsolete linux distribution to compile it like for TOS3 toolchain that does not compile on recent debian buster that I use.

Even though I created openwrt toolchain for master and it should just work I was unable to create installable opkg package with it but had not put really much effort into that as when I was not able to make LXC on TOS5 working I postpone fiddling with that until resolving working LXC. But to create just pihole-FTL binary it should just works and then you just replace /usr/bin/pihole-FTL that was installed following guide.

Another option is to assembly toolchain for openwrt master or 19.07 in another debian LXC container on Turris directly. It takes long time and lot of tries of running make -j1 but if it is viable option at the end you will end up with working toolchain for TOS5 directly on Turris… Also big SDHC card is necessary for this as it download lot of sources that are used to build toolchain. I succesfully done that with TOS3. Once you got it the advantage is that you can compile small C/C++ projects on Turris itself and no need crosscompilation. It works great for some small projects and experiments and it is not needed to have all packages compiled and created. Just the toolchain part is enough.