LXC container CGROUP problem

blbeczech82 · July 15, 2020, 8:47am

After migration from TOS4 to TOS5 I have problem with my Debian Buster LXC running Nextcloud.
MariaDB fails to start due to cgroup problems.
I even tried to create fresh new buster container but it behaves exactly same.
Anyone can help?

Migated LXC (Nextcloud):

root@turris:~# lxc-start -n nextcloud -F
lxc-start: nextcloud: utils.c: safe_mount: 1179 No such file or directory - Failed to mount "/usr/lib/lxc/rootfs/proc/tty" onto "/usr/lib/lxc/rootfs/proc/sys/net"
                                                                                                                                                                  systemd 241 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid)
Detected virtualization lxc.
Detected architecture arm.

Welcome to Debian GNU/Linux 10 (buster)!

Set hostname to <nextcloud>.
Failed to attach 1 to compat systemd cgroup /init.scope: No such file or directory
Couldn't move remaining userspace processes, ignoring: Input/output error
File /lib/systemd/system/systemd-journald.service:12 configures an IP firewall (IPAddressDeny=any), but the local system does not support BPF/cgroup based firewalling.
Proceeding WITHOUT firewalling in effect! (This warning is only shown for the first loaded unit using IP firewalling.)
/lib/systemd/system/mariadb.service:16: Failed to parse protect system value, ignoring: 
/lib/systemd/system/mariadb.service:18: Failed to parse protect home value, ignoring: 
[  OK  ] Listening on Network Service Netlink Socket.
[  OK  ] Started Forward Password Requests to Wall Directory Watch.
[  OK  ] Listening on Journal Socket.
[  OK  ] Created slice system-container\x2dgetty.slice.
Failed to attach 15 to compat systemd cgroup /system.slice/dev-mqueue.mount: No such file or directory
         Mounting POSIX Message Queue File System...
Failed to attach 15 to compat systemd cgroup /system.slice/dev-mqueue.mount: No such file or directory
[  OK  ] Reached target Remote File Systems.
[  OK  ] Created slice User and Session Slice.
[  OK  ] Reached target Slices.
         Starting Remount Root and Kernel File Systems...
[  OK  ] Listening on Journal Socket (/dev/log).
Failed to attach 17 to compat systemd cgroup /system.slice/systemd-journald.service: No such file or directory
         Starting Journal Service...
[  OK  ] Reached target Swap.
[  OK  ] Created slice system-getty.slice.
[  OK  ] Listening on initctl Compatibility Named Pipe.
Failed to attach 17 to compat systemd cgroup /system.slice/systemd-journald.service: No such file or directory
[  OK  ] Started Dispatch Password Requests to Console Directory Watch.
[  OK  ] Reached target Local Encrypted Volumes.
[  OK  ] Reached target Paths.
         Starting Helper to synchronize boot up for ifupdown...
[  OK  ] Mounted POSIX Message Queue File System.
[  OK  ] Started Remount Root and Kernel File Systems.
         Starting Create System Users...
[  OK  ] Started Journal Service.
         Starting Flush Journal to Persistent Storage...
[  OK  ] Started Helper to synchronize boot up for ifupdown.
[  OK  ] Started Create System Users.
         Starting Create Static Device Nodes in /dev...
[  OK  ] Started Flush Journal to Persistent Storage.
[  OK  ] Started Create Static Device Nodes in /dev.
         Starting Network Service...
[  OK  ] Reached target Local File Systems (Pre).
[  OK  ] Reached target Local File Systems.
         Starting Create Volatile Files and Directories...
         Starting Raise network interfaces...
[  OK  ] Started Create Volatile Files and Directories.
[  OK  ] Reached target System Time Synchronized.
         Starting Update UTMP about System Boot/Shutdown...
[  OK  ] Started Network Service.
         Starting Wait for Network to be Configured...
         Starting Network Name Resolution...
[  OK  ] Started Update UTMP about System Boot/Shutdown.
[  OK  ] Reached target System Initialization.
[  OK  ] Started Daily apt download activities.
[  OK  ] Started Daily Cleanup of Temporary Directories.
[  OK  ] Started Daily apt upgrade and clean activities.
[  OK  ] Listening on D-Bus System Message Bus Socket.
[  OK  ] Listening on Avahi mDNS/DNS-SD Stack Activation Socket.
[  OK  ] Reached target Sockets.
[  OK  ] Started Clean PHP session files every 30 mins.
[  OK  ] Reached target Basic System.
         Starting Avahi mDNS/DNS-SD Stack...
         Starting Login Service...
[  OK  ] Started D-Bus System Message Bus.
         Starting WPA supplicant...
[  OK  ] Started Regular background program processing daemon.
         Starting Modem Manager...
[  OK  ] Started Daily rotation of log files.
[  OK  ] Reached target Timers.
         Starting Rotate log files...
         Starting Clean php session files...
[  OK  ] Started Login Service.
[  OK  ] Started WPA supplicant.
[  OK  ] Started Avahi mDNS/DNS-SD Stack.
[  OK  ] Started Wait for Network to be Configured.
[  OK  ] Started Modem Manager.
[  OK  ] Started Network Name Resolution.
[  OK  ] Reached target Host and Network Name Lookups.
[  OK  ] Started Raise network interfaces.
[  OK  ] Reached target Network.
[  OK  ] Reached target Network is Online.
[  OK  ] Started ClamAV virus database updater.
         Starting The Apache HTTP Server...
         Starting Advanced key-value store...
         Starting coTURN STUN/TURN Server...
         Starting Permit User Sessions...
         Starting OpenBSD Secure Shell server...
         Starting MariaDB 10.3.22 database server...
         Starting Daily apt download activities...
[  OK  ] Started Permit User Sessions.
[  OK  ] Started Console Getty.
[  OK  ] Started Container Getty on /dev/pts/2.
[  OK  ] Started Container Getty on /dev/pts/3.
[  OK  ] Started Container Getty on /dev/pts/0.
[  OK  ] Started Container Getty on /dev/pts/1.
[  OK  ] Reached target Login Prompts.
[  OK  ] Started OpenBSD Secure Shell server.
[  OK  ] Started Advanced key-value store.
[  OK  ] Started The Apache HTTP Server.
[  OK  ] Started Clean php session files.
[  OK  ] Started coTURN STUN/TURN Server.
[FAILED] Failed to start MariaDB 10.3.22 database server.
See 'systemctl status mariadb.service' for details.
         Starting LSB: exim Mail Transport Agent...

Debian GNU/Linux 10 nextcloud console

nextcloud login:

new test buster LXC:

root@turris:~# lxc-start -n test -F
lxc-start: test: utils.c: safe_mount: 1179 No such file or directory - Failed to mount "/usr/lib/lxc/rootfs/proc/tty" onto "/usr/lib/lxc/rootfs/proc/sys/net"
                                                                                                                                                             systemd 241 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid)
Detected virtualization lxc.
Detected architecture arm.

Welcome to Debian GNU/Linux 10 (buster)!

Set hostname to <LXCNAME>.
Failed to attach 1 to compat systemd cgroup /init.scope: No such file or directory
Couldn't move remaining userspace processes, ignoring: Input/output error
File /lib/systemd/system/systemd-journald.service:12 configures an IP firewall (IPAddressDeny=any), but the local system does not support BPF/cgroup based firewalling.
Proceeding WITHOUT firewalling in effect! (This warning is only shown for the first loaded unit using IP firewalling.)
[  OK  ] Listening on Journal Socket.
[  OK  ] Started Forward Password Requests to Wall Directory Watch.
         Starting Helper to synchronize boot up for ifupdown...
[  OK  ] Listening on initctl Compatibility Named Pipe.
[  OK  ] Created slice system-container\x2dgetty.slice.
[  OK  ] Created slice User and Session Slice.
         Starting Remount Root and Kernel File Systems...
[  OK  ] Reached target Swap.
[  OK  ] Reached target Slices.
[  OK  ] Reached target Remote File Systems.
[  OK  ] Started Dispatch Password Requests to Console Directory Watch.
[  OK  ] Reached target Local Encrypted Volumes.
[  OK  ] Reached target Paths.
[  OK  ] Listening on Journal Socket (/dev/log).
Failed to attach 17 to compat systemd cgroup /system.slice/systemd-journald.service: No such file or directory
         Starting Journal Service...
Failed to attach 19 to compat systemd cgroup /system.slice/dev-mqueue.mount: No such file or directory
Failed to attach 17 to compat systemd cgroup /system.slice/systemd-journald.service: No such file or directory
Failed to attach 19 to compat systemd cgroup /system.slice/dev-mqueue.mount: No such file or directory
         Mounting POSIX Message Queue File System...
[  OK  ] Created slice system-getty.slice.
[  OK  ] Started Helper to synchronize boot up for ifupdown.
[  OK  ] Started Remount Root and Kernel File Systems.
[  OK  ] Mounted POSIX Message Queue File System.
         Starting Create System Users...
[  OK  ] Started Journal Service.
         Starting Flush Journal to Persistent Storage...
[  OK  ] Started Create System Users.
         Starting Create Static Device Nodes in /dev...
[  OK  ] Started Flush Journal to Persistent Storage.
[  OK  ] Started Create Static Device Nodes in /dev.
[  OK  ] Reached target Local File Systems (Pre).
[  OK  ] Reached target Local File Systems.
         Starting Raise network interfaces...
         Starting Create Volatile Files and Directories...
[  OK  ] Started Create Volatile Files and Directories.
         Starting Update UTMP about System Boot/Shutdown...
[  OK  ] Reached target System Time Synchronized.
[  OK  ] Started Update UTMP about System Boot/Shutdown.
[  OK  ] Reached target System Initialization.
[  OK  ] Started Daily Cleanup of Temporary Directories.
[  OK  ] Listening on D-Bus System Message Bus Socket.
[  OK  ] Reached target Sockets.
[  OK  ] Started Daily apt download activities.
[  OK  ] Reached target Basic System.
         Starting Login Service...
[  OK  ] Started D-Bus System Message Bus.
[  OK  ] Started Daily apt upgrade and clean activities.
[  OK  ] Reached target Timers.
[  OK  ] Started Login Service.
[  OK  ] Started Raise network interfaces.
[  OK  ] Reached target Network.
         Starting Daily apt download activities...
         Starting Permit User Sessions...
[  OK  ] Started Permit User Sessions.
[  OK  ] Started Container Getty on /dev/pts/0.
[  OK  ] Started Container Getty on /dev/pts/3.
[  OK  ] Started Container Getty on /dev/pts/2.
[  OK  ] Started Console Getty.
[  OK  ] Started Container Getty on /dev/pts/1.
[  OK  ] Reached target Login Prompts.
[  OK  ] Reached target Multi-User System.
[  OK  ] Reached target Graphical Interface.
         Starting Update UTMP about System Runlevel Changes...
[  OK  ] Started Update UTMP about System Runlevel Changes.
[  OK  ] Started Daily apt download activities.
         Starting Daily apt upgrade and clean activities...

Debian GNU/Linux 10 LXCNAME console

LXCNAME login:

Maxmilian_Picmaus · July 15, 2020, 9:21pm

There is available lxc-cgroup, but maybe your issue is related to systemd in lxc is not able to safely move userspace processes (like mount) due missing/wrong rights on host (but this is reported for kernel 5.5.x after upgrade from 5.4.x , TOS5 has 4.4.x right ?).
It seems that Debian Buster is now using apparmor, causing some issues to services managed by systemd (apache2,mariadb,mount…)

Enabling nesting for the container will fix this issue for now, until we get an upstream fix.
lxc.apparmor.profile = generated
lxc.apparmor.allow_nesting = 1

I am not using nextcloud anymore, so this is wild hint. but i hope it will help

EDIT: maybe just reboot is needed …

blbeczech82 · July 17, 2020, 7:34am

Thank you for hint. Unfortunately this is not working.

root@turris:~# lxc-start -n nextcloud -F
lxc-start: nextcloud: confile.c: parse_line: 2262 Unknown configuration key "lxc.apparmor.allow_nesting"
lxc-start: nextcloud: parse.c: lxc_file_for_each_line_mmap: 142 Failed to parse config file "/srv/lxc/nextcloud/config" at line "lxc.apparmor.allow_nesting = 1"
Failed to load config for nextcloud
lxc-start: nextcloud: tools/lxc_start.c: main: 264 Failed to create lxc_container
root@turris:~#

Could you be so kind and try to download and run Buster container if you will have the same problem or it is on my Omnia only?

Thank you…

Maxmilian_Picmaus · July 17, 2020, 9:38am

I am getting

error

lxc-create -n bustertester -t download
Setting up the GPG keyring
ERROR: Unable to fetch GPG key from keyserver.

Trying with lxc-debian template does not help either (different error). So i have to check and fix my lxc stuff (on the other hand my all three containers are up and running …)

Maxmilian_Picmaus · July 17, 2020, 11:34am

I sorted out the lxc-templates. Now i am able to create and start debian.buster with this message

lxc-start 1594985310.318 ERROR    lxc_cgfs - cgfs.c:handle_cgroup_settings:2091 - Resource busy - failed to set memory.use_hierarchy to 1; continuing
  lxc-start 1594985310.318 ERROR    lxc_cgfs - cgfs.c:handle_cgroup_settings:2091 - Resource busy - failed to set memory.use_hierarchy to 1; continuing

but seems i can lxc-attach it stop/start it…

blbeczech82 · July 30, 2020, 6:51am

Thank you to try. Seems you have cgroups problem too with buster.
@cynerd don’t you have any idea please?