LXC containers - how to restore them?

Guys about LXC running with your old LXC instance. First change the lxc.conf from the /srv/lxc to the path you used before.

In my case the path was /mnt/LXC referring to my mSATA ssd.

Do fix the mounting first. LXC directory in the mnt i created it with the mkdir command. (mkdir /mnt/LXC)

Although in luci it now recognized you having the LXC (sees the config file), it cannot start the LXC instance.

root@K-Router:/mnt/LXC/K-Router-LXC# lxc-start -n K-Router-LXC
The configuration file contains legacy configuration keys.
Please update your configuration file!

lxc-start: K-Router-LXC: lxccontainer.c: wait_on_daemonized_start: 760 Received container state "STOPPING" instead of "RUNNING"
lxc-start: K-Router-LXC: tools/lxc_start.c: main: 371 The container failed to start.
lxc-start: K-Router-LXC: tools/lxc_start.c: main: 373 To get more details, run the container in foreground mode.
lxc-start: K-Router-LXC: tools/lxc_start.c: main: 375 Additional information can be obtained by setting the --logfile and --logpriority options.

You need to update the config file of your lxc instance. You can find this in PATH/LXC Directory/config file.

In my case it was /mnt/LXC/K-Router-LXC/

Use this url to adjust your setting to version 2.1 as the version we used in TOS 3.X was based on LXC 1.5, which used different configuration settings.

https://linuxcontainers.org/lxc/manpages/man5/lxc.container.conf.5.html

Your config should look something like this.

New LXC config file
# Template used to create this container: /usr/share/lxc/templates/lxc-download
# Parameters passed to the template: --server repo.turris.cz/lxc --dist Ubuntu --release Bionic --arch armv7l
# For additional config options, please look at lxc.container.conf(5)
# Debian workaround
# Template to generate fixed MAC address

 Distribution configuration
lxc.arch = armv7l

 Container specific configuration
lxc.tty.max = 4
lxc.pty.max = 1024
lxc.rootfs.path = /mnt/LXC/K-Router-LXC/rootfs
lxc.uts.name = K-Router-LXC

 Network configuration
lxc.net.0.type = veth
lxc.net.0.link = br-lan
lxc.net.0.flags = up
lxc.net.0.name = eth0
lxc.net.0.ipv4.address = <IP ADDRESS>/24
lxc.net.0.ipv4.gateway = <GATEWAY>
lxc.net.0.script.up = /usr/share/lxc/hooks/tx-off
lxc.net.0.hwaddr = <hwaddr>

Although solved the first problem, now i am being stopped by another error.

      lxc-start K-Router-LXC 20190316122303.532 ERROR    lxc_conf - conf.c:run_buffer:438 - Script exited with status 127.

      lxc-start K-Router-LXC 20190316122303.590 ERROR    lxc_network - network.c:lxc_create_network_priv:2402 - Failed to create network device
      lxc-start K-Router-LXC 20190316122303.590 ERROR    lxc_start - start.c:lxc_spawn:1206 - Failed to create the network.
      lxc-start K-Router-LXC 20190316122303.590 ERROR    lxc_start - start.c:__lxc_start:1459 - Failed to spawn container "K-Router-LXC".
      lxc-start K-Router-LXC 20190316122303.590 ERROR    lxc_container - lxccontainer.c:wait_on_daemonized_start:760 - Received container state "STOPPING" instead of "RUNNING"
      lxc-start K-Router-LXC 20190316122303.591 ERROR    lxc_start_ui - tools/lxc_start.c:main:371 - The container failed to start.
      lxc-start K-Router-LXC 20190316122303.591 ERROR    lxc_start_ui - tools/lxc_start.c:main:373 - To get more details, run the container in foreground mode.
      lxc-start K-Router-LXC 20190316122303.591 ERROR    lxc_start_ui - tools/lxc_start.c:main:375 - Additional information can be obtained by setting the --logfile and --    logpriority options.

lxc-update-config detects any legacy configuration keys in the given config file and will replace them with the appropriate new configuration keys.

lxc-update-config will first create a backup of the old config file in the same directory and name it config.backup and then update the original config file in place. In case the update fails to apply or leads to an invalid config file that cannot be used to start a container users can either compare config with config.backup and try to manually repair any the invalid configuration keys or simply rollback to the legacy configuration file by copying config.backup to config.


You could, as suggested, run the container in foreground mode (perhaps also add --logfile) and see if additional information is displayed.

From the above bits is seems like a privilege issue in the namespace.

The problem is, there is no lxc-update-config package as far as i could search.

opkg find lxc
root@K-Router:~# opkg find lxc
lxc - 2.1.1-2.0 - LXC is the userspace control package for Linux Containers, a lightweight
 virtual system mechanism sometimes described as "chroot on steroids".
root@K-Router:~# opkg find lxc*
lxc - 2.1.1-2.0 - LXC is the userspace control package for Linux Containers, a lightweight
 virtual system mechanism sometimes described as "chroot on steroids".
lxc-attach - 2.1.1-2.1 - Utility lxc-attach from the LXC userspace tools
lxc-auto - 2.1.1-2.0 - LXC is the userspace control package for Linux Containers, a lightweight
 virtual system mechanism sometimes described as "chroot on steroids".
 This package adds and initscript for starting and stopping the containers
 on boot and shutdown.
lxc-autostart - 2.1.1-2.1 - Utility lxc-autostart from the LXC userspace tools
lxc-cgroup - 2.1.1-2.1 - Utility lxc-cgroup from the LXC userspace tools
lxc-checkconfig - 2.1.1-2.0 - Utility lxc-checkconfig from the LXC userspace tools
lxc-common - 2.1.1-2.0 - LXC common files
lxc-config - 2.1.1-2.1 - Utility lxc-config from the LXC userspace tools
lxc-configs - 2.1.1-2.0 - LXC virtual machine common config files
lxc-console - 2.1.1-2.1 - Utility lxc-console from the LXC userspace tools
lxc-copy - 2.1.1-2.1 - Utility lxc-copy from the LXC userspace tools
lxc-create - 2.1.1-2.1 - Utility lxc-create from the LXC userspace tools
lxc-destroy - 2.1.1-2.1 - Utility lxc-destroy from the LXC userspace tools
lxc-device - 2.1.1-2.1 - Utility lxc-device from the LXC userspace tools
lxc-execute - 2.1.1-2.1 - Utility lxc-execute from the LXC userspace tools
lxc-freeze - 2.1.1-2.1 - Utility lxc-freeze from the LXC userspace tools
lxc-hooks - 2.1.1-2.0 - LXC virtual machine hooks
lxc-info - 2.1.1-2.1 - Utility lxc-info from the LXC userspace tools
lxc-init - 2.1.1-2.1 - LXC Lua bindings
lxc-ls - 2.1.1-2.1 - Utility lxc-ls from the LXC userspace tools
lxc-lua - 2.1.1-2.1 - LXC Lua bindings
lxc-monitor - 2.1.1-2.1 - Utility lxc-monitor from the LXC userspace tools
lxc-monitord - 2.1.1-2.1 - Utility lxc-monitord from the LXC userspace tools
lxc-snapshot - 2.1.1-2.1 - Utility lxc-snapshot from the LXC userspace tools
lxc-start - 2.1.1-2.1 - Utility lxc-start from the LXC userspace tools
lxc-stop - 2.1.1-2.1 - Utility lxc-stop from the LXC userspace tools
lxc-templates - 2.1.1-2.0 - LXC virtual machine templates
lxc-top - 2.1.1-2.1 - Utility lxc-top from the LXC userspace tools
lxc-unfreeze - 2.1.1-2.1 - Utility lxc-unfreeze from the LXC userspace tools
lxc-unprivileged - 2.1.1-2.0 - Support for unprivileged containers requires newuidmap and newguidmap.
 This package makes sure they are available & have correct permissions.
lxc-unshare - 2.1.1-2.1 - Utility lxc-unshare from the LXC userspace tools
lxc-user-nic - 2.1.1-2.1 - Utility lxc-user-nic from the LXC userspace tools
lxc-usernsexec - 2.1.1-2.1 - Utility lxc-usernsexec from the LXC userspace tools
lxc-wait - 2.1.1-2.1 - Utility lxc-wait from the LXC userspace tools

The foreground mode is just putting the output that i could read in the log file. Sadky it is the same output as i posted earlier.

You could check the source code from the Open-WRT repo and see whether it perhaps been compiled in one of the available packages, or contact the package maintainer.

Oke, i have managed to fix it to run. I created an exactly the same container and looked at the configuration of the new container what i was missing. I changed and tested it with these new configuration settings and i got it working. It mostly was the script option to be blamed for not starting. I would just advice to create a new LXC in TOS4.0 with the same parameters like the one you made in TOS3.x and try to copy what you are missing in your old config. For example, in my old config there was no include, or rootfs with btrfs or the arch being linux32 while i had armv7i. So that is my advice. Before you know it, you will get it working again. BTW, if you are using Ubuntu, it now uses netplan, intead of going to /network/interfaces to change network settings. This is now what i am battling against, to have a static ipaddress with a working dns :). However It runs, so that is the first step :D.

Container config
root@K-Router:/mnt/LXC/K-Router-LXC# cat config
# Template used to create this container: /usr/share/lxc/templates/lxc-download
# Parameters passed to the template: --dist ubuntu --release bionic --arch armhf --server images.linuxcontainers.org --no-validate
# For additional config options, please look at lxc.container.conf(5)

# Uncomment the following line to support nesting containers:
#lxc.include = /usr/share/lxc/config/nesting.conf
# (Be aware this has security implications)

# Distribution configuration
lxc.include = /usr/share/lxc/config/ubuntu.common.conf
lxc.arch = linux32

# Container specific configuration
lxc.rootfs.path = btrfs:/mnt/LXC/K-Router-LXC/rootfs
lxc.uts.name = K-Router-LXC

# Network configuration
lxc.net.0.type = veth
lxc.net.0.link = br-lan
lxc.net.0.flags = up
lxc.net.0.name = eth0
lxc.net.0.ipv4.address = <IPADDRESS>/24
lxc.net.0.ipv4.gateway = <GATEWAY>
#lxc.net.0.script.up = /usr/share/lxc/hooks/tx-off
lxc.net.0.hwaddr = a2:61:29:5c:0f:48

The file /etc/lxc/default.conf i have also populated with some extra information.

default.conf
lxc.net.0.type = veth
lxc.net.0.link = br-lan
lxc.net.0.flags = up
lxc.net.0.hwaddr = a2:61:29:5c:0f:48

# autostart + 3 sec delay
lxc.start.auto = 1
lxc.start.delay = 3

The dns issue just solved with the /etc/resolv.conf in the lxc container, using google dns. I still need to fix booting the lxc with the Omnia andi am having an issue with systemd not booting with the system.

System has not been booted with systemd as init system (PID 1). Can't operate.

For that a alternatief is to make some script using

service <service> start

Example with samba deamon (smbd) to start:
service smbd start

I suspect that this error

System has not been booted with systemd as init system (PID 1). Can't operate.

Has something to do with the host OS (being TOS4.0), as even the newly created container i get this error.

i dont know, if this is the correct thread, i try to copy the lxc-directory (Ubuntu-Container) from the OS3.X to the same location on the OS4.x Beta 1

The LXC is shown in the WebFrontend /cgi-bin/luci/admin/services/lxc.

When i start i will get:
lxc_start - start.c:start:1535 - Permission denied - Failed to exec “/sbin/init”.

I played around with the permissions an set them to 777, which no Change.
ls -al /srv/lxc/test/rootfs/sbin/init
-rwxrwxrwx 1 root root 48 May 12 13:48 /srv/lxc/test/rootfs/sbin/init

(I see https://gitlab.labs.nic.cz/turris/turris-os-packages/issues/354
But the errror-messages are not the same)

Do you have an idea?

Can you tell me where your LXC folder is located?
Secondly, can you put what is in your lxc.conf.

The container are located in /srv/lxc/, e.g. an empty UbuntuBionic-Container is stored in:
/srv/lxc/UbuntuBionic/rootfs/….
My old container
/srv/lxc/test/rootfs/….

The Bionic is created with the WebFrontend and can be started + attached to the running container.

Bionic’s config:

lxc.arch = armv7l

lxc.tty.max = 4
lxc.pty.max = 1024
lxc.rootfs.path = btrfs:/srv/lxc/UbuntuBionic/rootfs
lxc.uts.name = UbuntuBionic

lxc.net.0.type = veth
lxc.net.0.link = br-lan
lxc.net.0.flags = up
lxc.net.0.name = eth0
lxc.net.0.hwaddr = b2:86:a9:88:53:c5

Old container:

lxc.arch = armv7l

lxc.tty.max = 4
lxc.pty.max = 1024
lxc.rootfs.path = btrfs:/srv/lxc/test/rootfs
lxc.uts.name = test

lxc.net.0.type = veth
lxc.net.0.link = br-lan
lxc.net.0.flags = up
lxc.net.0.name = eth0
lxc.net.0.hwaddr = d2:72:7b:4b:b4:b0

Take a look at my config. Is the letter after the 7 in the architecture an capital “i” or a lowercase L? Because it should be a lowercase L.

BTW, you ARE storing the lxc on a separate drive right? (mSata, USB…and NOT the Flash).

Config used in TOS4-B2

#Container specific configuration
root@K-Router:/mnt/LXC/K-Router-LXC# cat config
#Template used to create this container: /usr/share/lxc/templates/lxc-download
#Parameters passed to the template: --dist ubuntu --release bionic --arch armhf --server images.linuxcontainers.org --no-validate
#For additional config options, please look at lxc.container.conf(5)

#Uncomment the following line to support nesting containers:
#lxc.include = /usr/share/lxc/config/nesting.conf
#(Be aware this has security implications)

#Distribution configuration
lxc.arch = armv7l

#Container specific configuration
lxc.include = /usr/share/lxc/config/common.conf
lxc.hook.start-host = /usr/share/lxc/hooks/systemd-workaround
lxc.rootfs.path = btrfs:/mnt/LXC/K-Router-LXC/rootfs
lxc.uts.name = K-Router-LXC

lxc.tty.max = 3

#Network configuration
lxc.net.0.type = veth
lxc.net.0.link = br-lan
lxc.net.0.flags = up
lxc.net.0.name = eth0
lxc.net.0.mtu = 1500
lxc.net.0.ipv4.address = 192.168.1.210/24
lxc.net.0.ipv4.gateway = 192.168.1.1
#lxc.net.0.script.up = /usr/share/lxc/hooks/tx-off
lxc.net.0.hwaddr = a2:61:29:5c:0f:48

lxc.start.auto = 1
lxc.start.delay = 3

Have LXC from TOS 3.x, now I’m on TOS 4.0.5. I edited config file and correct the value names.

But:

root@turris:~# lxc-start test -F -l debug
Failed to lookup module alias ‘autofs4’: Function not implemented
Failed to lookup module alias ‘unix’: Function not implemented
Failed to mount cgroup at /sys/fs/cgroup/systemd: Operation not permitted
[!!!] Failed to mount API filesystems.
Exiting PID 1…

Can you post your config files please.

Thank you for reply. Corrected also others values and seems to be ok.

Is better to reinstall/change LXC for new one or difference is only in config files?

Well I’m not sure what OS you are using in your LXC. Let me take my example. I use currently Ubuntu 18.04. With each new LTS version (every two years) I start a clean LXC OS with migration of some applications. That way I am sure things are up to date. The more you keep an old LXC the harder it becomes to migrate in the future.

So in your case it just depends on your preferences.

I have more LXC, but for example 1/2 - year old.

I think, that in new version are other changes, than config files.

As long as the config files are compatible with your lxc version it should do its work just fine.