Upgrade from 3.x to 5.2.3 and LXC problems

TL;DR

To migrate LXC Container config file from 2.x to 3.x do:

  1. Find on this page Config upgrade script and save the script content to a file lxc-config-upgrade on turris.
    Alternativly download a lxc src release from https://github.com/lxc/lxc/releases and download a zip file, open it and get the script: lxc-lxc-4.0.9.zip/lxc-lxc-4.0.9/src/lxc/cmd/lxc-update-config.in
  2. Run: chmod u+x lxc-config-upgrade
  3. Run: ./lxc-config-upgrade -c <your-lxc-config>

Intro

I run an upgrade of my Turris Omnia from version 3.x to latest 5.x and I encounter following problems.

I would like to share my (no yet finished) way of fixing these errors, because I didn’t find any upgrade manual.

I will appreciate If you could provide any suggestion :wink:

Install utilities

You have to enable LXC utilities in reForis → Package Management → Packages.
A small thing, but you have to know about.

Old config file format

The LXC version was upgraded from 2.x to 3.0.3-1. Small note, latest LXC version is 4.0.6.

And you cannot do anything with your containers because of config file:

root@turris:/srv/lxc# lxc-ls
Failed to load config for syncthing

Use debug logging to get more information:

root@turris:/srv/lxc# lxc-ls -l debug
lxc-ls: confile.c: parse_line: 2262 Unknown configuration key "lxc.tty"
lxc-ls: parse.c: lxc_file_for_each_line_mmap: 142 Failed to parse config file "/srv/lxc/syncthing/config" at line "lxc.tty = 4"
Failed to load config for syncthing

This is caused by config file format, which remains still in version 2.x and version 3.x can’t work with it.

In this phase I could not create any new container in LuCi because of an error. It forced me to relogin immediately after clicking on Create button.

Error: Session expired - A new login is required since the authentication session expired.

TurrisOS - LuCi - LXC - Error while creating a container

Config migration

I didn’t find an upgrade guide.

I found a suggestion to use lxc-update-config utility, which is a shell script that takes care of that.

lxc-update-config is not part of Turris 5.x installation. So I install LXC on Ubuntu and I found this script.

Package: https://packages.ubuntu.com/focal/lxc-utils
Or you can find it in https://github.com/lxc/lxc/releases in lxc-lxc-4.0.9.zip\lxc-lxc-4.0.9\src\lxc\cmd\lxc-update-config.in

I also checked the 3.0.4 script version and there are some differences. I used the 4.0.

Note: There is an utility for check config file lxc-lxc-4.0.9.zip\lxc-lxc-4.0.9\src\lxc\cmd\lxc-checkconfig.in. I did not used that.

Example of 2.x config

# Template used to create this container: /usr/share/lxc/templates/lxc-download
# Parameters passed to the template: --server api.turris.cz/lxc --no-validate --dist Debian --release Jessie --arch armv7l
# For additional config options, please look at lxc.container.conf(5)

# Distribution configuration
lxc.arch = armv7l

# Container specific configuration
lxc.tty = 4
lxc.pts = 1024
lxc.rootfs = /srv/lxc/syncthing/rootfs
lxc.utsname = syncthing

# Network configuration
lxc.network.type = veth
lxc.network.link = br-lan
lxc.network.flags = up
lxc.network.name = eth0

# Additional config
lxc.network.ipv4 = 192.168.1.10/24
lxc.network.ipv4.gateway = 192.168.1.1
xc.network.hwaddr = 02:01:de:00:00:01

Example of 3.x migrated config

# Template used to create this container: /usr/share/lxc/templates/lxc-download
# Parameters passed to the template: --server api.turris.cz/lxc --no-validate --dist Debian --release Jessie --arch armv7l
# For additional config options, please look at lxc.container.conf(5)

# Distribution configuration
lxc.arch = armv7l

# Container specific configuration
lxc.tty.max = 4
#lxc.pty.max = 1024
lxc.rootfs.path = dir:/srv/lxc/syncthing/rootfs
lxc.uts.name = syncthing

# Network configuration
lxc.net.0.type = veth
lxc.net.0.link = br-lan
lxc.net.0.flags = up
lxc.net.0.name = eth0

# Additional config
lxc.net.0.ipv4.address = 192.168.1.10/24
lxc.net.0.ipv4.gateway = 192.168.1.1
lxc.net.0.hwaddr = 02:01:de:00:00:01

Config upgrade script

I decided to include my version of upgrade script, which fixes problems with PATH in Windows10 WSL.

Source: https://github.com/lxc/lxc/releases, lxc-lxc-4.0.9.zip\lxc-lxc-4.0.9\src\lxc\cmd\lxc-update-config.in

#!/bin/sh
# SPDX-License-Identifier: LGPL-2.1+

# Make sure the usual locations are in PATH
export PATH="$PATH:/usr/sbin:/usr/bin:/sbin:/bin"

set -e

usage()
{
cat <<EOF
$1 -h|--help [-c|--config]
config: the container configuration to update
EOF
    return 0
}

# Check whether any arguments are provided.
if [ $# -eq 0 ]; then
    usage "${0}"
    exit 0
fi

OPTIONS=$(getopt -o c:h --long config:,help -- "${@}")
eval set -- "${OPTIONS}"

while true; do
	case "${1}" in
		-h|--help)
			usage "${0}"
			exit 0
			;;
		-c|--config)
			CONFIGPATH="${2}"
			shift 2
			;;
		--)
			shift 1
			break
			;;
		*)
			break
			;;
	esac
done

cp "${CONFIGPATH}" "${CONFIGPATH}.backup"

# Deal with lxc.rootfs.backend lines
DRIVER=""
while read -r LINE; do
	DRIVER=$(echo $LINE | sed -n 's/\([[:blank:]]*\|#*\)\(lxc\.rootfs\.backend\)\([[:blank:]]*\|\)\(=[[:blank:]]*\|\)\([[:alnum:]]*\)\([[:space:]]*\)/\5/p')
done < "${CONFIGPATH}"

if [ -z "${DRIVER}" ]; then
	DRIVER="dir"
fi
sed -i 's/\([[:blank:]*]\|#*\)\(lxc\.rootfs\)\([[:blank:]*]\|\)\(=[[:blank:]]*\)\(.*\)/\1lxc\.rootfs\.path\3\4'"${DRIVER}"':\5/g' "${CONFIGPATH}"

sed -i \
-e 's/\([[:blank:]*]\|#*\)\(lxc\.id_map\)\([[:blank:]*]\|=\)/\1lxc\.idmap\3/g' \
-e 's/\([[:blank:]*]\|#*\)\(lxc\.pts\)\([[:blank:]*]\|=\)/\1lxc\.pty\.max\3/g' \
-e 's/\([[:blank:]*]\|#*\)\(lxc\.tty\)\([[:blank:]*]\|=\)/\1lxc\.tty\.max\3/g' \
-e 's/\([[:blank:]*]\|#*\)\(lxc\.devttydir\)\([[:blank:]*]\|=\)/\1lxc\.tty\.dir\3/g' \
-e 's/\([[:blank:]*]\|#*\)\(lxc\.aa_profile\)\([[:blank:]*]\|=\)/\1lxc\.apparmor\.profile\3/g' \
-e 's/\([[:blank:]*]\|#*\)\(lxc\.aa_allow_incomplete\)\([[:blank:]*]\|=\)/\1lxc\.apparmor\.allow_incomplete\3/g' \
-e 's/\([[:blank:]*]\|#*\)\(lxc\.se_context\)\([[:blank:]*]\|=\)/\1lxc\.selinux\.context\3/g' \
-e 's/\([[:blank:]*]\|#*\)\(lxc\.mount\)\([[:blank:]*]\|=\)/\1lxc\.mount\.fstab\3/g' \
-e 's/\([[:blank:]*]\|#*\)\(lxc\.utsname\)\([[:blank:]*]\|=\)/\1lxc\.uts\.name\3/g' \
-e 's/\([[:blank:]*]\|#*\)\(lxc\.seccomp\)\([[:blank:]*]\|=\)/\1lxc\.seccomp\.profile\3/g' \
-e 's/\([[:blank:]*]\|#*\)\(lxc\.console\)\([[:blank:]*]\|=\)/\1lxc\.console\.path\3/g' \
-e 's/\([[:blank:]*]\|#*\)\(lxc\.haltsignal\)\([[:blank:]*]\|=\)/\1lxc\.signal\.halt\3/g' \
-e 's/\([[:blank:]*]\|#*\)\(lxc\.rebootsignal\)\([[:blank:]*]\|=\)/\1lxc\.signal\.reboot\3/g' \
-e 's/\([[:blank:]*]\|#*\)\(lxc\.stopsignal\)\([[:blank:]*]\|=\)/\1lxc\.signal\.stop\3/g' \
-e 's/\([[:blank:]*]\|#*\)\(lxc\.syslog\)\([[:blank:]*]\|=\)/\1lxc\.log\.syslog\3/g' \
-e 's/\([[:blank:]*]\|#*\)\(lxc\.loglevel\)\([[:blank:]*]\|=\)/\1lxc\.log\.level\3/g' \
-e 's/\([[:blank:]*]\|#*\)\(lxc\.logfile\)\([[:blank:]*]\|=\)/\1lxc\.log\.file\3/g' \
-e 's/\([[:blank:]*]\|#*\)\(lxc\.init_cmd\)\([[:blank:]*]\|=\)/\1lxc\.init\.cmd\3/g' \
-e 's/\([[:blank:]*]\|#*\)\(lxc\.init_uid\)\([[:blank:]*]\|=\)/\1lxc\.init\.uid\3/g' \
-e 's/\([[:blank:]*]\|#*\)\(lxc\.init_gid\)\([[:blank:]*]\|=\)/\1lxc\.init\.gid\3/g' \
-e 's/\([[:blank:]*]\|#*\)\(lxc\.limit\)\([[:blank:]*]\|=\)/\1lxc\.prlimit\3/g' \
-e 's/\([[:blank:]*]\|#*\)\(lxc\.network\)\(\.[[:digit:]*]\)\(\.ipv4\)/\1lxc\.net\3\4/g' \
-e 's/\([[:blank:]*]\|#*\)\(lxc\.network\)\(\.[[:digit:]*]\)/\1lxc\.net\3/g' \
-e 's/\([[:blank:]*]\|#*\)\(lxc\.network\)\([[:blank:]*]\|=\)/\1lxc\.net\3/g' \
-e '/\([[:blank:]*]\|#*\)\(lxc\.rootfs\.backend\)\([[:blank:]*]\|=\)/d' \
-e '/\([[:blank:]*]\|#*\)\(lxc\.pivotdir\)\([[:blank:]*]\|=\)/d' \
-e '/\([[:blank:]*]\|#*\)\(lxc\.kmsg\)\([[:blank:]*]\|=\)/d' \
	"${CONFIGPATH}"

# Finally, deal with network definitions of the following form:
#
# lxc.network.type = veth
# lxc.network.flags = up
# lxc.network.link = lxdbr0
# lxc.network.name= eth0
#
# lxc.network.type = veth
# lxc.network.flags = up
# lxc.network.link = lxdbr0
# lxc.network.name = eth1

set +e

TMPFILE=$(mktemp -p "${PWD}" XXXXXXXXXX)
cp "${CONFIGPATH}" "${TMPFILE}"

LINE_NUM=0
IDX=-1
while read -r LINE; do
	LINE_NUM=$((LINE_NUM+1))
	# A "lxc.network.type" key defines a new network. So everytime we see
	# one we bump IDX and replace any "lxc.network.<subkey>" keys we
	# encounter with "lxc.network.<IDX>.<subkey>".
	echo "${LINE}" | grep -q "lxc.network.type" && IDX=$((IDX+1))
	sed -i \
-e "${LINE_NUM} s/\([[:blank:]*]\|#*\)\(lxc\.network\)\(\.ipv[[:digit:]]\)\([[:blank:]]*\)=\(.*\)/\1lxc\.net\.${IDX}\3\.address\4=\5/g" \
-e "${LINE_NUM} s/\([[:blank:]*]\|#*\)\(lxc\.network\)\.\([^[:digit:]*]\)/\1lxc\.net\.${IDX}\.\3/g" \
	"${CONFIGPATH}"
done < "${TMPFILE}"

rm "${TMPFILE}" 

Container start problems

When I started the container I received following errors:

root@turris:/srv/lxc# lxc-start syncthing
lxc-start: syncthing: lxccontainer.c: wait_on_daemonized_start: 842 Received container state "ABORTING" instead of "RUNNING"
lxc-start: syncthing: tools/lxc_start.c: main: 330 The container failed to start
lxc-start: syncthing: tools/lxc_start.c: main: 333 To get more details, run the container in foreground mode
lxc-start: syncthing: tools/lxc_start.c: main: 336 Additional information can be obtained by setting the --logfile and --logpriority options

Started in foreground

root@turris:/srv/lxc# lxc-start -F syncthing
lxc-start: syncthing: conf.c: lxc_allocate_ttys: 974 No such file or directory - Failed to create tty 0
lxc-start: syncthing: conf.c: lxc_create_ttys: 1075 Failed to allocate ttys
  lxc-start: syncthing: start.c: do_start: 1263 Failed to setup container "syncthing"
 lxc-start: syncthing: sync.c: __sync_wait: 62 An error occurred in another process (expected sequence number 5)
lxc-start: syncthing: start.c: __lxc_start: 1939 Failed to spawn container "syncthing"
  lxc-start: syncthing: tools/lxc_start.c: main: 330 The container failed to start
lxc-start: syncthing: tools/lxc_start.c: main: 336 Additional information can be obtained by setting the --logfile and --logpriority options

I found the issue. While testing I commented out the line #lxc.pty.max = 1024. Uncommenting it fixed the issue!

Final config file

# Template used to create this container: /usr/share/lxc/templates/lxc-download
# Parameters passed to the template: --server api.turris.cz/lxc --no-validate --dist Debian --release Jessie --arch armv7l
# For additional config options, please look at lxc.container.conf(5)

# Distribution configuration
lxc.arch = armv7l

# Container specific configuration
lxc.tty.max = 4
lxc.pty.max = 1024
lxc.rootfs.path = dir:/srv/lxc/syncthing/rootfs
lxc.uts.name = syncthing

# Network configuration
lxc.net.0.type = veth
lxc.net.0.link = br-lan
lxc.net.0.flags = up
lxc.net.0.name = eth0

# Additional config
lxc.net.0.ipv4.address = 192.168.1.10/24
lxc.net.0.ipv4.gateway = 192.168.1.1
lxc.net.0.hwaddr = 02:01:de:00:00:01

Final suggestion for Turris OS

The shell script should be included in Turris OS. This script should run as part of 5.x migration.

1 Like

Hey, @cynerd can you take a look at this? :arrow_heading_up:
Thanks. :clap:

1 Like

I had no idea that there is some “official” migration script but we have for some time as part of migration this script: updater/tos3to4/files/lxc · master · Turris / Turris OS / Turris OS packages · GitLab.

That is applied only if the update is performed from 3.x using 3.x migration - Turris Documentation. It is not clear from your post the way you performed the update so you might have skipped this step.

Hello,
I performed migration from Turris 3.x to 5.2.7 on Turris Omnia and the LXC configs wasn’t migrated:

# lxc-ls -l debug
lxc-ls: confile.c: parse_line: 2262 Unknown configuration key "lxc.tty"
lxc-ls: parse.c: lxc_file_for_each_line_mmap: 142 Failed to parse config file "/mnt/flash/lxc/server/config" at line "lxc.tty = 4"
Failed to load config for server

Now I’m looking at the official migration script mentioned in previous post and the problem seems to be, that I have custom lxc.lxcpath defined in /etc/lxc/lxc.conf.

Is there a way, to trigger the script once more and prospectively update it to look at my path /mnt/flash/lxc instead of the default one /srv/lxc?

UPDATE: I find the script on path /usr/lib/tos3to4/lxc, so I updated the path on line 45 and launch the script and the config were updated.
Wouldn’t be possible to load the lxc.lxcpath dynamically from the /etc/lxc/lxc.conf configuration?

1 Like

The migration is designed to migrate only settings available from web GUI. In most cases that means only Foris but for LXC that means LuCI. The custom path is not possible to be configured there so there is no support for it. Honestly, I do not think that there are a lot of users with very custom configurations so I am reluctant to implement it. At the same time, it is not hard to do so I am open for patches on the following file updater/tos3to4/files/lxc · master · Turris / Turris OS / Turris OS packages · GitLab.

Hi @cynerd,

I just ran into the same issue as @amra after my Omnia upgraded automatically to TurrisOS 5.3.3. Although I understand your policy is just to support changes done through Foris / LuCI, using the hardcoded path in the migration script doesn’t seem to be a good practice as you tightly couple the paths defined on multiple places. Because of this, I’m strongly advocating for fixing the script you referenced.

I have tried to fork the Turris OS packages repo from your link and provide you patch via merge request, but my attempts to fork always failed with generic error An error occurred while forking the project. Please try again.. For this reason, please find the patch for the lxc migration script you referenced above below in the unidiff format. I tested the patched version of the script on my Turris and it migrated my lxc configs as expected.

Cheers,

T.

--- ".\\lxc"	2021-12-19 15:52:38.000235500 +0100
+++ ".\\lxc.fix"	2021-12-19 15:52:03.046225600 +0100
@@ -41,8 +41,9 @@
 	' "$conf.lxc1" > "$conf"
 }
 
-[ -d /srv/lxc ] || exit 0 # LXC is not used
-for config in /srv/lxc/*/config; do
+LXCPATH=`grep '^\s*lxc.lxcpath\s*=' /etc/lxc/lxc.conf 2>/dev/null | cut -d= -f2 | xargs`
+[ -d "$LXCPATH" ] || exit 0 # LXC is not used
+for config in $LXCPATH/*/config; do
 	[ -f "$config" ] || continue
 	migrate_config "$config"
 done

This is unfortunate, but this applies to all projects, including ours in CZ.NIC GitLab. We need to ask CZ.NIC GitLab’s administrators to allow it for you, which is somehow protection against spammers, robots, and of course, contributors. :frowning:

Because of that, all of our repositories are mirrored on GitHub. It is mentioned in our documentation, but I understand that we need to make some changes in each repository clearer.

Regarding your patch, thanks! We will take a look, but I am not sure if it makes it to the final migration wave, which will be pushed shortly and there are also holidays incoming.