Upgrade from 3.x to 5.2.3 and LXC problems

TL;DR

To migrate LXC Container config file from 2.x to 3.x do:

  1. Find on this page Config upgrade script and save the script content to a file lxc-config-upgrade on turris.
    Alternativly download a lxc src release from https://github.com/lxc/lxc/releases and download a zip file, open it and get the script: lxc-lxc-4.0.9.zip/lxc-lxc-4.0.9/src/lxc/cmd/lxc-update-config.in
  2. Run: chmod u+x lxc-config-upgrade
  3. Run: ./lxc-config-upgrade -c <your-lxc-config>

Intro

I run an upgrade of my Turris Omnia from version 3.x to latest 5.x and I encounter following problems.

I would like to share my (no yet finished) way of fixing these errors, because I didn’t find any upgrade manual.

I will appreciate If you could provide any suggestion :wink:

Install utilities

You have to enable LXC utilities in reForisPackage ManagementPackages.
A small thing, but you have to know about.

Old config file format

The LXC version was upgraded from 2.x to 3.0.3-1. Small note, latest LXC version is 4.0.6.

And you cannot do anything with your containers because of config file:

root@turris:/srv/lxc# lxc-ls
Failed to load config for syncthing

Use debug logging to get more information:

root@turris:/srv/lxc# lxc-ls -l debug
lxc-ls: confile.c: parse_line: 2262 Unknown configuration key "lxc.tty"
lxc-ls: parse.c: lxc_file_for_each_line_mmap: 142 Failed to parse config file "/srv/lxc/syncthing/config" at line "lxc.tty = 4"
Failed to load config for syncthing

This is caused by config file format, which remains still in version 2.x and version 3.x can’t work with it.

In this phase I could not create any new container in LuCi because of an error. It forced me to relogin immediately after clicking on Create button.

Error: Session expired - A new login is required since the authentication session expired.

TurrisOS - LuCi - LXC - Error while creating a container

Config migration

I didn’t find an upgrade guide.

I found a suggestion to use lxc-update-config utility, which is a shell script that takes care of that.

lxc-update-config is not part of Turris 5.x installation. So I install LXC on Ubuntu and I found this script.

Package: https://packages.ubuntu.com/focal/lxc-utils
Or you can find it in https://github.com/lxc/lxc/releases in lxc-lxc-4.0.9.zip\lxc-lxc-4.0.9\src\lxc\cmd\lxc-update-config.in

I also checked the 3.0.4 script version and there are some differences. I used the 4.0.

Note: There is an utility for check config file lxc-lxc-4.0.9.zip\lxc-lxc-4.0.9\src\lxc\cmd\lxc-checkconfig.in. I did not used that.

Example of 2.x config

# Template used to create this container: /usr/share/lxc/templates/lxc-download
# Parameters passed to the template: --server api.turris.cz/lxc --no-validate --dist Debian --release Jessie --arch armv7l
# For additional config options, please look at lxc.container.conf(5)

# Distribution configuration
lxc.arch = armv7l

# Container specific configuration
lxc.tty = 4
lxc.pts = 1024
lxc.rootfs = /srv/lxc/syncthing/rootfs
lxc.utsname = syncthing

# Network configuration
lxc.network.type = veth
lxc.network.link = br-lan
lxc.network.flags = up
lxc.network.name = eth0

# Additional config
lxc.network.ipv4 = 192.168.1.10/24
lxc.network.ipv4.gateway = 192.168.1.1
xc.network.hwaddr = 02:01:de:00:00:01

Example of 3.x migrated config

# Template used to create this container: /usr/share/lxc/templates/lxc-download
# Parameters passed to the template: --server api.turris.cz/lxc --no-validate --dist Debian --release Jessie --arch armv7l
# For additional config options, please look at lxc.container.conf(5)

# Distribution configuration
lxc.arch = armv7l

# Container specific configuration
lxc.tty.max = 4
#lxc.pty.max = 1024
lxc.rootfs.path = dir:/srv/lxc/syncthing/rootfs
lxc.uts.name = syncthing

# Network configuration
lxc.net.0.type = veth
lxc.net.0.link = br-lan
lxc.net.0.flags = up
lxc.net.0.name = eth0

# Additional config
lxc.net.0.ipv4.address = 192.168.1.10/24
lxc.net.0.ipv4.gateway = 192.168.1.1
lxc.net.0.hwaddr = 02:01:de:00:00:01

Config upgrade script

I decided to include my version of upgrade script, which fixes problems with PATH in Windows10 WSL.

Source: https://github.com/lxc/lxc/releases, lxc-lxc-4.0.9.zip\lxc-lxc-4.0.9\src\lxc\cmd\lxc-update-config.in

#!/bin/sh
# SPDX-License-Identifier: LGPL-2.1+

# Make sure the usual locations are in PATH
export PATH="$PATH:/usr/sbin:/usr/bin:/sbin:/bin"

set -e

usage()
{
cat <<EOF
$1 -h|--help [-c|--config]
config: the container configuration to update
EOF
    return 0
}

# Check whether any arguments are provided.
if [ $# -eq 0 ]; then
    usage "${0}"
    exit 0
fi

OPTIONS=$(getopt -o c:h --long config:,help -- "${@}")
eval set -- "${OPTIONS}"

while true; do
	case "${1}" in
		-h|--help)
			usage "${0}"
			exit 0
			;;
		-c|--config)
			CONFIGPATH="${2}"
			shift 2
			;;
		--)
			shift 1
			break
			;;
		*)
			break
			;;
	esac
done

cp "${CONFIGPATH}" "${CONFIGPATH}.backup"

# Deal with lxc.rootfs.backend lines
DRIVER=""
while read -r LINE; do
	DRIVER=$(echo $LINE | sed -n 's/\([[:blank:]]*\|#*\)\(lxc\.rootfs\.backend\)\([[:blank:]]*\|\)\(=[[:blank:]]*\|\)\([[:alnum:]]*\)\([[:space:]]*\)/\5/p')
done < "${CONFIGPATH}"

if [ -z "${DRIVER}" ]; then
	DRIVER="dir"
fi
sed -i 's/\([[:blank:]*]\|#*\)\(lxc\.rootfs\)\([[:blank:]*]\|\)\(=[[:blank:]]*\)\(.*\)/\1lxc\.rootfs\.path\3\4'"${DRIVER}"':\5/g' "${CONFIGPATH}"

sed -i \
-e 's/\([[:blank:]*]\|#*\)\(lxc\.id_map\)\([[:blank:]*]\|=\)/\1lxc\.idmap\3/g' \
-e 's/\([[:blank:]*]\|#*\)\(lxc\.pts\)\([[:blank:]*]\|=\)/\1lxc\.pty\.max\3/g' \
-e 's/\([[:blank:]*]\|#*\)\(lxc\.tty\)\([[:blank:]*]\|=\)/\1lxc\.tty\.max\3/g' \
-e 's/\([[:blank:]*]\|#*\)\(lxc\.devttydir\)\([[:blank:]*]\|=\)/\1lxc\.tty\.dir\3/g' \
-e 's/\([[:blank:]*]\|#*\)\(lxc\.aa_profile\)\([[:blank:]*]\|=\)/\1lxc\.apparmor\.profile\3/g' \
-e 's/\([[:blank:]*]\|#*\)\(lxc\.aa_allow_incomplete\)\([[:blank:]*]\|=\)/\1lxc\.apparmor\.allow_incomplete\3/g' \
-e 's/\([[:blank:]*]\|#*\)\(lxc\.se_context\)\([[:blank:]*]\|=\)/\1lxc\.selinux\.context\3/g' \
-e 's/\([[:blank:]*]\|#*\)\(lxc\.mount\)\([[:blank:]*]\|=\)/\1lxc\.mount\.fstab\3/g' \
-e 's/\([[:blank:]*]\|#*\)\(lxc\.utsname\)\([[:blank:]*]\|=\)/\1lxc\.uts\.name\3/g' \
-e 's/\([[:blank:]*]\|#*\)\(lxc\.seccomp\)\([[:blank:]*]\|=\)/\1lxc\.seccomp\.profile\3/g' \
-e 's/\([[:blank:]*]\|#*\)\(lxc\.console\)\([[:blank:]*]\|=\)/\1lxc\.console\.path\3/g' \
-e 's/\([[:blank:]*]\|#*\)\(lxc\.haltsignal\)\([[:blank:]*]\|=\)/\1lxc\.signal\.halt\3/g' \
-e 's/\([[:blank:]*]\|#*\)\(lxc\.rebootsignal\)\([[:blank:]*]\|=\)/\1lxc\.signal\.reboot\3/g' \
-e 's/\([[:blank:]*]\|#*\)\(lxc\.stopsignal\)\([[:blank:]*]\|=\)/\1lxc\.signal\.stop\3/g' \
-e 's/\([[:blank:]*]\|#*\)\(lxc\.syslog\)\([[:blank:]*]\|=\)/\1lxc\.log\.syslog\3/g' \
-e 's/\([[:blank:]*]\|#*\)\(lxc\.loglevel\)\([[:blank:]*]\|=\)/\1lxc\.log\.level\3/g' \
-e 's/\([[:blank:]*]\|#*\)\(lxc\.logfile\)\([[:blank:]*]\|=\)/\1lxc\.log\.file\3/g' \
-e 's/\([[:blank:]*]\|#*\)\(lxc\.init_cmd\)\([[:blank:]*]\|=\)/\1lxc\.init\.cmd\3/g' \
-e 's/\([[:blank:]*]\|#*\)\(lxc\.init_uid\)\([[:blank:]*]\|=\)/\1lxc\.init\.uid\3/g' \
-e 's/\([[:blank:]*]\|#*\)\(lxc\.init_gid\)\([[:blank:]*]\|=\)/\1lxc\.init\.gid\3/g' \
-e 's/\([[:blank:]*]\|#*\)\(lxc\.limit\)\([[:blank:]*]\|=\)/\1lxc\.prlimit\3/g' \
-e 's/\([[:blank:]*]\|#*\)\(lxc\.network\)\(\.[[:digit:]*]\)\(\.ipv4\)/\1lxc\.net\3\4/g' \
-e 's/\([[:blank:]*]\|#*\)\(lxc\.network\)\(\.[[:digit:]*]\)/\1lxc\.net\3/g' \
-e 's/\([[:blank:]*]\|#*\)\(lxc\.network\)\([[:blank:]*]\|=\)/\1lxc\.net\3/g' \
-e '/\([[:blank:]*]\|#*\)\(lxc\.rootfs\.backend\)\([[:blank:]*]\|=\)/d' \
-e '/\([[:blank:]*]\|#*\)\(lxc\.pivotdir\)\([[:blank:]*]\|=\)/d' \
-e '/\([[:blank:]*]\|#*\)\(lxc\.kmsg\)\([[:blank:]*]\|=\)/d' \
	"${CONFIGPATH}"

# Finally, deal with network definitions of the following form:
#
# lxc.network.type = veth
# lxc.network.flags = up
# lxc.network.link = lxdbr0
# lxc.network.name= eth0
#
# lxc.network.type = veth
# lxc.network.flags = up
# lxc.network.link = lxdbr0
# lxc.network.name = eth1

set +e

TMPFILE=$(mktemp -p "${PWD}" XXXXXXXXXX)
cp "${CONFIGPATH}" "${TMPFILE}"

LINE_NUM=0
IDX=-1
while read -r LINE; do
	LINE_NUM=$((LINE_NUM+1))
	# A "lxc.network.type" key defines a new network. So everytime we see
	# one we bump IDX and replace any "lxc.network.<subkey>" keys we
	# encounter with "lxc.network.<IDX>.<subkey>".
	echo "${LINE}" | grep -q "lxc.network.type" && IDX=$((IDX+1))
	sed -i \
-e "${LINE_NUM} s/\([[:blank:]*]\|#*\)\(lxc\.network\)\(\.ipv[[:digit:]]\)\([[:blank:]]*\)=\(.*\)/\1lxc\.net\.${IDX}\3\.address\4=\5/g" \
-e "${LINE_NUM} s/\([[:blank:]*]\|#*\)\(lxc\.network\)\.\([^[:digit:]*]\)/\1lxc\.net\.${IDX}\.\3/g" \
	"${CONFIGPATH}"
done < "${TMPFILE}"

rm "${TMPFILE}" 

Container start problems

When I started the container I received following errors:

root@turris:/srv/lxc# lxc-start syncthing
lxc-start: syncthing: lxccontainer.c: wait_on_daemonized_start: 842 Received container state "ABORTING" instead of "RUNNING"
lxc-start: syncthing: tools/lxc_start.c: main: 330 The container failed to start
lxc-start: syncthing: tools/lxc_start.c: main: 333 To get more details, run the container in foreground mode
lxc-start: syncthing: tools/lxc_start.c: main: 336 Additional information can be obtained by setting the --logfile and --logpriority options

Started in foreground

root@turris:/srv/lxc# lxc-start -F syncthing
lxc-start: syncthing: conf.c: lxc_allocate_ttys: 974 No such file or directory - Failed to create tty 0
lxc-start: syncthing: conf.c: lxc_create_ttys: 1075 Failed to allocate ttys
  lxc-start: syncthing: start.c: do_start: 1263 Failed to setup container "syncthing"
 lxc-start: syncthing: sync.c: __sync_wait: 62 An error occurred in another process (expected sequence number 5)
lxc-start: syncthing: start.c: __lxc_start: 1939 Failed to spawn container "syncthing"
  lxc-start: syncthing: tools/lxc_start.c: main: 330 The container failed to start
lxc-start: syncthing: tools/lxc_start.c: main: 336 Additional information can be obtained by setting the --logfile and --logpriority options

I found the issue. While testing I commented out the line #lxc.pty.max = 1024. Uncommenting it fixed the issue!

Final config file

# Template used to create this container: /usr/share/lxc/templates/lxc-download
# Parameters passed to the template: --server api.turris.cz/lxc --no-validate --dist Debian --release Jessie --arch armv7l
# For additional config options, please look at lxc.container.conf(5)

# Distribution configuration
lxc.arch = armv7l

# Container specific configuration
lxc.tty.max = 4
lxc.pty.max = 1024
lxc.rootfs.path = dir:/srv/lxc/syncthing/rootfs
lxc.uts.name = syncthing

# Network configuration
lxc.net.0.type = veth
lxc.net.0.link = br-lan
lxc.net.0.flags = up
lxc.net.0.name = eth0

# Additional config
lxc.net.0.ipv4.address = 192.168.1.10/24
lxc.net.0.ipv4.gateway = 192.168.1.1
lxc.net.0.hwaddr = 02:01:de:00:00:01

Final suggestion for Turris OS

The shell script should be included in Turris OS. This script should run as part of 5.x migration.

1 Like

Hey, @cynerd can you take a look at this? :arrow_heading_up:
Thanks. :clap:

1 Like

I had no idea that there is some “official” migration script but we have for some time as part of migration this script: updater/tos3to4/files/lxc · master · Turris / Turris OS / Turris OS packages · GitLab.

That is applied only if the update is performed from 3.x using 3.x migration - Turris Documentation. It is not clear from your post the way you performed the update so you might have skipped this step.

Hello,
I performed migration from Turris 3.x to 5.2.7 on Turris Omnia and the LXC configs wasn’t migrated:

# lxc-ls -l debug
lxc-ls: confile.c: parse_line: 2262 Unknown configuration key "lxc.tty"
lxc-ls: parse.c: lxc_file_for_each_line_mmap: 142 Failed to parse config file "/mnt/flash/lxc/server/config" at line "lxc.tty = 4"
Failed to load config for server

Now I’m looking at the official migration script mentioned in previous post and the problem seems to be, that I have custom lxc.lxcpath defined in /etc/lxc/lxc.conf.

Is there a way, to trigger the script once more and prospectively update it to look at my path /mnt/flash/lxc instead of the default one /srv/lxc?

UPDATE: I find the script on path /usr/lib/tos3to4/lxc, so I updated the path on line 45 and launch the script and the config were updated.
Wouldn’t be possible to load the lxc.lxcpath dynamically from the /etc/lxc/lxc.conf configuration?

1 Like