Unable to update LXC container OS - "E: The repository 'http://ports.ubuntu.com/ubuntu-ports focal-updates InRelease' is not signed."

[Bebe]

Hello,

I’m trying to perform update of lxc-container running with “Ubuntu 20.04.3 LTS”. But I’m getting the following error:

root@pihole:/usr/bin# apt-get update
Get:1 http://ports.ubuntu.com/ubuntu-ports focal-updates InRelease [114 kB]
Get:2 http://ports.ubuntu.com/ubuntu-ports focal-security InRelease [114 kB]
Get:3 http://ports.ubuntu.com/ubuntu-ports focal InRelease [265 kB]
Err:1 http://ports.ubuntu.com/ubuntu-ports focal-updates InRelease
Unknown error executing apt-key
Err:2 http://ports.ubuntu.com/ubuntu-ports focal-security InRelease
Unknown error executing apt-key
Err:3 http://ports.ubuntu.com/ubuntu-ports focal InRelease
Unknown error executing apt-key
Reading package lists… Done
W: GPG error: http://ports.ubuntu.com/ubuntu-ports focal-updates InRelease: Unknown error executing apt-key
E: The repository ‘http://ports.ubuntu.com/ubuntu-ports focal-updates InRelease’ is not signed.
…

After quick search I’ve found the problem might be caused by this issue with lib Seccomp:

Seccomp blocks install of "libc6" in Ubuntu 20.04 "focal" image on armhf (arm32v7)

I get it from this forum:

apt update throws signature error in Ubuntu 20.04 container on arm

As I don’t want to install manually additional library to Turris Omnia. Maybe better to run the container as privileged temporary just for the update process, so it will be running under root. But how can I achieve that? I do not see such option for lxc-start nor in lxc.container.conf.

Thank you.

[petrprochy]

IMHO I think LXC container is not same as Docker container. Container is always run as root user.
I tried fresh installation of Ubuntu Focal from Turris repo and “apt update” works correctly. Actual is installed libseccomp2 v2.5.1.

What version is actual installed? Is any error in /var/log/apt/ directory?

Of course you can install libseccomp2 package locally via dpkg tool.

[Bebe]

I’ve already installed the latest one, but I’m still getting the same error.

root@pihole:~# dpkg -l |grep libseccomp
ii libseccomp2:armhf 2.5.3-2 armhf high level interface to Linux seccomp filter

There is no error visible within logs in /var/log/apt…
But since I’m running just pihole on that container, I decided to deploy new container in between with debian 11. Update is working fine there.
So might be this Ubuntu installation had some troubles. I’ll keep it installed and try some other thing to fix it.

[petrprochy]

Hmm, what keys are installed for apt? Execute apt-key list. I have installed this keys:
root@LXCNAME:~# apt-key list
/etc/apt/trusted.gpg.d/ubuntu-keyring-2012-archive.gpg
------------------------------------------------------
pub rsa4096 2012-05-11 [SC]
790B C727 7767 219C 42C8 6F93 3B4F E6AC C0B2 1F32
uid [ unknown] Ubuntu Archive Automatic Signing Key (2012) ftpmaster@ubuntu.com

/etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cdimage.gpg
------------------------------------------------------
pub   rsa4096 2012-05-11 [SC]
      8439 38DF 228D 22F7 B374  2BC0 D94A A3F0 EFE2 1092
uid           [ unknown] Ubuntu CD Image Automatic Signing Key (2012) <cdimage@ubuntu.com>

/etc/apt/trusted.gpg.d/ubuntu-keyring-2018-archive.gpg
------------------------------------------------------
pub   rsa4096 2018-09-17 [SC]
      F6EC B376 2474 EDA9 D21B  7022 8719 20D1 991B C93C
uid           [ unknown] Ubuntu Archive Automatic Signing Key (2018) <ftpmaster@ubuntu.com>

[Bebe]

Looks like there is some trouble with installed packages:

root@pihole:~# apt-key list
E: gnupg, gnupg2 and gnupg1 do not seem to be installed, but one of them is required for this operation

I can see the following:

root@pihole:~# apt-cache search gnupg
gpgv - GNU privacy guard - signature verification tool
libgpg-error0 - GnuPG development runtime library
libhogweed5 - low level cryptographic library (public-key cryptos)
libnettle7 - low level cryptographic library (symmetric and one-way cryptos)
ubuntu-keyring - GnuPG keys of the Ubuntu archive

[petrprochy]

OK, so can you execute apt-get in verbose mode?
apt-get -oDebug::pkgAcquire::Worker=1 update 2>&1 | sed ‘s/%20/ /g; s/%0a/ /g’

In log file are interesting this lines:
→ gpgv:600 URI Acquire URI: gpgv:/var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_focal_InRelease Filename: /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_focal_InRelease Target-Release: focal Target-Type: index Target-Base-URI: Index of /ubuntu-ports/dists/focal Target-Repo-URI: Index of /ubuntu-ports Target-Site: http://ports.ubuntu.com/ubuntu-ports Index-File: true Maximum-Size: 10000000 Last-Modified: Thu, 23 Apr 2020 17:34:17 GMT Fail-Ignore: true
Hit:1 http://ports.ubuntu.com/ubuntu-ports focal InRelease
← gpgv:201 URI Done GPGVOutput: GOODSIG 3B4FE6ACC0B21F32 GOODSIG 871920D1991BC93C Signed-By: 790BC7277767219C42C86F933B4FE6ACC0B21F32! F6ECB3762474EDA9D21B7022871920D1991BC93C! Filename: /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_focal_InRelease URI: gpgv:/var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_focal_InRelease

[Bebe]

sorry for later response…
I ran it in debug mode as you suggested, here are the lines you wanted to see.
For all gpgv:600 URI, the response is same:

→ gpgv:600 URI Acquire URI: gpgv:/var/lib/apt/lists/partial/ports.ubuntu.com_ubuntu-ports_dists_focal-security_InRelease Filename: /var/lib/apt/lists/partial/ports.ubuntu.com_ubuntu-ports_dists_focal-security_InRelease Target-Release: focal-security Target-Type: index Target-Base-URI: Index of /ubuntu-ports/dists/focal-security Target-Repo-URI: Index of /ubuntu-ports Target-Site: http://ports.ubuntu.com/ubuntu-ports Index-File: true Maximum-Size: 10000000 Fail-Ignore: true
Err:1 http://ports.ubuntu.com/ubuntu-ports focal-updates InRelease
Unknown error executing apt-key
← gpgv:400 URI Failure Message: Unknown error executing apt-key URI: gpgv:/var/lib/apt/lists/partial/ports.ubuntu.com_ubuntu-ports_dists_focal-security_InRelease

No other error is thrown.

[petrprochy]

What version is install of apt? Is installed any gpg package? Check via dpkg -l | grep keyword

I have in LXC installed this packages:

root@LXCNAME:/var/log/apt# dpkg -l | grep -E -e "apt|gpg"
ii  apt                            2.0.6                        armhf        commandline package manager
ii  apt-utils                      2.0.6                        armhf        package management related utility programs
ii  aptitude                       0.8.12-1ubuntu4              armhf        terminal-based package manager
ii  aptitude-common                0.8.12-1ubuntu4              all          architecture independent files for the aptitude package manager
ii  gpgv                           2.2.19-3ubuntu2.1            armhf        GNU privacy guard - signature verification tool
ii  libapt-pkg6.0:armhf            2.0.6                        armhf        package management runtime library
ii  libgpg-error0:armhf            1.37-1                       armhf        GnuPG development runtime library
ii  python-apt-common              2.0.0ubuntu0.20.04.7         all          Python interface to libapt-pkg (locales)
ii  python3-apt                    2.0.0ubuntu0.20.04.7         armhf        Python 3 interface to libapt-pkg

[Bebe]

Looks like the version is the same:

root@pihole:~# dpkg -l |grep -E "apt|gpg"
ii  apt                         2.0.6                             armhf        commandline package manager
ii  apt-transport-https         2.0.6                             all          transitional package for https support
ii  apt-utils                   2.0.6                             armhf        package management related utility programs
ii  gpgv                        2.2.19-3ubuntu2.1                 armhf        GNU privacy guard - signature verification tool
ii  libapt-pkg6.0:armhf         2.0.6                             armhf        package management runtime library
ii  libgpg-error0:armhf         1.37-1                            armhf        GnuPG development runtime library

[petrprochy]

Ok, so second chance. Try call apt-get with this options:

apt-get -oDebug::Acquire::gpgv=1 update

I have this result of of one part:

Hit:1 http://ports.ubuntu.com/ubuntu-ports focal InRelease
inside VerifyGetSigners
Get:2 http://ports.ubuntu.com/ubuntu-ports focal-updates InRelease [114 kB]
Preparing to exec:  /usr/bin/apt-key --quiet --readonly verify --status-fd 3 /tmp/apt.sig.AWaUo7 /tmp/apt.data.g2vPl8
Get:3 http://ports.ubuntu.com/ubuntu-ports focal-security InRelease [114 kB]
Read: [GNUPG:] NEWSIG

Read: [GNUPG:] KEY_CONSIDERED 790BC7277767219C42C86F933B4FE6ACC0B21F32 0

Read: [GNUPG:] SIG_ID ZFgbfrBIfmPmMkl7QrbQKd6p5Es 2020-04-23 1587663257

Read: [GNUPG:] KEY_CONSIDERED 790BC7277767219C42C86F933B4FE6ACC0B21F32 0

Read: [GNUPG:] GOODSIG 3B4FE6ACC0B21F32 Ubuntu Archive Automatic Signing Key (2012) <ftpmaster@ubuntu.com>

Got GOODSIG 3B4FE6ACC0B21F32 !
Read: [GNUPG:] VALIDSIG 790BC7277767219C42C86F933B4FE6ACC0B21F32 2020-04-23 1587663257 0 4 0 1 10 01 790BC7277767219C42C86F933B4FE6ACC0B21F32

Got trusted VALIDSIG, key ID: 790BC7277767219C42C86F933B4FE6ACC0B21F32
Read: [GNUPG:] VERIFICATION_COMPLIANCE_MODE 23

Read: [GNUPG:] NEWSIG

Read: [GNUPG:] KEY_CONSIDERED F6ECB3762474EDA9D21B7022871920D1991BC93C 0

Read: [GNUPG:] SIG_ID Xr4bMtWp5CXUImKlL/AAd/+isKA 2020-04-23 1587663257

Read: [GNUPG:] KEY_CONSIDERED F6ECB3762474EDA9D21B7022871920D1991BC93C 0

Read: [GNUPG:] GOODSIG 871920D1991BC93C Ubuntu Archive Automatic Signing Key (2018) <ftpmaster@ubuntu.com>

Got GOODSIG 871920D1991BC93C !
Read: [GNUPG:] VALIDSIG F6ECB3762474EDA9D21B7022871920D1991BC93C 2020-04-23 1587663257 0 4 0 1 10 01 F6ECB3762474EDA9D21B7022871920D1991BC93C

Got trusted VALIDSIG, key ID: F6ECB3762474EDA9D21B7022871920D1991BC93C
Read: [GNUPG:] VERIFICATION_COMPLIANCE_MODE 23

gpgv exited with status 0
Summary:
  Good: GOODSIG 3B4FE6ACC0B21F32, GOODSIG 871920D1991BC93C
  Valid: 790BC7277767219C42C86F933B4FE6ACC0B21F32, F6ECB3762474EDA9D21B7022871920D1991BC93C
  Bad:
  Worthless:
  SoonWorthless:
  NoPubKey:
  Signed-By: 790BC7277767219C42C86F933B4FE6ACC0B21F32!, F6ECB3762474EDA9D21B7022871920D1991BC93C!
  NODATA: no
apt-key succeeded

I hope that result of this gets more info of issue.

[Bebe]

Again without deeper details…
at least I cannot find what is the exit code 135 for gpgv

Get:1 http://ports.ubuntu.com/ubuntu-ports focal-updates InRelease [114 kB]
Get:2 http://ports.ubuntu.com/ubuntu-ports focal-security InRelease [114 kB]
0% [Working]inside VerifyGetSigners
Preparing to exec:  /usr/bin/apt-key --quiet --readonly verify --status-fd 3 /tmp/apt.sig.3HumH9 /tmp/apt.data.g5UIXa
Get:3 http://ports.ubuntu.com/ubuntu-ports focal InRelease [265 kB]
0% [Working]gpgv exited with status 135                                                                                                                                                              
Summary:
  Good: 
  Valid: 
  Bad: 
  Worthless: 
  SoonWorthless: 
  NoPubKey: 
  Signed-By: 
  NODATA: no
Err:1 http://ports.ubuntu.com/ubuntu-ports focal-updates InRelease
  Unknown error executing apt-key

[petrprochy]

OK, so call apt-key directly, for ex.:

apt-key verify /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_focal-security_InRelease

I get this result:

gpgv: Signature made Fri May  6 18:05:20 2022 UTC
gpgv:                using RSA key 3B4FE6ACC0B21F32
gpgv: Good signature from "Ubuntu Archive Automatic Signing Key (2012) <ftpmaster@ubuntu.com>"
gpgv: Signature made Fri May  6 18:05:20 2022 UTC
gpgv:                using RSA key 871920D1991BC93C
gpgv: Good signature from "Ubuntu Archive Automatic Signing Key (2018) <ftpmaster@ubuntu.com>"

[Bebe]

I get just this, but together with it I can see the new dmesg entry…

root@pihole:~# apt-key verify /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_focal-security_InRelease 
Bus error
root@pihole:~# dmesg |tail -1
[823182.664441] BTRFS warning (device sda1): csum failed root 5 ino 1986 off 90112 csum 0x1dd40ceb expected csum 0x12994098 mirror 1

So this looks like the issue might be caused by failed blocks on USB key where containers are stored. And maybe some binaries/libraries are sitting on these corrupted blocks…
I’ll try to check the usb offline when I’ll be at home.

[Bebe]

So yes, there are some checksum errors on FS level. As this is just container and I’ve already migrated to new one, I think this topic might be closed. I don’t think this could be fixed (at least for sure not on that container wich is already affected). Hard to say what exactly is corrupted there. Probably some part of usb key is worn out and caused issues on btrfs level.
It is quite old usb key. I’m redirecting os logging on usb keys to ram by default, but you never know…

Thanks a lot for your help and patience.

$ sudo btrfs check -p --check-data-csum /dev/sdb1
Opening filesystem to check...
Checking filesystem on /dev/sdb1
UUID: bcbb6ed7-3760-41ae-8ece-ed02481a5d67
[1/7] checking root items                      (0:00:02 elapsed, 198283 items checked)
[2/7] checking extents                         (0:00:06 elapsed, 4768 items checked)
block group 3452960768 has wrong amount of free space, free space cache has 222867456 block group has 250429440
failed to load free space cache for block group 3452960768
[3/7] checking free space cache                (0:00:00 elapsed, 9 items checked)
[4/7] checking fs roots                        (0:00:01 elapsed, 3282 items checked)
mirror 1 bytenr 425459712 csum 0x09fa5474 expected csum 0x538c7c4b032 items checked)
mirror 1 bytenr 425463808 csum 0x0c6a6ae4 expected csum 0xb201f83c
mirror 1 bytenr 425467904 csum 0xe790515b expected csum 0xba00c1fa
mirror 1 bytenr 425472000 csum 0xeb0cd41d expected csum 0x98409912
[5/7] checking csums against data              (0:02:46 elapsed, 56482 items checked)
ERROR: errors found in csum tree
[6/7] checking root refs                       (0:00:00 elapsed, 4 items checked)
[7/7] checking quota groups skipped (not enabled on this FS)
found 3669475328 bytes used, error(s) found
total csum bytes: 3495104
total tree bytes: 77971456
total fs tree bytes: 54018048
total extent tree bytes: 17989632
btree space waste bytes: 12461938
file data blocks allocated: 3788791808
 referenced 2703204352

[petrprochy]

It’s fine to see that problem is solved.

You can call btrfs scrub command:

btrfs scrub start -B /

And in syslog you will see which files are corrupted. I see broken files on openSUSE, so on Ubuntu can be too.