LXC Ubuntu LTS stopped working

omichalek · March 19, 2019, 11:32pm

I have Ubuntu Bionic running as a container and it worked fine until I restarted the router today – ever since then the container has problems setting up network it seems (or at least SSHD)

Can somebody help me understand what is going on or even fix it please?

journalctl -b

-- Logs begin at Wed 2019-03-20 00:03:44 CET, end at Wed 2019-03-20 00:17:01 CET. --
Mar 20 00:03:44 ubuntu systemd-journald[26]: Journal started
Mar 20 00:03:44 ubuntu systemd-journald[26]: Runtime journal (/run/log/journal/9532e8f459a744e299cbf1f41115a752) is 8.0M, max 101.0M, 93.0M free.
Mar 20 00:03:44 ubuntu systemd-sysctl[50]: Couldn't write '1' to 'net/ipv4/tcp_syncookies', ignoring: No such file or directory
Mar 20 00:03:44 ubuntu systemd-sysctl[50]: Couldn't write '1' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory
Mar 20 00:03:44 ubuntu systemd-sysctl[50]: Couldn't write 'fq_codel' to 'net/core/default_qdisc', ignoring: No such file or directory
Mar 20 00:03:44 ubuntu keyboard-setup.sh[16]: Couldn't get a file descriptor referring to the console
Mar 20 00:03:44 ubuntu systemd[1]: Starting Flush Journal to Persistent Storage...
Mar 20 00:03:44 ubuntu keyboard-setup.sh[16]: Couldn't get a file descriptor referring to the console
Mar 20 00:03:44 ubuntu keyboard-setup.sh[16]: setupcon: We are not on the console, the console is left unconfigured.
Mar 20 00:03:44 ubuntu systemd[1]: Started udev Kernel Device Manager.
Mar 20 00:03:44 ubuntu systemd[1]: Started Set the console keyboard layout.
Mar 20 00:03:44 ubuntu systemd-journald[26]: Runtime journal (/run/log/journal/9532e8f459a744e299cbf1f41115a752) is 8.0M, max 101.0M, 93.0M free.
Mar 20 00:03:44 ubuntu systemd[1]: Started Flush Journal to Persistent Storage.
Mar 20 00:03:44 ubuntu systemd[1]: Reached target Local File Systems (Pre).
Mar 20 00:03:44 ubuntu systemd[1]: Reached target Local File Systems.
Mar 20 00:03:44 ubuntu systemd[1]: Starting Set console font and keymap...
Mar 20 00:03:44 ubuntu systemd[1]: Starting Create Volatile Files and Directories...
Mar 20 00:03:44 ubuntu systemd[1]: Starting Network Service...
Mar 20 00:03:44 ubuntu systemd[1]: Started Set console font and keymap.
Mar 20 00:03:44 ubuntu systemd[1]: Started Create Volatile Files and Directories.
Mar 20 00:03:44 ubuntu systemd[1]: Starting Update UTMP about System Boot/Shutdown...
Mar 20 00:03:44 ubuntu systemd[1]: Reached target System Time Synchronized.
Mar 20 00:03:44 ubuntu systemd-networkd[94]: Enumeration completed
Mar 20 00:03:44 ubuntu systemd[1]: Started Network Service.
Mar 20 00:03:44 ubuntu systemd[1]: Starting Network Name Resolution...
Mar 20 00:03:44 ubuntu systemd[1]: Started Uncomplicated firewall.
Mar 20 00:03:45 ubuntu systemd[1]: Started Update UTMP about System Boot/Shutdown.
Mar 20 00:03:45 ubuntu systemd-udevd[137]: could not open moddep file '/lib/modules/4.4.169-7bc33afbb1b35f5830b2b1b42c9cd8a0-2/modules.dep.bin'
Mar 20 00:03:45 ubuntu systemd-udevd[137]: could not open moddep file '/lib/modules/4.4.169-7bc33afbb1b35f5830b2b1b42c9cd8a0-2/modules.dep.bin'
Mar 20 00:03:45 ubuntu systemd-resolved[113]: Positive Trust Anchors:
Mar 20 00:03:45 ubuntu systemd-resolved[113]: . IN DS 19036 8 2 49
Mar 20 00:03:45 ubuntu systemd-resolved[113]: . IN DS 20326 8 2 e0
Mar 20 00:03:45 ubuntu systemd-resolved[113]: Negative trust anchors: 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa 168.192.in-addr.arpa d.f.ip6.arpa corp home internal intranet lan local private test
Mar 20 00:03:45 ubuntu systemd-resolved[113]: Using system hostname 'ubuntu'.
Mar 20 00:03:45 ubuntu systemd[1]: Started Network Name Resolution.
Mar 20 00:03:45 ubuntu systemd[1]: Reached target Network.
Mar 20 00:03:45 ubuntu systemd[1]: Reached target Host and Network Name Lookups.
Mar 20 00:03:45 ubuntu systemd[1]: Started udev Coldplug all Devices.
Mar 20 00:03:45 ubuntu systemd[1]: Reached target System Initialization.
Mar 20 00:03:45 ubuntu systemd[1]: Started Daily apt download activities.
Mar 20 00:03:45 ubuntu systemd[1]: Started Run certbot twice daily.
Mar 20 00:03:45 ubuntu systemd[1]: Started Message of the Day.
Mar 20 00:03:45 ubuntu systemd[1]: Started Clean PHP session files every 30 mins.
Mar 20 00:03:45 ubuntu systemd[1]: Started Discard unused blocks once a week.
Mar 20 00:03:45 ubuntu systemd[1]: Listening on D-Bus System Message Bus Socket.
Mar 20 00:03:45 ubuntu systemd[1]: Reached target Sockets.
Mar 20 00:03:45 ubuntu systemd[1]: Started Daily apt upgrade and clean activities.
Mar 20 00:03:45 ubuntu systemd[1]: Reached target Basic System.
Mar 20 00:03:45 ubuntu systemd[1]: Starting MariaDB 10.1.38 database server...
Mar 20 00:03:45 ubuntu systemd[1]: Starting Permit User Sessions...
Mar 20 00:03:45 ubuntu systemd[1]: Started D-Bus System Message Bus.
Mar 20 00:03:45 ubuntu systemd[1]: Starting The PHP 7.2 FastCGI Process Manager...
Mar 20 00:03:45 ubuntu systemd[1]: Starting Dispatcher daemon for systemd-networkd...
Mar 20 00:03:45 ubuntu systemd[1]: Starting Login Service...
Mar 20 00:03:45 ubuntu systemd[1]: Starting System Logging Service...
Mar 20 00:03:45 ubuntu systemd[1]: Starting A high performance web server and a reverse proxy server...
Mar 20 00:03:45 ubuntu systemd-udevd[149]: could not open moddep file '/lib/modules/4.4.169-7bc33afbb1b35f5830b2b1b42c9cd8a0-2/modules.dep.bin'
Mar 20 00:03:45 ubuntu systemd[1]: Started Regular background program processing daemon.
Mar 20 00:03:45 ubuntu systemd-udevd[144]: inotify_add_watch(9, /dev/mmcblk0, 10) failed: No such file or directory
Mar 20 00:03:45 ubuntu systemd-udevd[139]: inotify_add_watch(9, /dev/mmcblk0boot1, 10) failed: No such file or directory
Mar 20 00:03:45 ubuntu systemd-udevd[139]: inotify_add_watch(9, /dev/mmcblk0p1, 10) failed: No such file or directory
Mar 20 00:03:45 ubuntu systemd-udevd[139]: inotify_add_watch(9, /dev/mmcblk0rpmb, 10) failed: No such file or directory
Mar 20 00:03:45 ubuntu ata_id[157]: unable to open '/dev/sda'
Mar 20 00:03:46 ubuntu systemd[1]: Starting OpenBSD Secure Shell server...
Mar 20 00:03:46 ubuntu systemd[1]: Started Daily Cleanup of Temporary Directories.
Mar 20 00:03:46 ubuntu cron[158]: (CRON) INFO (pidfile fd = 3)
Mar 20 00:03:46 ubuntu rsyslogd[155]: imuxsock: Acquired UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd.  [v8.32.0]
Mar 20 00:03:46 ubuntu systemd-udevd[144]: inotify_add_watch(9, /dev/mmcblk0boot0, 10) failed: No such file or directory
Mar 20 00:03:46 ubuntu rsyslogd[155]: rsyslogd's groupid changed to 106
Mar 20 00:03:46 ubuntu systemd-networkd[94]: eth0: Gained IPv6LL
Mar 20 00:03:46 ubuntu rsyslogd[155]: rsyslogd's userid changed to 102
Mar 20 00:03:46 ubuntu systemd-udevd[149]: inotify_add_watch(9, /dev/sda, 10) failed: No such file or directory
Mar 20 00:03:46 ubuntu rsyslogd[155]:  [origin software="rsyslogd" swVersion="8.32.0" x-pid="155" x-info="http://www.rsyslog.com"] start
Mar 20 00:03:46 ubuntu systemd[1]: Started System Logging Service.
Mar 20 00:03:46 ubuntu systemd[1]: Started Permit User Sessions.
Mar 20 00:03:46 ubuntu systemd-logind[154]: New seat seat0.
Mar 20 00:03:46 ubuntu systemd[1]: Started Login Service.
Mar 20 00:03:46 ubuntu systemd[1]: Started Unattended Upgrades Shutdown.
Mar 20 00:03:46 ubuntu systemd[1]: Started Container Getty on /dev/pts/3.
Mar 20 00:03:46 ubuntu systemd[1]: Started Container Getty on /dev/pts/0.
Mar 20 00:03:46 ubuntu systemd[1]: Started Container Getty on /dev/pts/2.
Mar 20 00:03:46 ubuntu cron[158]: (CRON) INFO (Running @reboot jobs)
Mar 20 00:03:46 ubuntu systemd[1]: Started Container Getty on /dev/pts/1.
Mar 20 00:03:46 ubuntu systemd[1]: Started Console Getty.
Mar 20 00:03:46 ubuntu systemd[1]: Created slice system-getty.slice.
Mar 20 00:03:46 ubuntu systemd-udevd[144]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Mar 20 00:03:46 ubuntu systemd-udevd[149]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Mar 20 00:03:46 ubuntu systemd-udevd[136]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Mar 20 00:03:46 ubuntu systemd-udevd[140]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Mar 20 00:03:46 ubuntu systemd-udevd[143]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Mar 20 00:03:46 ubuntu systemd-udevd[143]: Could not generate persistent MAC address for ip6tnl0: No such file or directory
Mar 20 00:03:46 ubuntu systemd-udevd[139]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Mar 20 00:03:46 ubuntu sshd[171]: error: Bind to port 22 on 10.0.0.80 failed: Cannot assign requested address.
Mar 20 00:03:46 ubuntu sshd[171]: fatal: Cannot bind any address.
Mar 20 00:03:46 ubuntu systemd[1]: ssh.service: Main process exited, code=exited, status=255/n/a
Mar 20 00:03:46 ubuntu systemd[1]: ssh.service: Failed with result 'exit-code'.
Mar 20 00:03:46 ubuntu systemd[1]: Failed to start OpenBSD Secure Shell server.

Mar 20 00:03:46 ubuntu systemd[1]: Started A high performance web server and a reverse proxy server.
Mar 20 00:03:46 ubuntu systemd[1]: Started The PHP 7.2 FastCGI Process Manager.
Mar 20 00:03:47 ubuntu networkd-dispatcher[153]: No valid path found for iwconfig
Mar 20 00:03:47 ubuntu networkd-dispatcher[153]: No valid path found for iw
Mar 20 00:03:47 ubuntu systemd[1]: Started Dispatcher daemon for systemd-networkd.

Mar 20 00:03:48 ubuntu systemd[1]: Reached target Timers.
Mar 20 00:05:14 ubuntu systemd[1]: dev-getty.device: Job dev-getty.device/start timed out.
Mar 20 00:05:14 ubuntu systemd[1]: Timed out waiting for device dev-getty.device.
Mar 20 00:05:14 ubuntu systemd[1]: Dependency failed for Serial Getty on getty.
Mar 20 00:05:14 ubuntu systemd[1]: serial-getty@getty.service: Job serial-getty@getty.service/start failed with result 'dependency'.
Mar 20 00:05:14 ubuntu systemd[1]: dev-getty.device: Job dev-getty.device/start failed with result 'timeout'.
Mar 20 00:05:14 ubuntu systemd[1]: Reached target Login Prompts.
Mar 20 00:05:14 ubuntu systemd[1]: Reached target Multi-User System.
Mar 20 00:05:14 ubuntu systemd[1]: Reached target Graphical Interface.
Mar 20 00:05:14 ubuntu systemd[1]: Starting Update UTMP about System Runlevel Changes...
Mar 20 00:05:14 ubuntu systemd[1]: Started Update UTMP about System Runlevel Changes.
Mar 20 00:05:14 ubuntu systemd[1]: Startup finished in 1min 30.375s.

Mar 20 00:09:03 ubuntu systemd[1]: Starting Clean php session files...
Mar 20 00:09:03 ubuntu systemd[1]: Started Clean php session files.

edit:
Debian container works fine

AreYouLoco · March 20, 2019, 12:42pm

Just after you restart the container please attach to the shell and run:
systemctl status sshd.service

So we know a bit more what have failed.

omichalek · March 20, 2019, 10:15pm

I am actually just battling more critical issue with the Omnia right now: Update to 3.11.2 broke my DHCP
(it might be connected?)

this is the output:

root@turris:~# lxc-attach -n ubuntu -- systemctl status sshd.service

● ssh.service - OpenBSD Secure Shell server
   Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Wed 2019-03-20 23:09:17 CET; 3min 4s ago
  Process: 155 ExecStart=/usr/sbin/sshd -D $SSHD_OPTS (code=exited, status=255)
  Process: 144 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
 Main PID: 155 (code=exited, status=255)
   CGroup: /system.slice/ssh.service

Mar 20 23:09:17 ubuntu systemd[1]: Starting OpenBSD Secure Shell server...
Mar 20 23:09:17 ubuntu systemd[1]: ssh.service: Main process exited, code=exited, status=255/n/a
Mar 20 23:09:17 ubuntu systemd[1]: ssh.service: Failed with result 'exit-code'.
Mar 20 23:09:17 ubuntu systemd[1]: Failed to start OpenBSD Secure Shell server.

omichalek · March 20, 2019, 11:35pm

So it seems that my container did not get an IP address assigned due to the other issue. For some reason the rest of the network still worked (until the router got restarted I guess).
With help from another member I fixed my broken DHCP server configuration. The container now works so I hope it did not due to the DHCP misconfiguration…
Thank you for your help!