Ugrade from 5.3.1 to 5.3.2 breaks OpenVPN

This is really upsetting, had to roll back few days in Snapshots. I got the following errors in the system logs, when I tried to reinstall OpenVPN:

Dec  8 17:18:44 turris firewall: Reloading firewall due to ifup of vpn_turris (tun_turris)
Dec  8 17:18:47 turris foris-controller[8477]: WARNING:zeroconf:Error sending through socket 17
Dec  8 17:18:47 turris foris-controller[8477]: Traceback (most recent call last):
Dec  8 17:18:47 turris foris-controller[8477]:   File "/usr/lib/python3.7/site-packages/zeroconf/__init__.py", line 2826, in send
Dec  8 17:18:47 turris foris-controller[8477]: OSError: [Errno 19] No such device

It is really upsetting because the device is 1000km away and it is not easy to get a person and troubleshoot. Do you have any solution how to avoid being locked out from a device due to the automatic updates?

The usual support questions:

  • Are we talking about OpenVPN server or OpenVPN client?
  • What is exactly broken? You said that the update breaks OpenVPN. How?
    OpenVPN was not updated.
  • How did you configure OpenVPN?
  • Which device do you have?

I would like to help you more, but we need some more details because what you provided is completely irrelevant. There is an issue with getting rid of this warning.

Please could you follow this:
https://docs.turris.cz/geek/contributing/issues/

and if you also prefer this one:
https://docs.turris.cz/basics/support/

Yes. Configure updates with approval. Before giving the approval, ssh into the router, and run something like sleep 1800 && schnapps rollback; reboot inside a screen session. If the update goes okay, terminate the screen before it rolls back.

3 Likes
  1. OpenVPN server
  2. The system successfully upgraded, however OpenVPN server is not running. Hitting the button in Luci to start it does nothing
  3. Everything is done in reForis, generated CA and then the rest is everything standard, client certificates also created via reForis
  4. None of my clients were able to connect, so it was def a problem with the server

I do not want to debug a system, I want a system that is robust enough that it survives a reboot, update, etc. Am I asking for too much? It was an upgrade from v5.3.1 to v5.3.2 :-/

This should be a standard setup in reForis, if an approval is set for updates, it should be also set when the system reboots. For example, if one does not approves the system after reboot within cca 5 mins, it should roll back automatically.

Still, I am curious, which device do you have? As requested, you need to go through the steps which I mentioned. Otherwise, I am not able to help you further, because in that case, I can only guess what could be wrong.

Anyway, you said that none of your clients were possible to connect to the server. Was it after the reboot on Turris MOX/Turris Shield or before the reboot when the update was applied? Was the LAN part working after reboot or was it necessary to unplug and plug the power cord back?

If you hit the button in the LuCI and it doesn’t do anything. Was it possible that the OpenVPN server was already running? Are there any logs, which you can share with us?

I just received an automatic update here (about an hour or so ago), and OpenVPN also broke.

Also, experiencing the “start” button not doing anything.

Device Turris Omnia
reForis version 1.1.2
Turris OS version 5.3.2
Turris OS branch HBS
Kernel version 4.14.254

I don’t see anything in the logs regarding the VPN.

I tried setting up a new OpenVPN connection as well, and when I tried to start it, it doesn’t do anything.

It is a TO 2020. I received an update with a list of packages that I approved, through the wokring OpenVPN connection. This means it was still functioning after the actual update, however, the device requested a reboot, so I granted it. After that the device rebooted, and everything worked fine (local clients on LAN had access to the Internet), except the OpenVPN server. LAN was also working, my port-forwarding was working, the server behind the device was reachable (another computer). All was fine, the only thing that was visibly not working was the OpenVPN server.

I tried disabling and re-enabling/stop+start the OpenVPN server. In reForis the disable/enable seemed to work, in LuCi hitting the STOP button did nothing. I tried uninstalling the openvpn package and reinstalling it. Nothing worked. Finally, out of desperation and not having time to further troubleshoot, I rolled back to a time when the server was working. After applying it, the openvpn came back as normal and all my clients were able to connect again.

I emphasised that I created the certificates through reForis, as my understanding is that the deployment of the server should be smooth and easy. Hence I am not digging deeper, the only errors I have seen were the ones listed above in the logs, otherwise there were timed out messages from the clients, I remember from the logs that after the reinstall the server seemed to be “running”, but it was not. There were no changes made to the firewall.

Apologies I do not have the log from before the reboot, I had enough issues to instruct the non-techie person on premises what to do to roll back the firmware.

Thank you for mentioning this! I was able to rollback as well, and it fixed my issue too! I didn’t even know rolling back was an option! :blush: But I’m glad it’s there and that it worked!

1 Like

Very strange I updated to 5.3.2, rebooted the router and OpenVPN is fine. Not sure what the difference is. I have an original Omnia from the Indiegogo campaign. OpenVPN config is an old one that was generated by Foris “easy” VPN setup a long time ago. Not sure why this is breaking for others. Maybe the reForis generated configs are a problem? Mine was from Foris not reForis.

Hello,
you should still have the snapshot with 5.3.2 saved on your Omnia. Can you please export the snapshot with TOS5.3.2 and send it to us for investigation?

You can do it by issuing these commands:
To list saved snapshots, type schnapps list

root@turr1x:~# schnapps list
    # | Type      | Size        | Date                      | Description
------+-----------+-------------+---------------------------+------------------------------------
    1 | rollback  |   226.81MiB | 2020-06-18 19:26:12 +0200 | Rollback to snapshot factory
    3 | single    |     4.73MiB |                           | 
  409 | rollback  |   188.08MiB | 2021-06-18 13:39:57 +0200 | Rollback to snapshot 403
  412 | rollback  |    75.52MiB | 2021-06-18 13:51:49 +0200 | Rollback to snapshot 403
  650 | pre       |     9.63MiB | 2021-11-22 20:36:42 +0100 | Automatic pre-update snapshot (TurrisOS 5.3.1)
  651 | post      |     9.45MiB | 2021-11-22 20:36:48 +0100 | Automatic post-update snapshot (TurrisOS 5.3.1)
  652 | pre       |    11.95MiB | 2021-11-24 13:06:11 +0100 | Automatic pre-update snapshot (TurrisOS 5.3.1)
  653 | post      |     9.41MiB | 2021-11-24 13:06:34 +0100 | Automatic post-update snapshot (TurrisOS 5.3.1)
  654 | pre       |    10.23MiB | 2021-11-26 04:06:00 +0100 | Automatic pre-update snapshot (TurrisOS 5.3.1)
  655 | post      |     9.42MiB | 2021-11-26 04:06:28 +0100 | Automatic post-update snapshot (TurrisOS 5.3.1)

To export the snapshot (in this example 655) in current directory, type command schnapps export 655 ./

root@turr1x:~# schnapps export 655 ./
./
./overlay/
./dev/
./mnt/
./mnt/.snapshots/
./sbin/
./sbin/ubusd
./sbin/reload_config
./sbin/upgraded
./sbin/udevtrigger
./sbin/init
<output ommited>
./netmetr-client/.git/hooks/fsmonitor-watchman.sample
./netmetr-client/.git/hooks/pre-applypatch.sample
./boot.scr
Snapshot 655 was exported into localhost on ./ as schnapps-medkit-655

You will find the snapshot in the current working directory in the form of tar.gz archive.

Where can we submit the export?

We do not have any service (yet) for upload of big files, so in my opinion the best way will be to encrypt the whole tar.gz with password by running command
openssl aes-256-cbc -a -salt -pbkdf2 -in snapshot.tar.gz -out snapshot.tar.gz.enc
It will ask for the password (you will send the password as a private message to me), encrypts the file and you upload the encrypted file on some filesharing site, and send me the link for that file.

I will download it, decrypt it with the password, import it in my omnia and analyze what happened.

Just double-checking. Will this export contain the openvpn client certificates and keys?

Yes it will containg your whole root filesystem.

Can you clarify pls one more thing for me? Does this command survive the reboot required by the TO update? Or you mean running a terminal update of the packages, and observe functionality? I am concerned that some of the packages will require rebooting the device and the commands in screen will get interrupted by the restart and will never run as a consequence of the restart…

No, the commands running in screen will not survive reboot. The update-related reboots always need your approval, but I’ve seen cases where that was not the case and a reboot happened without me approving it. If you want to be also resilient to this case, the best you can do is add a cron job or an at command in rc.local (be sure to run it daemonized (or in a screen session) - blocking calls in rc.local would definitely break the startup).