VPN policy based routing possible?

p4p4 · June 9, 2018, 8:51am

After installing OS 3.10.1 I cant’t get VPN PBR to work anymore. The VPN connection is open, but the rules aren’t used. The clients that should go over the VPN tunnel don’t have any connection to the internet. I also reinstalled the packages, but still doesn’t work. How is it for you?

anon50890781 · June 9, 2018, 9:14am

@stangri

Same here, seems like the work around the reboot issue with pressing Save & Apply (/cgi-bin/luci/admin/services/vpn-policy-routing) is not working, perhaps a bug in the gui.

Only way I got it working by connecting to the TO over ssh and then from the command line

/etc/init.d/vpn-policy-routing reload

p4p4 · June 9, 2018, 9:31am

Thanks alot @anon50890781 ! that did it for me too! Working now. So i have too ssh into the TO after every reboot for now.
@stangri is there any chance that this can be fixed for the Turris router in your PBR package?

anon50890781 · June 9, 2018, 9:36am

Seems the only way at the moment. Best would be to get the reboot issue fixed.

p4p4 · June 9, 2018, 11:15am

I tried putting the command in the startup script:

openvpn --cd /etc/openvpn --config /etc/openvpn/my_expressvpn_switzerland.ovpn --remote switzerland-2-ca-version-2.expressnetw.com 1195 &

/etc/init.d/vpn-policy-routing reload

exit 0

but that didn’t work either to get the reboot issue fixed.

stangri · June 9, 2018, 11:33am

If you guys can help me figure out why VPR isn’t being reloaded automatically when VPN connects on Turris I’d be much obliged.

On a freshly booted up router, could you please do:
logread -e vpn-policy-routing
and
logread -e vpn
And post the results of both?

@anon50890781 – I’ve replied to your bug on GitHub, but come to think of it, if the CLI reload works but WebUI doesn’t cause the reload, it may have something to do with the new Turris update. I’m not sure what the expected behaviour for luci is for when there’re no changes on the page, but Save & Apple is clicked, I believe it should do nothing.

p4p4 · June 9, 2018, 11:50am

root@turris:~# logread -e vpn-policy-routing
Failed to find log object: Not found

root@turris:~# logread -e vpn
Failed to find log object: Not found

doesn’t work. typed that after fresh reboot without

/etc/init.d/vpn-policy-routing reload

before

anon50890781 · June 9, 2018, 12:12pm

With TOS 3.10 that been the workaround the reboot issue and got PBR working after a reboot. With the update to TOS 3.10.1 it is not anymore however. Either might be the correct/expected behaviour.

Since logread did not produce any result I checked the syslog manually and there no correspondig entries

stangri · June 9, 2018, 1:50pm

Wait, there should be something at least for the OpenVPN client connecting to a server. Can either of you gentlemen do the logread --help please?

Also, the output of
for F in /etc/init.d/* ; do $F enabled && echo $F on || echo $F **disabled**; done
the line for VPR at least, ideally OpenVPN too.

anon50890781 · June 9, 2018, 2:24pm

Probably not if the log file is specified otherwise, e.g. option log '/tmp/log/openvpn.log' in “/etc/config/openvpn”

root@router:~# for F in /etc/init.d/* ; do $F enabled && echo $F on || echo $F **disabled**; done

Summary

/etc/init.d/asm1062-fix on
/etc/init.d/atd on
/etc/init.d/boot on
/etc/init.d/br2684ctl disabled

/etc/init.d/btrfs-scan on
/etc/init.d/conntrackd disabled
/etc/init.d/cron on
/etc/init.d/ddns disabled
/etc/init.d/dnsmasq on
/etc/init.d/done on
/etc/init.d/firewall on
/etc/init.d/foris-controller on
/etc/init.d/foris-ws on
/etc/init.d/fstab on
/etc/init.d/haas-proxy on
/etc/init.d/hd-idle on
/etc/init.d/led_autoconfig on
/etc/init.d/libatsha204 on
/etc/init.d/lighttpd on
/etc/init.d/lvm2 on
/etc/init.d/lxc-auto on
/etc/init.d/minidlna disabled
/etc/init.d/mountd on
/etc/init.d/netdata disabled
/etc/init.d/nethist on
/etc/init.d/network on
/etc/init.d/nfsd disabled
/etc/init.d/ntpd on
/etc/init.d/ntpdate on
/etc/init.d/openvpn on
/etc/init.d/pakon-handler on
/etc/init.d/pakon-monitor on
/etc/init.d/portmap disabled
/etc/init.d/pptpd disabled
/etc/init.d/rainbow on
/etc/init.d/relayd on
/etc/init.d/resolver on
/etc/init.d/rpcd on
/etc/init.d/rsyncd disabled
/etc/init.d/setup_led on
/etc/init.d/sfpled disabled
/etc/init.d/sfpswitch on
/etc/init.d/smartd on
/etc/init.d/socat disabled
/etc/init.d/sqm on
/etc/init.d/srv on
/etc/init.d/sshd on
/etc/init.d/start-indicator on
/etc/init.d/suricata-pakon on
/etc/init.d/sysctl on
/etc/init.d/sysfixtime on
/etc/init.d/syslog-ng on
/etc/init.d/sysntpd disabled
/etc/init.d/system on
/etc/init.d/telnet disabled
/etc/init.d/umount disabled
/etc/init.d/unbound disabled
/etc/init.d/update_mac on
/etc/init.d/updater on
/etc/init.d/usbmode on
/etc/init.d/vpn-policy-routing on
/etc/init.d/watchdog_adjust on

root@router:~# logread --help

Summary

logread: unrecognized option: -
Usage: logread [options]
Options:
-s Path to ubus socket
-l Got only the last ‘count’ messages
-e Filter messages with a regexp
-r Stream message to a server
-F Log file
-S Log size
-p PID file
-h Add hostname to the message
-P Prefix custom text to streamed messages
-f Follow log messages
-u Use UDP as the protocol
-0 Use \0 instead of \n as trailer when using TCP

dibdot · June 9, 2018, 7:45pm

Hi Stan, if you need some debugging support feel free to contact me - I have a turris test device at home, this device is “special” in many ways … e.g. logread is not functional, they use syslog-ng which writes to /var/log/messages …

anon50890781 · June 10, 2018, 7:15am

That could be the likely root cause of the (re)boot issue for VPBR service not being able to monitor interface changes. Updated the github ticket

It is likely to change with TOS 4, which is then OpenWRT vanilla with TO patches plus the TO Foris gui.
Though there is no eta for TOS 4 and what the TO patches exactly will be either. I would reckon that syslog-ng will remain in place though and thus logread not working, just in case VPR interface monitoring relying on it.

stangri · June 10, 2018, 1:19pm

Thank you Dirk, that’d be quite useful. So if I use logger in my script, does it do anything on Turris?

Well, it shouldn’t be. I only use system log for the status output, the interface updates are registered thru PROCD and should be CC backwards-compatible.

anon50890781 · June 10, 2018, 1:40pm

Well, there was hope to get the (re)boot issue sorted but then it was poking the wrong avenue.

dibdot · June 10, 2018, 3:15pm

Yes, logger works normally - output is written to /var/log/messages.

e.g.:

console:
    root@turris:/etc/init.d# /etc/init.d/vpn-policy-routing reload
    Creating table 'wan/eth1/192.168.0.1' [✓]
    vpn-policy-routing 0.0.2-2 started on wan/eth1/192.168.0.1 [✓]
    vpn-policy-routing 0.0.2-2 monitoring interfaces: wan [✓]

log:
    2018-06-10 17:16:43 err modprobe[]: xt_set is already loaded
    2018-06-10 17:16:43 err modprobe[]: ip_set is already loaded
    2018-06-10 17:16:43 err modprobe[]: ip_set_hash_ip is already loaded
    2018-06-10 17:16:43 notice [5344]: Creating table 'wan/eth1/192.168.0.1' [✓]
    2018-06-10 17:16:43 notice [5344]: service started on wan/eth1/192.168.0.1 [✓]
    2018-06-10 17:16:44 notice [5344]: service monitoring interfaces: wan [✓]

stangri · June 13, 2018, 12:59pm

If any of you gentlemen could post the output of: grep 'vpn' /var/log/messages and grep 'vpn-policy-routing' /var/log/messages right after boot and then after you manually invoke reload thru CLI command I can look into why it’s failing to properly initialize on boot.

p4p4 · June 13, 2018, 7:20pm

If I type the commands, nothing happens, no output…

root@turris:~# grep ‘vpn’ /var/log/messages
root@turris:~# grep ‘vpn-policy-routing’ /var/log/messages

p4p4 · June 13, 2018, 7:28pm

I found this manually in the messages log:

2018-06-13 21:17:48 notice [2221]: Creating table ‘VPN_Client/tun0/0.0.0.0’ [✗]
2018-06-13 21:17:48 notice [2221]: Routing ‘Local_IP’ via VPN_Client [✓]
2018-06-13 21:17:48 notice [2221]: Routing ‘IP Range’ via wan [✓]
2018-06-13 21:17:48 notice [2221]: service started on wan/eth1/XXX.XXX.XX.X with errors [✗]
2018-06-13 21:17:48 err kresd[2025]: [priming] cannot resolve address ‘m.root-servers.net.’, type: 1
2018-06-13 21:17:48 notice [2221]: ERROR: Failed to set up ‘VPN_Client/tun0/0.0.0.0’
2018-06-13 21:17:48 notice [2221]: service monitoring interfaces: wan VPN_Client [✓]
2018-06-13 21:17:48 info smartd[3554]: smartd 6.5 2016-05-07 r4318 [armv7l-linux-4.4.134-8f7f4132dc88fb7034ce9648e5961be5-0] (localbuild)
2018-06-13 21:17:48 info smartd[3554]: Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
2018-06-13 21:17:48 info smartd[3554]: Opened configuration file /etc/smartd.conf
2018-06-13 21:17:48 info smartd[3554]: Configuration file /etc/smartd.conf parsed.
2018-06-13 21:17:48 info smartd[3554]: Device: /dev/hdb, open() failed: No such device
2018-06-13 21:17:48 info smartd[3554]: Monitoring 0 ATA/SATA, 0 SCSI/SAS and 0 NVMe devices
2018-06-13 21:17:48 info smartd[3557]: smartd has fork()ed into background mode. New PID=3557.
2018-06-13 21:17:48 emerg turris: Router Turris successfully started.
2018-06-13 21:17:48 info procd: - init complete -
2018-06-13 21:17:50 notice netifd: Interface ‘VPN_Client’ is enabled
2018-06-13 21:17:50 notice netifd: Network device ‘tun0’ link is up
2018-06-13 21:17:50 notice netifd: Interface ‘VPN_Client’ has link connectivity
2018-06-13 21:17:50 notice netifd: Interface ‘VPN_Client’ is setting up now
2018-06-13 21:17:50 notice netifd: Interface ‘VPN_Client’ is now up

Does this help? (I XXX’ed the IP in the post)

Edit: When I then manually reload I get this (XXX’ed the IP’s again):

2018-06-13 21:32:58 notice [4143]: Creating table ‘wan/eth1/XXX.XXX.XX.X’ [✓]
2018-06-13 21:32:58 notice [4143]: Creating table ‘VPN_Client/tun0/XX.XX.X.XXX’ [✓]
2018-06-13 21:32:58 notice [4143]: Routing ‘Local_IP’ via VPN_Client [✓]
2018-06-13 21:32:58 notice [4143]: Routing ‘IP Range’ via wan [✓]
2018-06-13 21:32:58 notice [4143]: service started on wan/eth1/XXX.XXX.XX.X VPN_Client/tun0/XX.XX.X.XXX [✓]
2018-06-13 21:32:58 notice [4143]: service monitoring interfaces: wan VPN_Client [✓]

Is it the problem that he creates a table with 0.0.0.0 for tun0 at the beginning?

stangri · June 13, 2018, 8:12pm

Thanks for investigation. I’ve asked to grep certain things as I was hoping the name of the service would also be in the log, alas it was not.

Anyways, to solve the boot-up problem you can try running (once):
sed -i '$i \sleep 20 && /etc/init.d/vpn-policy-routing reload &' /etc/rc.local

That won’t fix automatic reload for when OpenVPN goes down and then back up tho.

Can you guys tag Turris devs in this thread?

I’m using PROCD on 17.x/18.x to be notified when some interfaces are up: https://github.com/stangri/openwrt_packages/blob/master/vpn-policy-routing/files/vpn-policy-routing.init#L550-L556

This doesn’t seem to work on Turris. Do I need to dynamically create hotplug scripts then?

p4p4 · June 13, 2018, 8:23pm

@Pepe

Hi Pepe, can you answer stan’s question? Would be great if we would get VPN PBR working on Turris!