Mwan3 after update not working

Hi guys,

i configured my mwan3 module a few weeks ago and everything was working. I have 4 simultanious VPN connections which routes different network clients through the tunnels. After the update all interfaces are showing up as offline, even the wan interfaces comes up offline. The clients which should be routed throught the tunnels have no internet access anymore, even if the tunnels are online. The clients which do not use a tunnel can still access the internet.

Is there any known issue coming up with the latest update?

Thanks in advance for your support!

1 Like

Hello,
we have similar issue. We have zones: WAN (wan,wan6, wan2), lan + turris quest. After the upgrade it seems it routes only via wan2 or wan (not both). As we need to get outside connections form wan2 it helps when we restart the wan2 interface and than it routes thru it. Do you have any clue please?

Does not help for meā€¦ Even restarting the complete router has no effectā€¦

I lost mwan3 functionality in the last couple of days. All three wan interfaces show red / offline in the mwan3 overview. Two of the interfaces are operational, and I can ping the gateway and tracking IP on both interfaces. One device is out of service for unrelated reasons. When I select Check IP Rules I get a message ā€œMissing both of the required interface IP rulesā€. When I select Check Routing Table, I get the message ā€œMissing required interface routing table Nā€, where N is 2 or 3 depending on which interface I test. IF up/down make no difference. Rebooting multiple times has not helped. I even removed an entire interface from mwan3, saved, rebooted and rebuilt - no difference. IP addresses and other config all appear correct on the Omnia, and I can ping / browse / email when I plug my laptop into an available DMZ switch port between the Omnia and either of the above-mentioned in-service link devices. I even tried setting the tracking IP to blank to disable tracking. Interface still stays red / offline. Iā€™m still researching. Have not found solutions yet but have not exhausted all search options. I would appreciate any help / suggestions / insights on this.
Thanks,
Robert

I think mwan3 is not running anymore:

mwan3 status shows nothingā€¦

unfortunately i donā€™t know why.
Can anybody help? Or is there anybody without issues and mwan3 running?

Doing further research. I am still getting my head around the problem. These posts and a number of similar ones share the ā€œall offlineā€ pattern, but are not 100% what we are seeing on Omnia. Reposting here to add info to the discussion. I will post my results if/as I progress solving.

From the previous discussion linkā€¦ mwan3 requires that conntrac be running on the interfaces. Checked my Start Up page. Conntracd was disabled. I enabled, started, sent ifup to both active interfaces. No change thus far. Not sure why conntracd was disabled either.

Oddly when I click Detailed Status I see ā€œNo detailed status information availableā€, which does not seem right. Iā€™ve not used that feature much - generally go from the live view to diagnostics tab. Not sure if the lack of detail info is new or longstanding.

same here: no detailed information available.
Think mwan isnā€™t working at allā€¦

Same with me: mwan3 stopped working.
Running mwan status command via ssh, I get an iptables error:

Current ipv4 policies:
iptables v1.6.1: canā€™t initialize iptables table `mangleā€™: Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.

Current ipv6 policies:
ip6tables v1.6.1: canā€™t initialize ip6tables table `mangleā€™: Table does not exist (do you need to insmod?)
Perhaps ip6tables or your kernel needs to be upgraded.

mwan3 status is also not working anymoreā€¦
Seems like mwan3 is completely dead here!
Any ideas?

Hello guys,

Thank you for reporting to us bug about mwan3. We have created the issue on our Gitlab.

Iā€™ve looked it and upgraded mwan3 to the latest version. It will be part of Turris OS 3.11.1, which weā€™d like to release to RC soon.

1 Like

Hi,
in 3.11.1 it looks better, but mwan3 is still not working:

root@qicala:~# mwan3 restart
ip: invalid argument '0x3d00/0x3F00' to 'fwmark'
ip: invalid argument '0x3e00/0x3F00' to 'fwmark'
ip: invalid argument '0x3d00/0x3F00' to 'fwmark'
ip: invalid argument '0x3e00/0x3F00' to 'fwmark'
ip: invalid argument '0x100/0x3F00' to 'fwmark'
/sbin/hotplug-call: /etc/hotplug.d/iface/15-mwan3: line 481: network_get_metric: not found
uci: Entry not found
uci: Entry not found
ip: invalid argument '0x3d00/0x3F00' to 'fwmark'
ip: invalid argument '0x3e00/0x3F00' to 'fwmark'
ip: invalid argument '0x3d00/0x3F00' to 'fwmark'
ip: invalid argument '0x3e00/0x3F00' to 'fwmark'
ip: invalid argument '0x200/0x3F00' to 'fwmark'
/sbin/hotplug-call: /etc/hotplug.d/iface/15-mwan3: line 481: network_get_metric: not found
uci: Entry not found
uci: Entry not found
uci: Entry not found
uci: Entry not found
root@qicala:~# uci: Entry not found
uci: Entry not found

Here is my config:

config globals 'globals'
        option mmx_mask '0x3F00'
        option rtmon_interval '5'

config member 'WiFi'
        option interface 'Bla'
        option weight '10'
        option metric '5'

config member 'Mobile'
        option interface 'Vodafone'
        option metric '10'
        option weight '10'

config rule 'https'
        option sticky '1'
        option dest_port '443'
        option proto 'tcp'
        option use_policy 'Failover'
        option timeout '60'

config rule 'default_rule'
        option dest_ip '0.0.0.0/0'
        option proto 'all'
        option sticky '0'
        option use_policy 'Failover'

config interface 'Bla'
        option enabled '1'
        option reliability '1'
        option count '3'
        option timeout '2'
        option interval '5'
        option down '3'
        option up '3'
        list track_ip '8.8.8.8'
        list track_ip '8.8.4.4'
        list track_ip '1.1.1.1'

config interface 'Vodafone'
        option enabled '1'
        option reliability '1'
        option timeout '2'
        option interval '5'
        option down '3'
        option up '3'
        option count '3'
        list track_ip '8.8.8.8'
        list track_ip '8.8.4.4'
        list track_ip '1.1.1.1'

config policy 'Failover'
        list use_member 'WiFi'
        list use_member 'Mobile'
        option last_resort 'unreachable'

Itā€™s not working even without mmx_mask, because then it uses some default values.

Interface status:
 interface Bla is error and tracking is active
 interface Vodafone is error and tracking is active

Current ipv4 policies:
Failover:
 Bla (100%)

Current ipv6 policies:
Failover:
 unreachable

Directly connected ipv4 networks:
...

i am receiving same messages like @jermenā€¦
Still not working

@Pepe: Did mwan work for you in tests? Maybe you can check again and fixā€¦

Same messages here.
It sounds very much like the following issue, but ip-full is installed.
mwan3: not working on 4.9.58 r5218-f90f94d2c1

Iā€™ve tried also ip-tiny package, but problem is the same. Also when I made some lame ā€œpatchesā€ for those network_get_metric errors, itā€™s still broken.

The main problem is imho in that ā€˜ipā€™ call. Without it, the rest of the environment configuration doesnā€™t make much sense (as far as I understand).

Maybe try ip-full version update? Because in mwan3.sh that line with ā€˜ipā€™ call is from 2017 (but I donā€™t know how about parameters/variables content).

Similar here, mwan3 unusable as it is. There is the network_get_metric function missing in network.sh and even with ip-full installed and calls to it changed to be /usr/sbin/ip instead of ip it is not any better (to make sure not to call busybox).

Unless someone finds a fix and with the season holidays in progress this might not get sorted soon. Thus a workaround is perhaps to either rollback to TOS 3.10.8 or try alternatively VPN policy based routing possible?, which works in 3.11.1 but does not provide load balancing.

My mwan3 also broke after the recent updates. It is a simple WAN/WWAN failover. I fixed it by doing the following:

opkg update
opkg remove luci-app-mwan3
opkg remove mwan3
mv /etc/config/mwan3 /root/mwan3
opkg install mwan3 luci-app-mwan3

I then changed the first couple of lines of /lib/mwan3/mwan3.sh to have the full path to ip, ipset and iptables:

IP4="/usr/sbin/ip -4"
IP6="/usr/sbin/ip -6"
IPS="/usr/sbin/ipset"
IPT4="/usr/sbin/iptables -t mangle -w"
IPT6="/usr/sbin/ip6tables -t mangle -w"

After that, I rebuilt my rules in /etc/config/mwan3 (basically, search-replaced all wanb to wwan, set all IPv6 rules to disabled, removed the default http policy and changed the default rule to policy wan_wwan) and then ran /usr/sbin/mwan3 restart - and it worked!

Turns out, with the new update, you need the full path to ip for fwmark to work, and if you donā€™t have IPv6 rules in your config (which I donā€™t, since I donā€™t have IPv6 natively on either of my connections), the rtmon script break everything (and spams the logs with uci errors).

The network_get_metric error is still there, but it does not appear to affect my setup as far as I can tell.

@Pepe, you may want to update your bug with the above info.

6 Likes

Adding the path to the ip commands did get back the mwan3 functionality. To prevent the metric errors, I also added

# determine the metric of the given logical interface
# 1: destination variable
# 2: interface
network_get_metric() { __network_ifstatus "$1" "$2" ".metric"; }

to /lib/functions/network.sh as quick fix.
I found the method at the master branch of openwrt at https://github.com/openwrt/openwrt/blob/master/package/base-files/files/lib/functions/network.sh.

At the moment, I also have turned off automatic updates to be sure, non of the changed files are overwritten by an update and break the installation again.

1 Like