OMNIA: Vlan on DSA port breaks arp responses (TOS 4.0.5)

Not sure I understand since frames between 10.0.0.5 and 10.0.0.98 exhibit

Only the traffic between 10.0.0.5 and 10.0.0.14 does not?


It seems that the switch is not retiring a MAC client address from its database instantly when the client gets disconnected from one of the switch’s downstream ports.

Address aging makes room for new active addresses by ensuring that when a node is disconnected from the network segment or when it becomes inactive, its entry is removed from the address database. An address is removed from the database after a predetermined interval from the time it last appeared as an Ingress frame’s source address (SA).
This interval is programmable in the 88E6060 device. The default Aging interval is about 5 minutes (282 to 304 seconds), but it can be set from 0 seconds (i.e., aging is disabled) to 4,080 seconds in 16 second increments.

That is probably within common industry standards. One might expect the SWITCHDEV (bridge fdb) to populate a MAC client address from the WLan port to switch’s address database but it looks like it does not happen - perhaps since it is an upstream port - from the switch perspective.
It may even conflict bridge fdb getting the same client MAC address from two different downstream ports (Lan and WLan) within that ageing period (5 minutes (282 to 304 seconds) specified by the switch’s address database - but I am not sure about that.

If you look at Linux it commonly is configured either as server or desktop, latter with WLan (as client) support. The server configuration, such as the router is, is commonly/mostly:

  • not with WLan but Lan, and
  • the concept of switching a client from Lan to WLan (within those 300 seconds nonetheless) may not be a common concept and thus not taken into consideration (negligible).

Not sure what you want to achieve and what it is that you expect. That aside, I reckon that bridging WAN with LAN is not a good concept since it makes the LAN traffic visible (sniff-able) on the WAN port (least potentially for the ISP). That is unless that WAN port is used as Lan port.

10.0.0.5 is where the capture was taken.
This machine is directly connected to Omnia 1 on lan3. That port is part of the lan bridge and set to VLAN 1 PVID untagged using bridge v. Now maybe I’m misunderstanding something, but it seems to me this means there should never be any tagged traffic on that port.

10.0.0.14 is Omnia 1
Omnia 1 does a lot (internet, iptv, vpn), but the relevant part here is that lan 4 is set to vlan 1 & 2, tagged. lan4.1 and lan3 are in the same bridge.
This port is connected to Omnia 2 which has lan port 4 set to vlan 1 & 2, tagged as well. That Omnia has a bridge with lan4.1 and the other lan ports. 10.0.0.98 is connected to one of those ports and running a ping (but never getting a reply).


That’s one theory at least, it seems it would explain the observed behavior. It would also explain why @yorik is able to solve it by forcing all traffic through the cpu. Therefore…

…if you connect a device to one of the lan ports the switch will create a mac entry for that. If you then move the device to the wan port you expect the switch to keep the entry and never notice traffic for that device should now go to the CPU port.
So you can start with the device on a lan port. Run a ping from a device on another lan port. That should work and the CPU should never see that traffic. Now if you move the device to the wan port and the switch keeps it’s old entry around it will send traffic to the old lan port instead of the CPU port and the device will be unreachable until the entry in the switch expires.
That, to me at least, seems to what would be the result if the theory above is true.

This Omnia isn’t anywhere near the internet connection, don’t worry :wink:

If router builders normally connect Wlan chips directly to the switch that could be true. But I don’t know how uncommon it is to have Wlan behind the CPU. Seems unavoidable when using PCI-E…

So I tried this, here’s the network config used:

config interface 'loopback'
        option ifname 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix 'fdc8:b61f:14b9::/48'

config interface 'lanip'
        option proto 'static'
        option ipaddr '10.0.0.15'
        option netmask '255.255.255.0'
        option gateway '10.0.0.14'
        option dns '194.109.104.104 194.109.9.99 194.109.6.66'
        option type 'bridge'
        option ifname 'eth2 lan0 lan1 lan2 lan3 lan4'

I configured the above initially without restarting the router. I tested with just two devices both pinging each other.

Test 1 was a naive test just plugging the device from a lan port into the wan port. On that test the connection always came back after quickly. I guess the Omnia’s switch has no issues in this case because it will loose the link on the lan port which probably invalidates all mac entries for that port instantly.

For test 2 I had the test device to an external switch which was connected to a lan port, that way I could move the device to the wan port without the link on the Omnia’s lan port going down. This is where things started going wrong. When moving the device from the external switch to the wan port it took 3-6 minutes before I got a reply again. In the other direction everything was fine.
I ran the test again with tcpdump running on the Omnia. When the test device was on the Wan port it would see the icmp trafic, when the test device was connected to the external switch it would not, confirming the traffic never hit the CPU.
As far as I can tell this confirms the Omnia’s switch does retain mac address entries for devices that move to the cpu (atleast when running Turris OS 4.0.5)

Test 3 was done after rebooting the device, just to be sure. That got the same result as test 2.

For test 4 I did a factory reset, dropped the above config back in and rebooted. Again, same results as test 2.

Test 5 was done after reflashing using omnia-medkit-202003312359.tar.gz. I reproduced the same setup using this config:

config interface 'loopback'
        option ifname 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix 'fd7e:65de:8bfe::/48'

config interface 'lan'
        option force_link '1'
        option type 'bridge'
        option proto 'static'
        option netmask '255.255.255.0'
        option ip6assign '60'
        option ifname 'eth0 eth1 eth2'
        option ipaddr '10.0.0.15'
        option gateway '10.0.0.14'

config switch
        option name 'switch0'
        option reset '1'
        option enable_vlan '1'

config switch_vlan
        option device 'switch0'
        option vlan '1'
        option ports '0 1 2 3 5'

config switch_vlan
        option device 'switch0'
        option vlan '2'
        option ports '4 6'

With this setup the device is almost instantly reachable again when moved from the external switch to the wan port (like in test 1). To verify I ran tcpdump again on the Omnia and again I could not see icmp traffic when connecting through the external switch, but could see it when connected to the wan port. So here too the Omnia’s switch does actually do the switching and it does not push traffic through the CPU. Ping times also confirm this, when connected through the switch I get an average of 0.866ms, when connected to the wan port it’s 1.157ms (over 50 pings).

So there it is, whatever is causing this behavior is triggered in software, it’s not something that’s just impossible because of the hardware. It also is a pretty serious regression from Turris OS 3. @ErikCarlseen does have a point if he says Turris should put some effort into fixing this…

2 Likes

Here be dragons…

While I have everything setup I figured I might as well test the very latest Turris OS and flashed omnia-medkit-202004100102. Sadly no luck there, the issue is also still present in Turris OS 5…

Now while it’s something learned, I still have no idea what actually makes the difference here. However, my first guess would be it’s something to do with how the switch chip is initialized/configured from software. More digging will be needed…

TOS 4 and 5 utilize the same kernel, thus same DSA / SWITCHDEV code


TOS 3 utilises 4.4.x kernel (4.14.x in TOS 4 | 5) with fundamental differences:

  • different switch management concept
  • different driver model for the switch’s downstream ports
  • earnest development of the SWITCHDEV code started only with kernel 4.7.x and since then has progressed (more contemporary kernel versions like 5.4.x | 5.6.x)

From what I gather this regression (TOS 3 -> 4) pertains to a single use case:

  • moving a client node from one of the router switch’s downstream ports to a router CPU upstream port (WLan or WAN)
  • upon such move the client node is then unreachable (from any of router switch’s downstream ports) somewhere in the period of ~ 300 seconds
  • after the ~ 300 seconds the client becomes reachable again (from any of router switch’s downstream ports)

Since apparently this is a viable use case it does not seem something common however, considering:

  • desktop clients are commonly stationary in their network connectivity (not moving between the routers LAN and WLan | WAN ports frequently)
  • IoT clients are similar to desktop clients
  • portable phones | tablets commonly not connecting via LAN (commonly not hardware equipped) and potentially only roam between WLan APs
  • laptop clients might be moved between router switch’s downstream ports to a router CPU upstream port (WLan or WAN) but it seems rather uncommon
  • all above are personal observations only without any factual statistics to corroborate

In the overall scheme it thus would appear to be a minor use case (still viable though) that may not have crossed the minds of code developers, or being high on their agenda.


I do not see that the user has made any specific point other than some blanket statement (without any corroboration) that whatever things are broken.

Nor, do I see that TOS or OpenWrt developers are the ones that should fix this use case (though they may to elect to look into it). Perhaps the issue should be tested with 5.4.x | 5.6.x kernel first and if persistent then be reported to the Linux kernel development for their perusal and assessment of whether considered being a bug and how to potentially resolve it.


Not sure how you reached that guess, on my node:

  • the switch chip gets initialized (libphy) without any issue
  • the switch performs according to its specification (default clearing learned MAC entries from its database after ~ 300 seconds from the time it last appeared as an Ingress frame’s source address)
  • the switch chip not learning on CPU upstream ports (by design)

Frankly, I’m getting a bit tired of this downplaying of the issue. Having multiple wifi AP’s in a network is not some strange exotic edge case. And with multiple AP’s devices will roam between them, they might even do so without moving an inch if signal conditions change.
And the simple fact is that Turris OS 4 broke this functionality.

You also keep repeating that, but it’s not very helpful is it? Any constructive suggestions on how to make an external AP connected to a Omnia work properly?

Frankly, I might as well give up and just downgrade to Turris OS 3 again, because this isn’t getting anywhere useful.

1 Like

Merely pointed out the switch functionality for consideration but that appears to be inadequate then.


If there is no satisfactory input from forum users there are other potential avenues to pursue:

  • see whether the TOS developers pick up on this thread
  • lodge an issue at the TOS code repository
  • seek support directly from the TOS developers via their support channels
  • utilize the OpenWrt forum with a potentially broader user base

I wonder if it’s hardware design or software. Are there any way to change it and make all the ports equal?

It is possible by:

  • monitoring kernel’s MAC address database (bridge fdb)
  • taking appropriate action with the switch’s MAC address database (Address Translation Unit (ATU)) upon changes in bridge fdb

Either DB supports specific user commands to access and modify the contents of the respective MAC address database.

So if hostapd would clean the switch’s MAC address database on new client connection, that would help me. But it sounds like a terrible hack.

Other potential approaches could be:

  • disable ageing on the switch’s ATU
  • disable router’s onboard AP and replace it with an external AP
  • create a wireless mesh between the router’s onboard AP and the external AP
  • leverage the external AP as wireless repeaters to the router’s onboard AP

What would it do? No MAC cache and all the packages will go to all eth ports?

That means even more devices to manage. If I’d go that way I’d replace omnia completely.

I have a gigabit ethernet between Omnia and secondary AP. Why would I ever considering using wireless instead?

Same here.

Hmm, well it makes it worse actually (my bad did not look into the switch documentation first…):

When the AgeTime is set to 0x0 the Aging function is disabled, and all learned addresses will remain in the database forever.

Turning the ATU’s Address Learning off instead would seem neither desirable as it would result in flooding (sending frames out to) all the ports of the switch that have link up with PortState not “Disabled”, except for the port the frame came in on.


There is something ominous

The use of the Marvell header enables a CPU to define dynamically to which port or ports a frame is allowed to go.

Switch’s Ingress Header (Port 4 and Port 5 only)
The detailed information regarding this feature requires an NDA with Marvell® Semiconductor. Please contact your local Marvell Sales Representative for more information.

I guess NDA isn’t for me. I can leave with +~1ms of latency for now.

Alternatively change the ATU’s AgeTime control register default value from 0x13 (19 x 16 = 304 seconds) to the lowest possible value 0x1 (1 x 16 = 16 seconds), if that is more agreeable.

Well, Turris OS 3 did not manifest this behavior, so at least it should be possible to get it to work as desired. I don’t think there are any binary blobs involved, so it should also be possible to find how it does this.

Now I don’t know how it works in Turris OS 3 (yet), but I only see two options. Either the switch can do this in hardware but somehow that is disabled, which would mean the switch needs to be initialized differently. Or in the old scenario the bridge driver updated mac entries in the switch upon learning a new address itself.

What I do know is that the software bridge is aware of the mac entries in the switch. The switchdev spec states:

And this does indeed happen, when having connection issues I can see the bridge actually has two entries for the same mac address:

root@turris:~# bridge fdb | grep 52:1a
b8:27:eb:1b:52:1a dev eth2 master br-lanip
b8:27:eb:1b:52:1a dev lan3 vlan 1 self

While the spec states the switch should notify the bridge about learned mac addresses, it doesn’t seem to specify anything about the bridge notifying the switch about learned addresses. But I guess that might well be what is happening in Turris OS 3, as the software bridge could clearly has the information to detect this and could act upon it.

I’d like to test this with a more recent kernel as there is the possibility this has been fixed already. However, it will probably take quite some time to get everything setup to create an image with a newer kernel and I’m rather short om time. Perhaps the faster way would be to install Debian (see this and this) and test with 5.4 from backports (and/or compile mainline), but either way it will take some time.

If anyone here has a newer kernel running already and is willing to test this that would be great. (Or if there is a really easy way to get a newer kernel…)

2 Likes

That is likely because either VLAN tagging was turned on by default in TOS3.x or a lot users turned it on and therefore routed Lan traffic through the CPU anyway

and that is what

is replicating.


A a better solution of course would be if DSA or bridge driver would communicate with the switch’s ATU with updates of MAC address on upstream ports.

Also seeing this issue after upgrading my TO from TOS 3 to TOS 5. Compouding the issue is that my DNS server is a RPi connected via ethernet, so that means that when my devices roam back to Turris Omnia, they can’t resolve domain names any more.

SSH’ing into the Omnia and deleting the stale fdb entry with bridge fdb del 00:00:de:ad:be:ef dev lan4 vlan 1 self fixes the connectivity issue. I’m going to try the VLAN workaround mentioned before. I used to have VLAN tagging configured with TOS 3 to support primary and guest networks on both APs, didn’t re-replicate it on TOS 5 because it wasn’t clear how to do port-based VLAN tagging. Maybe figuring out how to replicate that previous setup would also resolve the issue?

According to a recent update on the issue tracker https://gitlab.nic.cz/turris/turris-build/-/issues/165#note_161006 the problem seems to be known kernel issue.


That is what @yorik mentioned solved (as in workaround) the issue on her/his node.

After about 4 hours of messing around with interface settings and bridge vlan commands, I give up for now. I first tried the bridge commands listed in this thread: Lan ports aren't accessible for WiFi client for the first 250-330 seconds after connection, but that seems to put each lan port on its own VLAN id, and afterwards nothing could talk to each other.

Then I tried to get the guest network and VLAN tagging working. I want to configure one of my LAN ports to be a trunk to my other AP, sending my main LAN on vlan 1, and my guest LAN on vlan 2, both tagged. This was how I had TOS 3 configured, and it seemed pretty straightforward at the time. Now on TOS 5 I can’t figure out what combination of interface names and bridge settings would make this work. It also seems that on TOS 4/5 the guest interface (GUEST_TURRIS) is now also on vlan 1, the same as LAN? I’m guessing the only reason this isolates guest traffic is because they are not bridged together anywhere. If I want lan0 to have the (tagged) traffic from both networks, then I must have to bridge them at some level. Also, just changing the interfaces to use ‘lan0.1’ and ‘lan0.2’ didn’t seem to enable tagging. The only way I could get tagging to remotely work was when I had the LAN bridge set to use ‘lan0’ (not 'lan0.1) and ran ‘bridge add dev lan0 vid 1 pvid’, but then I couldn’t figure out how to get br-guest_turris to bridge lan0 on VLAN id 2. Has anyone been able to get this to work?