OMNIA: Vlan on DSA port breaks arp responses (TOS 4.0.5)

AVee · March 29, 2020, 8:15pm

Hi,

I’ve spend the last few hours trying to get to the bottom of some network issues, and it seems that for some reason arp replies are not properly returned when send over a vlan interface on one of the switch ports. The same config works fine when using the WAN port.

This is the config I used to test this:

config interface 'lan'
        option type 'bridge'
        option proto 'static'
        option netmask '255.255.255.0'
        option ipaddr '10.0.0.15'
        option gateway '10.0.0.14'
        option dns '194.109.6.66'
        option ifname 'eth2.1 lan2 lan3.1 lan4'

I’ve got an upstream (from another Turris Omnia) which has tagged traffic on vlan 1 and 2. If I plug that into the WAN port (eth2) everything is fine. However if I plug the cable into lan3 I never receive arp responses. I can see the request reaching the target host and the response being send there. Arp requests from the Turris Omnia itself do work correctly. So it seems for some reason arp responses never cross the bridge on a tagged interface.
The connection is otherwise fine, as long as I don’t clear the arp cache there are no connection issues with systems for which the mac-address is already known.

I can’t really see what is causing this, but based on what I find searching it seems to me it has something to do with the DSA based configuration of the switch. Maybe it even is some kernel bug.

Are there any known issues which might cause this?

Now this is just the test case, for the real setup I can’t use the WAN port. So I’d like to find some way to work around this. For me the nicest solution would be if I could map one of the CPU ports directly onto one of the LAN ports, as I guess that would work around this issue. Is it possible to change the switch config in a way somewhat similar to how that was done in TOS 3.x?

And if anyone sees another workaround I’d be happy to hear that too

anon50890781 · March 30, 2020, 9:23am

Might be what is known as ARP Flux.

In general yes but currently there is no userland supplied to achieve it (perhaps though it could be manipulated through GPIO). As of TOS => 4.x the ports are mapped via device tree ^[1].

Switch’s CPU facing ports 5 & 6 are not exposed by DSA.

^[1] Files · hbd · Turris / Turris OS / Turris Build · GitLab

AVee · March 30, 2020, 10:49am

Might be what is known as ARP Flux.

I initially saw the problem testing towards a host with just a single interface, so at least in that case ARP Flux could not have been a factor. It might have been in other tests I have done, so perhaps it did affect my conclusions. I’ll try and figure out if that comes into play

In general yes but currently there is no userland supplied to achieve it (perhaps though it could be manipulated through GPIO). As of TOS => 4.x the ports are mapped via device tree [1] .

Switch’s CPU facing ports 5 & 6 are not exposed by DSA.

Here’s my reasoning:

Use the devicetree to map 1 cpu port directly towards one lan port.
Use that soc ethernet port set up the interface, basically creating a situation equal to using the WAN port.

Now if I’m reading the patch correctly that is what happens already, LAN 4 is mapped to CPU port 6 which is connected to eth0. What isn’t entirely clear to me is if this is already included in 4.0.5…

anon50890781 · March 30, 2020, 11:27am

Not sure whether it factors in but switch chip features its own MAC address database which the kernel is unaware of. Moreover, the switch ship features its own VLANTable registers that work in conjunction with aforementioned MAC address database, supposedly some packets may not be reaching the kernel but remain within the switch fabric (purpose of the switch).

Now, however with lan3.1 there is a virtual (software) interface controlled by the kernel but not the switch (DSA) and that may (or may not) lead to a ARP Flux situation.

As far as I understand the better practice to manage VLAN tagging for DSA ports is via the bridge v command, e.g. as mentioned here

AVee · March 30, 2020, 11:57am

Not sure whether it factors in but switch chip features its own MAC address database which the kernel is unaware of. Moreover, the switch ship features its own VLANTable registers that work in conjunction with aforementioned MAC address database, supposedly some packets may not be reaching the kernel but remain within the switch fabric (purpose of the switch).

I think you’re right about that. Not sure if it’s ARP Flux or something else, but it sure looks like virtual tagged interfaces on DSA ports are somehow a bad idea.

My initial idea of tagging on eth0 didnt work either, at least not in a way which can be configured through Luci.

As far as I understand the better practice to manage VLAN tagging for DSA ports is via the bridge v command

Looks like that’s the next thing to try, it does seem to be the correct tool for the job. That does put things outside of UCI which is a kinda annoying. But I’ll see if I can get it figured out.

AVee · March 30, 2020, 3:55pm

Ok, looks like I’ve got something working, at least in a test.

It took a bit for the penny to drop, but what the bridge command does is manage vlans which exist within the bridge. What I had been doing before is setting up a bridge for each separate network. To get those networks transported over a single cable I would just bridge over tagged interfaces (like eth2.1). All traffic within the bridge would just be normal untagged traffic. Now I need a single bridge containing multiple vlans.

So here’s what I did:

Enable vlan filtering on the bridge (seems off by default)

echo 1 > /sys/class/net/br-lan/bridge/vlan_filtering

Set lan4 to accept tagged traffic on vlan 1 & 2

bridge v a dev lan4 vid 1 tagged
bridge v a dev lan4 vid 2 tagged

Set a port to vlan 2 untagged

bridge v a dev lan0 vid 2 pvid
bridge v d dev lan0 vid 1 # To remove the default filter

The default filter on the other ports is to be on vlan 1 untagged, so they are already where they need to be. That setup seems to work without any issues (so far at least).

Now because it is all one bridge now there is just one interface (br-lan) which has an ip address. The Turris Omnia itself is now not a member of vlan 2. To fix this I added vlan 2 to the br-lan interface:

bridge v a dev br-lan vid 1 tagged self
bridge v a dev br-lan vid 2 tagged self

Once this was done I could setup an ip on br-lan.2 (well, i could before, but now it got traffic as well). I kinda expected this to break the br-lan interface, because there is no untagged traffic anymore. But that didn’t happen, the br-lan interface seems to be part of VLAN 1 without the need to use br-lan.1.
After reboot that didn’t work anymore, so a added a br-lan.1 interface anyway.

This is the end result:

# bridge v
port    vlan ids
lan0     2 PVID Egress Untagged

lan1     1 PVID Egress Untagged

lan2     1 PVID Egress Untagged

lan3     1 PVID Egress Untagged

lan4     1
         2

wlan1    1 PVID Egress Untagged

wlan0    1 PVID Egress Untagged

br-lan   1
         2

So far it’s just a test. I’ll set it up on the main router later (at a time I don’t piss everybody off when the internet drops out )

If I would want to make stuff like this permanent, should I just drop it in rc.local, of is there a better approach?

anon50890781 · March 30, 2020, 7:10pm

Probably a good bet. Only caveat might be the timing of when the script runs and the availability of the bridge interface during the boot process.

Theoretically it should not even be necessary to create software bridges (which in the end is just a virtual network switch) since the switch chip already provides a hardware bridge and tagging its ports should suffice. Exception might be with the WLan interfaces but it might work as well since the bridge v command apparently works there as well.

It would require though some changes in the DHCP configuration.

AVee · March 30, 2020, 9:35pm

Probably a good bet. Only caveat might be the timing of when the script runs and the availability of the bridge interface during the boot process.

It looks like that worked for both routers, so I guess rc.local won’t run before the bridge is up. I’ll leave it at that.

Theoretically it should not even be necessary to create software bridges (which in the end is just a virtual network switch) since the switch chip already provides a hardware bridge and tagging its ports should suffice. Exception might be with the WLan interfaces but it might work as well since the bridge v command apparently works there as well.

My impression was that the point of DSA is to basically do that where possible. A tcpdump on the bridge device seems to confirm that, all it sees is traffic actually directed to the local ip.

AVee · April 1, 2020, 12:43pm

It seemed ok at first, but there’s still weirdness happening.

Firstly, this happend:

# tcpdump -e -vvv -i eth0 arp or icmp
...
    10.0.0.98 > 10.0.0.5: ICMP echo request, id 1, seq 361, length 40
21:19:17.220499 00:50:b6:b4:c4:0f (oui Unknown) > a0:b3:cc:e3:ff:aa (oui Unknown), ethertype 802.1Q (0x8100), length 64: vlan 1, p 0, ethertype ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.0.0.5 (a0:b3:cc:e3:ff:aa (oui Unknown)) tell 10.0.0.98, length 46
21:19:18.220942 00:50:b6:b4:c4:0f (oui Unknown) > a0:b3:cc:e3:ff:aa (oui Unknown), ethertype 802.1Q (0x8100), length 64: vlan 1, p 0, ethertype ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.0.0.5 (a0:b3:cc:e3:ff:aa (oui Unknown)) tell 10.0.0.98, length 46
21:19:18.234475 00:50:b6:b4:c4:0f (oui Unknown) > a0:b3:cc:e3:ff:aa (oui Unknown), ethertype 802.1Q (0x8100), length 78: vlan 1, p 0, ethertype IPv4, (tos 0x0, ttl 128, id 5179, offset 0, flags [none], proto ICMP (1), length 60)
    10.0.0.98 > 10.0.0.5: ICMP echo request, id 1, seq 362, length 40
21:19:19.221091 00:50:b6:b4:c4:0f (oui Unknown) > a0:b3:cc:e3:ff:aa (oui Unknown), ethertype 802.1Q (0x8100), length 64: vlan 1, p 0, ethertype ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.0.0.5 (a0:b3:cc:e3:ff:aa (oui Unknown)) tell 10.0.0.98, length 46
21:19:40.238631 d8:58:d7:00:74:52 (oui Unknown) > a0:b3:cc:e3:ff:aa (oui Unknown), ethertype ARP (0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Request who-has 10.0.0.5 tell 10.0.0.14, length 46

That’s taken from a system connected to one of the lan ports in the bridge which is set to Vlan 1 PVID Untagged. But it shows vlan tagged traffic arriving there anyway.

After a reboot this was no longer happening. It seems that things can get corrupted when making changes to the switch configuration. So I’m now always rebooting when changing thing, that way at least I’m sure I’m not hunting ghosts.

The other thing was that on the main router I had two bridges which both contained lan ports. (Also one with Vlan filtering and one without.) Things got better once I lost that second bridge, it seems that having multiple bridges confuses the switch chip or DSA driver.

Wired networking now seems fine.

Wireless was still messy though. Devices would generally loose connectivity when changing AP. My theory is that this happens because a switch or bridge somewhere doesn’t realize a certain mac address moved elsewhere. I’ve enabled STP and dropped max_age to 1s on all the bridges and at first glance that seemed to improve things a bit, but not totally resolve it. It looks like I might see the behavior described here: Lan ports aren't accessible for WiFi client for the first 250-330 seconds after connection

I’ll try to dig a bit deeper into the Wlan roaming later. For now I’ve spend too much time on it already anyway

yorik · April 1, 2020, 1:20pm

Since I implemented that hack with vlans, which pushes all the traffic through CPU, I had no issues at all with WiFi roaming. So I’d recommend it.

AVee · April 2, 2020, 2:29pm

I haven’t had time to look into it much deeper. At first glance the theory there seems to make sense, although I’m kinda surprised to learn the switch will ignore upstream ports when learning addresses. Also I’m not sure it completely adds up with the behavior I’m seeing, because I think I mostly had issues moving away from the main router. In that case new traffic would be arriving over the downstream ports and which surely should cause the switch chip to learn the new location. But as I said, I haven’t really dug into it…

I’ve got two other access points connected to the main router, so for now I’ve just disabled the radio’s on the main router. That seems to have resolved the issue. For now this will have to do, although I would like to get it working at some point.

One thing that does interest me is how you are bridging the wlan. I started of with lan ports and wlan in the same (unmanaged) bridge, but that seemed to cause issues (but perhaps those issues where caused by something else). So now I have a bridge over the lan ports (br-lan) and a bridge over br-lan.1 and the wlan interfaces. This could be different from your setup and therefore might trigger different behavior…

yorik · April 2, 2020, 4:16pm

I have single bridge over all lan and wlan interfaces (default setting). I want to have single network to allow WiFi roaming between integrated wlan and AP connected via lan to my Omnia.

ErikCarlseen · April 7, 2020, 5:25pm

I haven’t checked in a few months, but after TOS4 was released the TOS software wasn’t correctly initializing the switch on the Omnia hardware - I figured this out after a very, very long trip down a similar rabbit hole. Specifically, TOS did not pass a description of the specific switch hardware to the routines being used by the underlying OpenWRT hardware init scripts, and so the “generic switch” settings were being used. This caused all sorts of issues when trying to do anything VLAN-wise that wasn’t exposed through the TOS GUI (and I was told that stuff on the LuCi interface was “unsupported”, even though this “unsupportedness” wasn’t documented anywhere). As far as I could tell, there was no attempt by Turris made to correct this during or after development and when several people brought it up their response was essentially “screw it, the TOS interface stuff works, so we don’t care.” Even though a lot of stuff that should have been bridged in hardware was actually being bridged by the CPU.

I reached the conclusion that Turris doesn’t give a crap about implementing things correctly (this would take very little effort to fix - you just need the technical documentation for the switch chip being used), and as long as their GUI appears to be functional that’s all that matters to them. This makes them more or less the same as every garbage-level SOHO router vendor out there and I’ve pretty much given up on them. Your mileage may vary.

AVee · April 8, 2020, 9:00am

It sure is a rabbit hole…

I guess you are referring to this topic: Only 4-bit (0-15) VLANs allowed with TurrisOS 4.0.1 on Turris Omnia - #12 by protree - SW bugs discussion - Turris forum? I’ll give that a proper read late, but if there are other details to be found elsewhere I’d appreciate any pointers.

I was already under the impression that one way or another the switch isn’t (always) handled correctly. It seems this is all related to the move to DSA. I get the impression it’s not just Turris, but also OpenWRT which still needs to get a good handle on this. So some of it probably should be fixed in OpenWRT or maybe even in the DSA driver for this switch.
That’s not to say that Turris shouldn’t take an active role in fixing those things and pushing that upstream. I bought these to have a properly supported OpenWRT device, personally I couldn’t care less about Forris…

Now I still haven’t dug deeper, but what I said earlier isn’t true:

I’ve got two other access points connected to the main router, so for now I’ve just disabled the radio’s on the main router. That seems to have resolved the issue.

I tested this using an Android phone which does mac-address randomization, and it seems that the mac-address changes also when switching AP in the same network. So that phone was fine. Other devices which don’t randomize their mac-address still run into issues.

anon50890781 · April 8, 2020, 10:57am

DSA (driver) does not require a specific init script as being part of the Linux kernel.

re/Foris is (currently) not designed to manage VLAN tagging of the switch’s downstream ports, that functionality always been deferred to L/UCI.

LuCI is coded by OpenWrt and the lack to manage VLAN tagging of the switch’s downstream ports (DSA driven) is documented there (since several months) ^[2]

some effort has been made by CZ.NIC developer ^[3] but:

not been accepted upstream (albeit appear also not been rejected either)
further development seems have stalled
CZ.NIC developer busy coping with the MOX issues

The Multi-CPU-SWITCH (DSA) chip/board design may factor into the matter:

it would seem that the design been a novelty concept that the Linux kernel was/is not catering for
chip designer (Marvell) does not appear to be contributing to Linux on the matter
kernel 4.14.x being not contemporary and later releases may improve DSA performance (patches may not been backported to 4.14)
OpenWrt’s switchconf driver been rejected by Linux development
TO been the first device in the OpenWrt target tree to deploy DSA (guinea pig)
OpenWrt developers currently struggling with this design in particular (other device targets with similar design suffering the same issue) and appear inexperienced with DSA in general

^[2] luci-mod-network: add support for DSA switch configuration ( 802.1q management) · Issue #2798 · openwrt/luci · GitHub
^[3] https://www.spinics.net/lists/netdev/msg595295.html

ErikCarlseen · April 8, 2020, 2:37pm

I actually dug into this down to the source code / detailed init script analysis level. You have literally no idea what you’re talking about. OpenWRT 18.x moved to a new framework for switch hardware initialization and management, based on a chipset description file(s) that are intended to be provided by the router vendor based on whatever’s physically in the unit. It’s a reasonably elegant solution that covers situations with multiple, dissimilar switching chips, but it does add a bit of complexity. OpenWRT comes with a “default switch” description file that … doesn’t completely break when used with the Omnia router. As mentioned, stuff is being bridged by the CPU that should be handled in hardware but that doesn’t break very basic functionality; it just wastes CPU cycles and adds latency. Anything beyond very basic functionality is completely broken.

anon50890781 · April 8, 2020, 3:18pm

Care to humour the uninitiated?

Which is, aside from DSA (driver) and switchdev (stateless framework) that are provided by the kernel?
Kernel provided libphy probes different buses and reads/writes device specific registers.

As far as I am aware DSA only be deployed with the TO in OpenWrt 18.6.x whilst all other devices with switches deploying swconfig which been rejected by Linux kernel development.

Are you referring to Device Tree? If that is the case TOS provides one tailored to the TO but that has not been uplifted to OpenWrt (far as I know)

I do not see evidence that traffic (data path) between switch’s downstream ports reaches the CPU (escapes the switch’s fabric). Frames between WLan and the switch have to pass through the CPU due to the design of the board and the switch.

What is broken?

AVee · April 11, 2020, 2:20pm

Any pointers to which file that would be? I haven’t got much OpenWRT experience, so I wouldn’t know where to start looking…

Neither do I, but…

Several things, firstly I had a similar setup before using Turris OS 3 on two Omnia’s and a separate third router als AP. Never noticed any issues roaming with that. So I’m very inclined to say there is a serious regression there. (It looks to be the switch, but theoretically a bug in the software bridges might cause similar issues).

Secondly, there is a capture a few posts back which shows both tagged and untagged traffic getting out of a port which was configured to be untagged. That’s clearly faulty behaviour…

Before upgrading to Turris OS 4 I had a similar setup, the main router had Wifi and a separate router (Fritz!Box) was connected running as a dumb AP. I never noticed any issues when roaming from that AP to the Omnia or the other way around. From the perspective of the Omnia that’s a client moving from a Lan port to a Wlan port. Exactly the thing described here:

Now this last statement bothers me. I really don’t see why a switch or bridge would not learn mac’s from an ‘upstream’ port. If that holds true pretty shouldn’t pretty much any router that has an internal switch as well as Wlan have these issues? I’d expect the ‘uptream’ port on a switch chip to be just another port, perhaps supporting extra control features or a higher datarate but not fundamentally different.
But most of all, I really don’t recall seeing this sort of thing when still running Turris OS 3, which suggests to me it’s not some fundamental limitation of the hardware.

anon50890781 · April 11, 2020, 3:38pm

The Linux kernel manipulates switches with three different operation structures: switchdev_ops, ethtool_ops, and netdev_ops. ^[4]

There are two data bases at work that may not play nicely with each other (bug?):

switch address database (ageing of 300 seconds)
bridge fdb ^[5]

That the one you then stated

?

The switch just switches packets among its downstream ports but does not route any packet. Anything not destined for a learned downstream port gets switched upstream to the CPU for kernel processing (routing); no need for the switch to learn about upstream ports (upstream ports are defined via the Device Tree).

In all likelihood it may be the case

unless the swtich chip is equipped with a WLan port, which the chip in TO is not (no physical connection/lane) and thus

Evidently the bridge fdb is aware of the WLan ports but the switch address database is not.

DSA / SWITCHDEV code is still evolving at kernel source development and may indeed exhibit some erroneous behaviour in some earlier kernel versions. Mind

^[4] The switchdev driver model [LWN.net]
^[5] https://github.com/torvalds/linux/blob/master/Documentation/networking/switchdev.txt

AVee · April 11, 2020, 5:55pm

Yes, I got that capture, rebooted with (I think) the same config and only saw untagged traffic again.

Well, except to be able to invalidate any entries for addresses which are now behind the CPU port

That’s clear to me, and it seems highly likely to be related to something in that stuff.

Now I dug up a Vlan capable switch and rewired some things, allowing me to take the second Omnia out of the network without issues (other then loosing some wifi coverage). So I could do some testing…

It seems to me that what applies to Wifi should also apply to the wan port, as that is connected to the CPU as well. So if I where to create a bridge over the lan ports and the wan port I should be able to observe the same behavior, right?
I could test that with both Turris OS 4 and with 3 and see if there’s a difference…