MOX wired throughput issues

ssdnvv · December 30, 2021, 10:30am

Hi,
now that all devices in my new setup are up and running I need to do some finetuning on the MOX devices.
A couple of measurements (all on VID 1; I use 3 VID’s in total, but main traffic is on VID 1):

All devices are wiredly (via Cat5e) connected to one of two Netgear GS110EMX (interconnected to each other via Cat 6a), connections are reported on Netgear GS110EMX to be 1GBE/10GBE. I also ran tests laptop<->TO while having the laptop connected through the cabling of the MOX’ to ensure there is no cabling issue behind this.
Turris devices have all turned off firewall, dnsmasq and odhcp.
TO.2 (192.168.1.2): AP Turris Omnia TOS v.6, connected via eth2 to Netgear1
MOX.5 (192.168.1.5): AP Turris MOX PAB(WLE1216v5-20) TOS 5.3.3, connected via eth0 to Netgear1
MOX.7 (192.168.1.7): AP Turris MOX A(SDIO)B(WLE1216v5-20) TOS 5.3.3, connected via eth0 to Netgear2
Laptop.77 (192.168.1.77): laptop Windows 10, connected via dock (Intel I210) to Netgear2

Iperf tests run on the TO:

root@Turris:~# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:  128 KByte (default)
------------------------------------------------------------
[  4] local TO.2 port 5001 connected with Laptop.77 port 49690
------------------------------------------------------------
Client connecting to Laptop.77, TCP port 5001
TCP window size:  131 KByte (default)
------------------------------------------------------------
[  6] local TO.2 port 57166 connected with Laptop.77 port 5001
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec   895 MBytes   749 Mbits/sec
[  6]  0.0-10.0 sec  1.04 GBytes   896 Mbits/sec
[  4] local 192.168.1.2 port 5001 connected with 192.168.1.7 port 35026
------------------------------------------------------------
Client connecting to MOX.7, TCP port 5001
TCP window size:  109 KByte (default)
------------------------------------------------------------
[  6] local TO.2 port 36688 connected with MOX.7 port 5001
[  4]  0.0-10.2 sec   418 MBytes   344 Mbits/sec
[  6]  0.0-10.8 sec   185 MBytes   144 Mbits/sec
[  4] local 192.168.1.2 port 5001 connected with 192.168.1.5 port 45472
------------------------------------------------------------
Client connecting to MOX.5, TCP port 5001
TCP window size: 43.8 KByte (default)
------------------------------------------------------------
[  6] local TO.2 port 51820 connected with 192.168.1.5 port 5001
[ ID] Interval       Transfer     Bandwidth
[  6]  0.0-10.0 sec   512 MBytes   429 Mbits/sec
[  4]  0.0-10.0 sec   783 MBytes   655 Mbits/sec

So it is clear that the connection TO-Netgear1-Netgear2-laptop is not not perfect, but acceptable in my eyes (at least for now).
The connections TO-Netgear1-MOX PAB and TO-Netgear1-Netgear2-MOX AB is really poor.
This is also reflected by ping-times between the aforementioned devices:

root@Turris:~# ping MOX.5
PING MOX.5 (MOX.5) 56(84) bytes of data.
64 bytes from MOX.5: icmp_seq=1 ttl=64 time=8.41 ms
64 bytes from MOX.5: icmp_seq=2 ttl=64 time=0.383 ms
64 bytes from MOX.5: icmp_seq=3 ttl=64 time=0.366 ms
64 bytes from MOX.5: icmp_seq=4 ttl=64 time=0.595 ms
64 bytes from MOX.5: icmp_seq=5 ttl=64 time=3.46 ms
64 bytes from MOX.5: icmp_seq=6 ttl=64 time=4.41 ms
64 bytes from MOX.5: icmp_seq=7 ttl=64 time=8.06 ms
64 bytes from MOX.5: icmp_seq=8 ttl=64 time=6.60 ms
64 bytes from MOX.5: icmp_seq=9 ttl=64 time=10.9 ms
64 bytes from MOX.5: icmp_seq=10 ttl=64 time=0.937 ms
64 bytes from MOX.5: icmp_seq=11 ttl=64 time=0.617 ms
64 bytes from MOX.5: icmp_seq=12 ttl=64 time=1.05 ms
64 bytes from MOX.5: icmp_seq=13 ttl=64 time=3.85 ms
64 bytes from MOX.5: icmp_seq=14 ttl=64 time=1.45 ms
^C
--- 192.168.1.5 ping statistics ---
14 packets transmitted, 14 received, 0% packet loss, time 13147ms
rtt min/avg/max/mdev = 0.366/3.649/10.922/3.414 ms
root@Turris:~# ping 192.168.1.7
PING 192.168.1.7 (192.168.1.7) 56(84) bytes of data.
64 bytes from MOX.7: icmp_seq=1 ttl=64 time=25.6 ms
64 bytes from MOX.7: icmp_seq=2 ttl=64 time=10.8 ms
64 bytes from MOX.7: icmp_seq=3 ttl=64 time=3.29 ms
64 bytes from MOX.7: icmp_seq=4 ttl=64 time=4.54 ms
64 bytes from MOX.7: icmp_seq=5 ttl=64 time=10.7 ms
64 bytes from MOX.7: icmp_seq=6 ttl=64 time=0.380 ms
64 bytes from MOX.7: icmp_seq=7 ttl=64 time=0.668 ms
64 bytes from MOX.7: icmp_seq=8 ttl=64 time=0.829 ms
64 bytes from MOX.7: icmp_seq=9 ttl=64 time=1.88 ms
64 bytes from MOX.7: icmp_seq=10 ttl=64 time=2.24 ms
64 bytes from MOX.7: icmp_seq=11 ttl=64 time=0.416 ms
64 bytes from MOX.7: icmp_seq=12 ttl=64 time=0.574 ms
64 bytes from MOX.7: icmp_seq=13 ttl=64 time=3.28 ms
^C
--- 192.168.1.7 ping statistics ---
13 packets transmitted, 13 received, 0% packet loss, time 12197ms
rtt min/avg/max/mdev = 0.380/5.009/25.604/6.858 ms
root@Turris:~# ping Laptop.77
PING Laptop.77 (Laptop.77) 56(84) bytes of data.
64 bytes from Laptop.77: icmp_seq=1 ttl=128 time=0.488 ms
64 bytes from Laptop.77: icmp_seq=2 ttl=128 time=0.477 ms
64 bytes from Laptop.77: icmp_seq=3 ttl=128 time=0.286 ms
64 bytes from Laptop.77: icmp_seq=4 ttl=128 time=0.444 ms
64 bytes from Laptop.77: icmp_seq=5 ttl=128 time=0.769 ms
64 bytes from Laptop.77: icmp_seq=6 ttl=128 time=0.575 ms
64 bytes from Laptop.77: icmp_seq=7 ttl=128 time=0.349 ms
64 bytes from Laptop.77: icmp_seq=8 ttl=128 time=0.267 ms
64 bytes from Laptop.77: icmp_seq=9 ttl=128 time=0.238 ms
64 bytes from Laptop.77: icmp_seq=10 ttl=128 time=0.611 ms
64 bytes from Laptop.77: icmp_seq=11 ttl=128 time=0.529 ms
64 bytes from Laptop.77: icmp_seq=12 ttl=128 time=0.447 ms
64 bytes from Laptop.77: icmp_seq=13 ttl=128 time=0.525 ms
64 bytes from Laptop.77: icmp_seq=14 ttl=128 time=0.365 ms
^C
--- Laptop ping statistics ---
14 packets transmitted, 14 received, 0% packet loss, time 13546ms
rtt min/avg/max/mdev = 0.238/0.455/0.769/0.141 ms

So the big question is: what can I do to improve wired connection speeds to/from the MOX? I know it doesn’t saturate the wired connection with NAT on, but the results above are nowhere near saturation. Disabling various foris services didn’t help either.

Thanks,
ssdnvv

edit:
Simple picture (ending are last part of IPv4-adresses (see above)):
TO.2 <1GBE>Netgear.4<10GBE>Netgear.3<1GBE>MOX.7
MOX.5<1GBE>Netgear.4<10GBE>Netgear.3<1GBE>Laptop.77

CPU-load is below 0.75% when running iperf on the MOX. Which tool should I use instead?
What I forgot: the speeds above are full duplex. Single throughput varies between 700Mbit (for MOX.5 and MOX.7) and 930 Mbit (for TO.2).
I do not even use it as a router but only as a access point…

tac2 · December 30, 2021, 5:20pm

It would have helped if you draw a simple picture of your net and perhaps edit the iperf output with names instead of ipaddresses.

Maybe I’’m misstaken but it looks like you are running iperf on the router(s)
I believe it would be better to run it on edge nodes when measure throughput.

My experience with mox ad simple router conf is that it will push near linespeed 1G (nat but no vlans)

EDIT:
i.e. Do not run iperf on mox as you may hit other resource limits of the mox board

ssdnvv · January 5, 2022, 12:21am

Added the additional information in op.

Edit: Thinking about it the term “Near line speed” would be in my opinion >=100 Mbit average, which is only reached via single connections.

tac2 · January 5, 2022, 1:41pm

Ok, i think I understand and will try to explain my thoughts (I hope)

Simpel explanation:
Iperf_server <> mox <> iperf_node

In you case perhaps something like this:
Iperf_server <> Omnia <> switch <> mox <> iperf_node

I.e. do not running iperf (or any other load tool) on devices you want to test for throughput as they will use resources which affect the result on the devices, especially on ”low” powered boards.

I understand that it is somewhat of a challange when you want to test an AP as you probably don’t have two wire connectors but I would suggest either connect via wireless (not optimal, but useful for testingwifi throughput) or getting a C module (or D module with 1G rj45 sfp)

I is probably possibel se if the mox running iperf is affected by some resource starvation, but I am unsure if it will be super obvious.

If I find time I will set up a test on my mox to se if I get similar results, with iperf on edgenode test and with iperf directly on mox

EDIT:
Another thougt, do you use VLAN in your network, and if so are you doing any form of VLAN routing, and if so which devices are doing the routing?

EDIT2:
*nah, vlan routing when using above addresses would be humm… non likely …

ssdnvv · January 6, 2022, 2:11am

Yes, there are VLANs defined to differentiate three different subnets. So this decreases the throughput by 200 MBit/s compared to the TO? Would be very kind if you could do a test without VLANs activated and iperf running directly on the MOX and connection via MOX A (not C or D) to have something equal to compare with.
I am unfortunately right now unable to this edgenode tests you descript, as I do not have C or D modules (but I understand the intention and that MOX has really not that much ressources).

tac2 · January 6, 2022, 9:36am

I’ll make a try later today, just need to find a slott between childrens gaming sessions. We do not want them yelling on me due to incresed latency

tac2 · January 6, 2022, 7:16pm

First I want to share my initial confustion as my first run resulted in more or less stable 800, but then remembered SQM
So just dubbel check you have disabled SQM…

Here is a couple of runs in different setups (not quite full line speed, but allmost,
If I recall correct 940:ish sould be max when taking headers and other? overhead in to account.

=====================================================

#MOX_iperf-client <> laptop1_iperf-server

MOX:~# iperf3 -c XXX.XXX.XXX.XXX -t 15
Connecting to host laptop1, port 5201
[ 5] local XXX.XXX.XXX.XXX port 40302 connected to XXX.XXX.XXX.XXX port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 113 MBytes 947 Mbits/sec 0 543 KBytes
[ 5] 1.00-2.01 sec 111 MBytes 921 Mbits/sec 0 645 KBytes
[ 5] 2.01-3.00 sec 111 MBytes 945 Mbits/sec 0 744 KBytes
[ 5] 3.00-4.00 sec 111 MBytes 930 Mbits/sec 0 744 KBytes
[ 5] 4.00-5.01 sec 111 MBytes 925 Mbits/sec 0 744 KBytes
[ 5] 5.01-6.00 sec 111 MBytes 946 Mbits/sec 0 894 KBytes
[ 5] 6.00-7.01 sec 110 MBytes 916 Mbits/sec 0 1.06 MBytes
[ 5] 7.01-8.00 sec 111 MBytes 941 Mbits/sec 0 1.16 MBytes
[ 5] 8.00-9.00 sec 112 MBytes 942 Mbits/sec 0 1.22 MBytes
[ 5] 9.00-10.00 sec 111 MBytes 933 Mbits/sec 0 1.32 MBytes
[ 5] 10.00-11.00 sec 111 MBytes 935 Mbits/sec 0 1.39 MBytes
[ 5] 11.00-12.00 sec 111 MBytes 931 Mbits/sec 0 1.46 MBytes
[ 5] 12.00-13.00 sec 112 MBytes 944 Mbits/sec 0 1.46 MBytes
[ 5] 13.00-14.00 sec 111 MBytes 935 Mbits/sec 0 1.46 MBytes
[ 5] 14.00-15.02 sec 111 MBytes 920 Mbits/sec 0 1.90 MBytes

[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-15.02 sec 1.63 GBytes 934 Mbits/sec 0 sender
[ 5] 0.00-15.06 sec 1.63 GBytes 931 Mbits/sec receiver

iperf Done.

=====================================================

#laptop1_iperf-client <> MOX_iperf-server

laptop1:~$ iperf3 -c XXX.XXX.XXX.XXX -t 15
Connecting to host MOX, port 5201
[ 5] local XXX.XXX.XXX.XXX port 44312 connected to XXX.XXX.XXX.XXX port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 113 MBytes 952 Mbits/sec 0 605 KBytes
[ 5] 1.00-2.00 sec 111 MBytes 933 Mbits/sec 8 542 KBytes
[ 5] 2.00-3.00 sec 111 MBytes 933 Mbits/sec 78 413 KBytes
[ 5] 3.00-4.00 sec 111 MBytes 933 Mbits/sec 62 368 KBytes
[ 5] 4.00-5.00 sec 109 MBytes 912 Mbits/sec 177 362 KBytes
[ 5] 5.00-6.00 sec 110 MBytes 923 Mbits/sec 99 342 KBytes
[ 5] 6.00-7.00 sec 111 MBytes 933 Mbits/sec 0 380 KBytes
[ 5] 7.00-8.00 sec 110 MBytes 923 Mbits/sec 10 366 KBytes
[ 5] 8.00-9.00 sec 110 MBytes 923 Mbits/sec 129 363 KBytes
[ 5] 9.00-10.00 sec 109 MBytes 912 Mbits/sec 127 318 KBytes
[ 5] 10.00-11.00 sec 111 MBytes 933 Mbits/sec 64 307 KBytes
[ 5] 11.00-12.00 sec 111 MBytes 933 Mbits/sec 0 392 KBytes
[ 5] 12.00-13.00 sec 108 MBytes 902 Mbits/sec 127 380 KBytes
[ 5] 13.00-14.00 sec 111 MBytes 933 Mbits/sec 0 409 KBytes
[ 5] 14.00-15.00 sec 109 MBytes 912 Mbits/sec 200 297 KBytes

[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-15.00 sec 1.62 GBytes 926 Mbits/sec 1081 sender
[ 5] 0.00-15.00 sec 1.61 GBytes 924 Mbits/sec receiver

iperf Done.

===================================================

#laptop1_iperf-client <> MOX <> switch <> laptop2-server

laptop1:~$ iperf3 -c XXX.XXX.XXX.XXX -t 15
Connecting to host laptop2, port 5201
[ 5] local XXX.XXX.XXX.XXX port 48364 connected to XXX.XXX.XXX.XXX port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 109 MBytes 915 Mbits/sec 144 345 KBytes
[ 5] 1.00-2.00 sec 110 MBytes 924 Mbits/sec 32 304 KBytes
[ 5] 2.00-3.00 sec 111 MBytes 927 Mbits/sec 62 256 KBytes
[ 5] 3.00-4.00 sec 111 MBytes 935 Mbits/sec 15 465 KBytes
[ 5] 4.00-5.00 sec 109 MBytes 917 Mbits/sec 165 233 KBytes
[ 5] 5.00-6.00 sec 110 MBytes 923 Mbits/sec 0 447 KBytes
[ 5] 6.00-7.00 sec 108 MBytes 910 Mbits/sec 168 294 KBytes
[ 5] 7.00-8.00 sec 110 MBytes 923 Mbits/sec 63 307 KBytes
[ 5] 8.00-9.00 sec 111 MBytes 927 Mbits/sec 104 331 KBytes
[ 5] 9.00-10.00 sec 106 MBytes 887 Mbits/sec 146 303 KBytes
[ 5] 10.00-11.00 sec 108 MBytes 907 Mbits/sec 100 250 KBytes
[ 5] 11.00-12.00 sec 108 MBytes 910 Mbits/sec 32 334 KBytes
[ 5] 12.00-13.00 sec 106 MBytes 887 Mbits/sec 85 252 KBytes
[ 5] 13.00-14.00 sec 110 MBytes 922 Mbits/sec 17 308 KBytes
[ 5] 14.00-15.00 sec 111 MBytes 930 Mbits/sec 25 331 KBytes

[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-15.00 sec 1.60 GBytes 916 Mbits/sec 1158 sender
[ 5] 0.00-15.00 sec 1.60 GBytes 915 Mbits/sec receiver

======================================================

And as laptop2 is a shitty 8 year old laptop you should probably not pay to much attention on the numbers on the last test…

ssdnvv · January 6, 2022, 7:42pm

You’re making fun of me
Thanks alot. So as there is no other software running on none of MOX’ (firewall, odhcp and dnsmasq are disabled), I can assume that VLAN kills the speed. That is really sad, but as I cannot go without VLANs (and don’t want to pay the difference in price between MOX an TO), I have to live with that.

It only once more marks MOX’ limitations

tac2 · January 6, 2022, 7:56pm

Haha, sorry, it was not my intention

There have been discussions here about vlan and dsa, when tos 4 was new, can not remember all details but there is something about kernel? vs switch chip? conf and potential related performance hits
But I may remember things wrong…

EDIT:
And just to be sure, tripple check your cables

ssdnvv · January 6, 2022, 8:42pm

DSA is not invoked/involved here, it has only an impact when you use a switch chip (like MOX C or D or TO - btw for TO it is broken by now because of the ARP bug), for a single ethernet socket every package has to be processed via CPU, so it is a “software VLAN”.

ssdnvv · January 6, 2022, 8:44pm

Just joking

I’ll do so, thanks for suggesting!

tac2 · January 7, 2022, 1:30pm

Hurm, it’s hard just to let these kind of things go…

I would like to understand what exactly limits the throughput.

Can you perhaps do a longer iperf run and monitor the % and load per cpu and se how hard irq:s is distributed per cpu ?

Is there anyone having other means ideas to really pinpoint the bottleneck here?

Or does the mox just have to low cpu frequency to handle the job?

I would have played with my mox to see if I could find any clues but right now the family depend on it and I do not want to switch to my spare huawei router if not absolutly nessesary……

ssdnvv · January 7, 2022, 3:17pm

It seems that laptop and/or network intermediates (cabling/switches?) are the culprits.
I directly connected my MOX to my router (Hyper-V vm) and am presented these unleashed powers:

root@MOX:~# iperf -c X86-64 Router VM
------------------------------------------------------------
Client connecting to X86-64 Router VM, TCP port 5001
TCP window size:  246 KByte (default)
------------------------------------------------------------
[  3] local MOX port 57238 connected with X86-64 Router VM port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.09 GBytes   935 Mbits/sec
root@MOX:~# iperf -c X86-64 Router VM -d
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:  224 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to X86-64 Router VM, TCP port 5001
TCP window size:  366 KByte (default)
------------------------------------------------------------
[  5] local MOX port 57240 connected with X86-64 Router VM port 5001
[  4] local MOX port 5001 connected with X86-64 Router VM port 54846
[ ID] Interval       Transfer     Bandwidth
[  5]  0.0-10.0 sec  1.03 GBytes   887 Mbits/sec
[  4]  0.0-10.1 sec   713 MBytes   595 Mbits/sec

root@X86-64 Router VM:~# iperf -c MOX 
------------------------------------------------------------
Client connecting to MOX , TCP port 5001
TCP window size:  230 KByte (default)
------------------------------------------------------------
[  3] local X86-64 Router VM port 54848 connected with MOX port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.07 GBytes   916 Mbits/sec

root@X86-64 Router VM:~# iperf -c MOX -d
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:  128 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to MOX, TCP port 5001
TCP window size:  221 KByte (default)
------------------------------------------------------------
[  5] local X86-64 Router VM port 54850 connected with MOX port 5001
[  4] local X86-64 Router VM port 5001 connected with MOX port 57242
[ ID] Interval       Transfer     Bandwidth
[  5]  0.0-10.0 sec   992 MBytes   831 Mbits/sec
[  4]  0.0-10.0 sec   828 MBytes   693 Mbits/sec

So the MOX is by no means slow in terms of connection speeds, CPU climbs up to a “maximum” of 0.5% usage (measuring traffic within one VLAN).
I think I need to kill my current Windows installation on the laptop with a kind format c: after five years of usage (upgrade from Win7 to Win10 in the meantime) to get back a more powerfull device (I also have problems in terms of Wi-Fi connection speeds btw). Allthough I do not understand why connecting laptop<->MOX and laptop<->TO makes a difference…

tac2 · January 7, 2022, 5:32pm

Puuhh, nice!

Really good you did not let it go, wouldn’t like the idea of us starting to believe the MOX wasn’t upp to the task.

tac2 · January 7, 2022, 8:57pm

I’ve had my share of network cable/ connectors/switch ”half” failures so I’m not surprised if the same cable/connector works on one device but not fully on another.
Or a switch port for that matter.

ssdnvv · January 7, 2022, 9:57pm

Wouldn’t have believed that beforehand, but right now it’s obvious to me.
Seems I need to have a look for bad cabling

system · January 10, 2022, 9:58pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.