The thing is that while Omnia does have two gigabit links from the switch chip to the CPU, the DSA architecture is broken and there is currently no way (I know of) to get those two links aggregated. If this has been fixed already, I welcome any and all contributions to how to properly configure the link aggregation.
But even if we had that, we couldn’t achieve more than a gigabit per second (or the numbers @moeller0 mentioned), as a single connection would only be switched across one of the two switch–CPU links and not both.
The only possibility how to route more than a gigabit via Omnia is to get the 2.5 Gbps metallic SFP and hook it up to a multi-gigabit switch. So that’s what I did for the purpose of this test (and because I also have a multi-gig WAN connection with PPPoE, this seems to fit well the OP’s question).
The set-up
- Omnia (the original edition with Wi-Fi and 1 GB RAM), TurrisOS 6.2.0 (I should probably upgrade this testing beast someday)
- WAN, PPPoE with VLAN tagging (VLAN 848), multi-gigabit port on the ONT, service provisioned at ~2100/1100 Mbps, capable of achieving ~2025/1080 Mbps in speedtests; ONT connected to the 2.5 Gbps port of my multi-gig switch at home
- the 2.5 Gbps module connected to Omnia’s SFP cage; SFP cage configured as
eth2; connected to the home switch, using trunk port configured for VLAN 848 and VLAN 9 (test LAN) - a home server connected using 10 Gbps (SFP+, fibre) to the home switch; VLAN 9 was used for this test, no other device besides this server and Omnia was on that network
- a multi-gig switch (Juniper EX2300-24MP)
Without SQM
When I run the speedtest binary on the Omnia against a speedtest that’s apparently using BBR, this is what I get:
root@turris-omnia:~# ./speedtest -s 30620
Speedtest by Ookla
Server: O2 Czechia, a.s. - Prague (id: 30620)
ISP:
Idle Latency: 8.41 ms (jitter: 0.03ms, low: 8.35ms, high: 8.46ms)
Download: 1538.16 Mbps (data used: 837.5 MB)
32.87 ms (jitter: 34.96ms, low: 7.88ms, high: 466.39ms)
Upload: 1084.60 Mbps (data used: 932.0 MB)
19.70 ms (jitter: 5.57ms, low: 8.59ms, high: 263.95ms)
Packet Loss: 0.0%
when I run the same from the home server, the speeds are even a bit better:
root@home-server:~# speedtest --ip=10.9.8.2 -s 30620 2>/dev/null
Speedtest by Ookla
Server: O2 Czechia, a.s. - Prague (id: 30620)
ISP:
Idle Latency: 8.69 ms (jitter: 0.04ms, low: 8.63ms, high: 8.73ms)
Download: 1749.59 Mbps (data used: 2.0 GB)
130.88 ms (jitter: 57.34ms, low: 7.31ms, high: 954.17ms)
Upload: 1081.51 Mbps (data used: 968.8 MB)
14.73 ms (jitter: 0.87ms, low: 8.27ms, high: 25.19ms)
With SQM/QoS (Cake)
I have attached Cake to the PPPoE interface and enabled piece_of_cake with the default settings and bandwidth limits set to 2000Mbit down and 1050Mbit up, slightly below the maximum. The rest of cake’s settings were kept in default.
root@turris-omnia:~# tc qdisc show dev ifb4pppoe-wan
qdisc cake 800a: root refcnt 2 bandwidth 2Gbit besteffort triple-isolate nonat wash no-ack-filter split-gso rtt 100ms raw overhead 0
root@turris-omnia:~# tc qdisc show dev pppoe-wan
qdisc cake 8009: root refcnt 2 bandwidth 1050Mbit besteffort triple-isolate nonat nowash no-ack-filter split-gso rtt 100ms raw overhead 0
qdisc ingress ffff: parent ffff:fff1 ----------------
speedtest ran from the router itself:
root@turris-omnia:~# ./speedtest -s 30620
Speedtest by Ookla
Server: O2 Czechia, a.s. - Prague (id: 30620)
ISP:
Idle Latency: 8.46 ms (jitter: 0.03ms, low: 8.38ms, high: 8.50ms)
Download: 716.04 Mbps (data used: 599.8 MB)
8.44 ms (jitter: 0.44ms, low: 7.81ms, high: 13.65ms)
Upload: 912.13 Mbps (data used: 587.4 MB)
76.58 ms (jitter: 39.25ms, low: 8.11ms, high: 1110.45ms)
and from the server:
root@home-server:~# speedtest --ip=10.9.8.2 -s 30620 2>/dev/null
Speedtest by Ookla
Server: O2 Czechia, a.s. - Prague (id: 30620)
ISP:
Idle Latency: 8.66 ms (jitter: 0.16ms, low: 8.42ms, high: 8.86ms)
Download: 727.20 Mbps (data used: 799.2 MB)
9.03 ms (jitter: 0.46ms, low: 8.09ms, high: 13.08ms)
Upload: 979.94 Mbps (data used: 515.7 MB)
11.22 ms (jitter: 1.70ms, low: 8.15ms, high: 20.36ms)
In the downlink direction, you can’t even max out gigabit with Cake. In the uplink direction, you can, and it seems to be quite okay-ish. One of the cores gets loaded up to 100 %, the other one is used at around 50 %.
Summary
Considering how old Omnia is and what did I have to do to get this all working, the result is exceeding my expectations. It is certainly capable of switching and routing multi-gig, although not using the integrated LAN switch (you’ll be throttled at 1 Gbps). It does not max out the 2.5 Gbps SFP and because I use it to transfer data between the WAN and LAN VLANs, the total combined throughput [up+down] will never exceed 2.5 Gbps.
Please note that any extra process running on the Omnia will significantly degrade the routing capabilities at those speeds (and with PPPoE), so forget about containers, Wi-Fi controllers, NAS, etc. There’s not enough power for that. Also, servers using Cubic instead of BBR might be more sensitive to any latency that might be added by the routing processes under full load.
(This one WAN connection does not support IPv6 and I will certainly add IPv6 results, as these might get a bit better thanks to missing NAT.)
P.S.: If you have considered using the Omnia SFP module with Routerboard RB5009, forget about it NOW. The Omnia SFP module does not properly report the 2.5 Gbps speed to that router (reports 10 Gbps instead of 2.5 Gbps) and your buffers will be heavily bloated, resulting in severely degraded connection. Trust me, I’ve been there, and eventually ordered a proper Mikrotik 1/2/5/10 Gbps SFP+ module which works well.