Sigh, false alarm, my own fault. I was using this method to measure the throughput:
On the Omnia:
nc -l -p 2222
On my PC:
dd if=/dev/zero bs=1024K count=512 | nc -v 192.168.1.1 2222
The issue was uncovered when I changed the source to
/dev/random … which then spewed out garbage on the Omnia. The bottleneck was the rendering of received zero chars on the SSH session (doh!). When I reverted to this on the Omnia:
nc -l -p 2222 > /dev/null
We’re back to the ~90MB/s I was getting before:
$ dd if=/dev/zero bs=1024K count=512 | nc -v 192.168.1.3 2222
Ncat: Version 7.80 ( https://nmap.org/ncat )
Ncat: Connected to 192.168.1.3:2222.
512+0 records in
512+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 5.7366 s, 93.6 MB/s
Ncat: 536870912 bytes sent, 0 bytes received in 5.75 seconds.
For those interested I still get around 90MB/s end to end through this config of Omnias:
PC <- Cat5e -> Omnia1 <- SM Fibre -> Omnia2 <- Cat5e -> Omnia3
With Omnia1 to Omnia2 having to pass through the CPU bound bridge.