Using the SFP+ for >1Gbps LAN

Omnia hardware Omnia HW tweaks
#1

Unfortunately we don’t have FttH here - and I as I work in a technical position at my local cable operator and have a very satisfactory internet connection, the advantages of fiber wouldn’t outweigh the benefits of being able to do my own support in any event.

So what to do with that SFP+ port?

The obvious application is >1Gbps LAN connection. Not that my NAS maxes out my current 1GbE in any way, but it will let me get the highest iperf throughput on the block :wink:

I emailed Turris a few months back about expected throughput and received this reply:

Yes, the SFP data differential pairs are connected to the SoC to a SERDES
interface. The SERDES is utilized by the Ethernet L2 element and Armada 385
datasheet says that it supports up to 2.5 Gbps transfer speed over SGMII
http://www.marvell.com/embedded-processors/armada-38x/ . I haven’t seen any 2.5
Gbps Ethernet SFP module so far and I do not know about the 2.5 GbE support in
mvneta driver in Linux kernel. Maybe there are some patches lying in mailing
lists but our kernel does not seem to support that so far.

We plan to investigate this option in the next generation of Turris Omnia,
because there are metalic PHY chips that support 2.5 GbE and that can be
utilized instead of our current 1 GbE PHY. When we have the support for 2.5 GbE
in kernel, then retro-fitting it for SFP cage is possible, if we have proper
SFP modules. But now we can not make any guarantees. However since Turris Omnia
software is OSS, you are always free to research, hack and conntribute support
for the 2.5 Gbps modules on your own.

So, >1Gbps throughput should be possible, but software support for 2.5GbE isn’t there yet. 2.5GBASE-T is by far the most practical way of getting higher speeds in terms of cabling, but this isn’t a practical exercise - and would require a lot of software work (or waiting for someone else to do it). Moreover 2.5GBASE-T SFP+ modules are rather expensive at the moment, which is a bit much for something purely for fun.

What is available cheaply is optical networking. LC connectors can easily be pulled through residential 5/8" conduits, multimode cable is cheap as are 10GBASE-SR SFPs.

That means the following should be feasible:

  • 10GBASE-SR SFP in the Omnia
  • 7m of LC to LC 10G OM3 multimode fiber
  • 10GBASE-SR SFP in another device, probably a NIC in my home server/NAS - a friend said he might have one to spare for me.

I’ve ordered the above parts from www.fs.com for EUR 39 incl shipping to EU (note: FS is a great and affordable supplier of optical equipment with excellent customer service, but be careful when ordering - very expensive express shipping is chosen by default. Make sure to select normal shipping unless you’re in a huge hurry or you end up paying EUR 25 too much…)

Now, when the SFP is used the WAN Ethernet port becomes inactive - but another port can be assigned to function as WAN instead, as explained here:

At this point in time I don’t have a specific question, but intend to keep you updated on this little experiment. Comments are welcome however :slight_smile:

2 Likes
Omnia networking - Use SFP for LAN, and lan4 for WAN?
#2

Yes the SoC supports this and some hints on getting it working are hidden in marvell’s github repositories and wiki.

I am glad i am not the only one wanting to experiment with this.

1 Like
#3

Hi,
if you mean the DSA support… I’m also thinking about how to get it working.

I guess it is necessary two have two things, the kernel modules and a modified device tree.
So I have already managed to get the kernel modules compile and made an attempt to modify the device tree… but unfortunately my Omnia isn’t shipped yet… so I didn’t test it and have no idea if anything I did is right.

If someone wants to test this, I would be happy to provide the files…

#4

DSA is another point and the third entry on my plan, SFP and 2.5G are later on. First will probably be wifi testing and second will be getting ArchLinux on my Omnia. Nothing is done as i seem to be one of the last ones to get mine shipped. But i got at least the final invoice so it may arrive in the next few days. :smiley:

DSA is different from the SFP 2.5G speed. DSA will make the switch more or less disappear but still work. I am still not sure how this will work out with the second cpu port.

#5

Yeah, DSA and 2.5G are different things.

As I haven’t found any example, I’m not sure about two CPU ports with DSA as well.
But support for multiple CPU ports should be in the kernel. I’m just not sure how they are used and if I made the changes in the device tree correct for this.

And for 2.5G, maybe a small change in the device tree would be a good first try to get it working:
https://gitlab.labs.nic.cz/turris/openwrt/blob/stable-omnia/target/linux/mvebu/files/arch/arm/boot/dts/armada-385-turris-omnia.dts
At the ethernet@34000 section, remove the fixed-link (speed 1000) parts.

Final invoice and “The order is being prepared for shipment” for a week now.
Want it now. Have to play with the (device) tree! :wink:

#6

speed should be set to 2500 but i can’t find where it was written.

https://github.com/MarvellEmbeddedProcessors/main/wiki/2.5G-Ethernet-Configuration was the initial hint that it is possible.

#7

i came across this link too.
maybe it is as simple as just add an extra section like phy-sgmii-sfp, name it phy-sgmii-sfp-2500 or so and change the speed option.
i think how to change to this phy type can be looked up in the sfpswitch.py.
someone with a omnia and a 2.5G sfp needs to try it :wink:

#8

@dionb: I suppose your ordered parts arrived by now - did you get the SFP-module working?

#9

did you mange to get this to work

#10

@adminX: Did you procede on this matter? dionb seems not to answer anymore (at least on this topic)…
And I think the SFP±topic is one of the most interesting topics to be discussed (I started the other topic https://forum.turris.cz/t/fast-connection-of-clients-with-turris-omnia-former-bonding-with-turris-omnia/853/13

I’m willing to pay the ammounts necessary to get this working but unfortunately I’m not a compiling-crack, so I would need assistance… :frowning:

#11

@ssdnvv: the amount necessary may be higher than the amount needed for a 10GE network.

It is not as simple as it looks.

The current problem is getting usable hardware.

The Intel X550 series is capable of 2.5 and 5 Gbit/s under Linux only and only available with 8P8C connectors. The connector problem is the same for Tehuti TN4010 + Aquantia PHY or Aquantia AQtion based cards.

For SFP there are 2 options: use another Turris Omnia to try it or use a MACCHIATOBin aka SolidRun ARMADA 8040 Community Board. The latter one uses SFP+ and should be compatible with Aquantia’s SFP+ modules. It should be possible to get the MACCHIATOBin to communicate with the Omnia with some SFP modules or maybe even some direct attach cable.

For the near future i think i will get a VDSL SFP module as it is more useful to me than a higher speed to my PC. Getting a MACCHIATOBin is still an option for me.

I come to think marvell added the 2.5Gbps as a proprietary extension for their switch chips like the 200 Mbit links found in some older router SoC.

Speed wise i come to think of 2.5G and 5G ethernet over copper as a version of 10GBASE-T for older cables and higher cable length. Simply said: lower speed for the same money and power consumption.

#12

Hm… The ammount for a usable 10GB-alternative for my network would be

  • 10GBE-Switch - when available ASUS XG-U2008 for 252€
  • 10GBE-Card for my Server, e.g. Intel X540-T1 for 219€
    (later upgrade, as not affordable yet: 10GBE-TB3-Adapter for one client, the only one yet available seems to be ThunderLink® NS 3101 for 1195$ - other solutions would use a TB2 to TB3-Adapter-Solution like “Thunder2 10G Network Adapter + T3T Legacy Adapter” for 399$)

I think you are right. When I start playing with my server, I will be buying this components. But perhaps till then someone else discovered a usable solution.

#13

@turris team: concerning the information you gave dionb (see post 1) - did you proceed on this matter?

#14

Apologies for very long silence. Due to a small child, an intensive training course at work and some home disasters (sewage flooding…) I have had other priorities than the Omnia for the last weeks.

However, things are improving and I have been able to play around a bit. I have focused on the hardware side so far - getting the fiber equipment up and running. After that I’ll see about swapping WAN and LAN around.

The good news:

  • The Omnia works fine with multiple 1Gbase-SX and at least one 10Gbase-SR SFP(+) modules.
  • Very surprisingly, the 10Gbase-SR SFP will connect with the 1Gbase-SX (at 1Gbps of course)

Working:

  • Cisco GLC-SX-MM 30-1301-02 (1Gbase-SX)
  • Fiberstore SFP-10GSR-85 (10Gbase-SR)
    Note that FS can flash their models with just about any firmware for compatibility purposes. I asked for Ethernet (not Fiber Channel) and Cisco compatibility and it works fine on the Omnia.

The bad news:

  • still no 10Gbase-SR link as the FS SFP refuses to work in the HP/Mellanox ConnectX-2 EN MT26448 PCIe 10Gbit Ethernet NIC in my server. I hadn’t expected this end of the connection to be the challenge when it came to compatibility…

So right now I have 1GbE-SX connected, and am trying to figure out how to get the Ethernet adapter to accept my 10Gb SFP+. Will update as and when I’ve progressed.

#15

I am not that surprised. This can work because these modules only take the bits from the serdes lane and move them out. And 10GBase-SR and 1GBase-SX use the same wavelength.
10GBase-LR(M) connected 1GBase-LX could in theory work too.

8b/10b encoding for 1GBase should be in the network card not the module.

I can’t find a NIC with 2.5GBase SFP capability. Some Broadcom chip had it mentioned in its datasheet but i couldn’t find a card with this chip.
The only other devices that could work and are available under 1000 EUR seem to be the Turris Omnia, SolidRun Clearfog Pro and the MACCHIATObin.

#16

What did you to to make the SFP port on the LAN rather than WAN?

#17

Remove eth2 from the lan bridge and move it to wan. Remove eth1 from wan and add it to lan. This is possible in Luci.

1 Like
#18

As I’m completely new to SPF(+)-networking:
Would a direct cabling (active/passive) work? If not - which single modules/cabling should I take into account for testing?

#19

As my devices aren’t in the same room, I haven’t played around with that yet myself. If distance is short enough, passive Direct Attach (twinax) Cables such as the one you link to should do the trick cheaper and with less compatibility headaches than using SFP+s and optical cables.

I’ve gotten my setup working, but only at 1Gbps, and via a detour, as both my Mellanox ConnectX 2 and an Intel X520 I recently acquired refuse to accept my 10Gbase-SR SFP+s and even with bog-standard Cisco 10Gbase-SX they won’t connect directly to the Omnia, only via a switch in between - which is also limited to 1Gbps (and draws a massive 137W even unloaded - it’s a Cisco C3560X 48 port PoE monster). Still no sign of my new SFP+s programmed for the Mellanox NICs :cry:

#20

I tried the following combination:

  • NIC: Startech PEX10000SFPI
  • Cabling: 2m StarTech.com Cisco compatible SFP+ Twinax (SFP-H10GB-CU2M)
    (thougt that cabling and NIC from same vendor should do the trick)

Unfortunately no connection is established (neither Turris Omnia nor Server). :sweat:
Any suggestions? Are there specific network options that have to be chosen under Windows (10)?