Performance degradation with pakon or device discovery

Hi

I have a Turris Omnia :
Turris Omnia - rtrom01
Turris OS version 3.11.2
Kernel version 4.4.169-7bc33afbb1b35f5830b2b1b42c9cd8a0-2

There is a significant performance degradation when i install pakon or device discovery (so everything that is somehow connected to suricata).

The config on the omnia is very simple…it’s basically a NAT box…

So without pakon and/or device detection i get a throughput of cca 940Mb/s (TCP L4) that is the max that the interface can handle.

But with pakon enabled or device discovery I get speeds around 600Mb/s (100Mb/s more or less it fluctuates a lot).

is this normal ? I know that suricata is cpu intensive…but cca 30% is quite a lot

1 Like

Can reproduce this as well, have logged this bug

1 Like

THX ! @tonyquan,
I’m really curious if it can be fixed or is the CPU just to weak to handle suricata at wirespeed…

Soooooo…any news about this ?

Any improvement here ?

This is not a defect but the property. Do you realistically use 940Mb/s? Or it si throughput ? What do you usually download and upload?

Yeah…it maybe is a “property” but since I couldn’t find anywhere a description of this “property” and it practically reduces max throughput that is achievable in NAT by 40%, I consider it a problem !

If there would be some kind of statement or comment somwhere that by enabling suricata this happens, then it would be a “property”, and a Known fact and I wouldn’t consider it as a problem, but like I said I didn’t find any documentation on this (at that point when the firs post was written)

As for our throughput…yeah at peak hours our throuhput through the turris is cca 800 Mb/s, which is not achievable with pakon turned on .

Filtration of all data streams and its analysis must have some overhead of CPU performance. It’s not free (pay free :-)). If you are basping not the speed of the Pakon you uninstall

Similarly, FTP and SFTP do not have the same transfer speed. You also want to solve this ?

Hi JardaB

thanks for feedback, yes I have also read about experimental device detection model can cause this, and indeed i have also installed it, however, I found it is a nice feature, that I didnt want to miss, so in the end I solved the problem by configuring the pakon custom version of suricata so that I whitelisted NFS traffic and installed device detection modul again.

Dude…did you read the whole thread ? Post nr.3 ? I just wanted to know if it is something you can fix/ tweak or if the CPU is to weak to handle suricata at wirespeed…

And what has FTP and SFTP to do with it ??? There are also a ton of other factors that make the speed difference between those two protocols besides the CPU (if that was your point)…

So in the end…what you wanted to say is that the CPU in the turris is too weak to handle suricata at wire speed…is that correct ?

there are numerous stong indications for it.

With which other routers you compare CPU and mem performance ?

It should be fairly simple to monitor CPU usage during that stress-testing. I expect faster CPU would just solve it, but (1) it’s better to confirm CPU is really loaded, and (2) there might be other ways around the limitation.

By me max 20% CPU “normal operation” (peak 30%) (Surikaca 2 proceses together max 6% )

And via iperf to LAN about 30%

root@Omnia:~# iperf -c 192.168.2.1 -m -i 5 -t 30 -r
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.2.1, TCP port 5001
TCP window size: 2.50 MByte (default)
------------------------------------------------------------
[  5] local 192.168.2.1 port 52496 connected with 192.168.2.1 port 5001
[  4] local 192.168.2.1 port 5001 connected with 192.168.2.1 port 52496
[ ID] Interval       Transfer     Bandwidth
[  5]  0.0- 5.0 sec   504 MBytes   845 Mbits/sec
[  4]  0.0- 5.0 sec   504 MBytes   845 Mbits/sec
[  5]  5.0-10.0 sec   514 MBytes   863 Mbits/sec
[  4]  5.0-10.0 sec   514 MBytes   863 Mbits/sec
[  5] 10.0-15.0 sec   514 MBytes   863 Mbits/sec
[  4] 10.0-15.0 sec   514 MBytes   863 Mbits/sec
[  5] 15.0-20.0 sec   511 MBytes   858 Mbits/sec
[  4] 15.0-20.0 sec   511 MBytes   858 Mbits/sec
[  5] 20.0-25.0 sec   517 MBytes   867 Mbits/sec
[  4] 20.0-25.0 sec   517 MBytes   868 Mbits/sec
[  5] 25.0-30.0 sec   516 MBytes   866 Mbits/sec
[  5]  0.0-30.0 sec  3.01 GBytes   860 Mbits/sec
[  5] MSS size 65483 bytes (MTU 65523 bytes, unknown interface)
[  4] 25.0-30.0 sec   516 MBytes   866 Mbits/sec
[  4]  0.0-30.0 sec  3.01 GBytes   860 Mbits/sec
[  4] MSS size 21888 bytes (MTU 21928 bytes, unknown interface)
root@Omnia:~#

1 Like

Like this thread there are various others of the same theme (and new threads keep popping up) implying that Suricata | Pakon inducing (severe) performance penalties on the NIC.CZ hardware:

  • substantial bandwidth throughput degradation (upto ~ 60% in some reported cases)
  • SQLite DB filling /tmp storage space (reducing amount of available RAM)

The suitability/mating of Suricata | Pakon with NIC.CZ hardware seems questionable since IDS (the likes of Suricata | Snort) requires serious CPU power to compensate for timely packet processing (DPI).

In the upstream forum some other DPI engine with a different approach (analysis in the cloud) is advertised. The caveat might be the cloud-based service (privacy of data) and paid subscription (cost).

2 Likes

Well, my somewhat boring take on that is, it depends :wink: what you expect.
See, I am totally amazed that my omnia can do traffic shaping bi-directionally up to 500/500 Mbps, but I really do not expect it to do anything but the absolute minimum in addition to that (NAT, PPPoE, firewall, wifi, next to no additional services). It would probably help, if on installation PAKON would sufficiently warn the user about the expected CPU cost (heck, dump an info line to that tune onto the log and the notifications once per week).
But that reminds me that I want to play with PAKON (with a 100/40 vdsl link my onmia will have enough cycles left for traffic shaping, lucky me that my access link is slow enough :wink: )

1 Like

Reading the various related threads it looks like that inherent penalties take users (repeatedly) by surprise, kind of unawareness to the correlation of the CPU (capabilities) vs. high bandwidth throughput (reaching/exceeding 1 GbE connectivity) vs. DPI (vs. any additional apps/services that require CPU cycles).

It is also not clear to which extent Suricata on its own is contributing to the performance degradation (probably the lion share) vs. what PAKON might contribute on its own.

Another unknown is the performance (potential improvements) of Suricata v5.0.x vs. the legacy v4.0.x that is provided by TOS.

1 Like

I actually came here to mention netifyd. If the capture engine is better optimized for openwrt class hardware (sounds like it might be) this is probably a better thing to build PaKon on top of. Suricata looks like it’s hitting some speedbumps in OpenWRT with recent versions anyways since it’s starting to use rust.

There is nothing inherent to it’s design that requires you use another persons computer (the cloud). The analysis could easily be done in the router box or on a separate designated machine. I just checked and it looks like the package is there if you opkg update && opkg find netifyd

I would love run some benchmarks to test differences but I’ve been spending a lot of cycles on learning networking and OpenWRT lately and I probably need to shift focus soon.

My experience ....

*I was using Pakon and DeviceDetection for some time without any issues. Later i add Ludus. For some time it was still fine, no issues. Later suricata was updated and splitted into suricata and suricata-bin , after that i face some performance issues. I’ve tried to sort it out (changing the .yaml configuration), did not helped. After some try-fail scenarios i actually removed all suricata related packages. Tried to install them all from scratch. Did not help with performance(as expected). Later i uninstall all again(via Foris/Luci) make cleanup and install only Ludus from scratch. So far so good. *

*note: that suricata package split cause my TOS to actually have suricata,suricata-bin, suricata-monitor, suricata-pakon, ludus, pakon folders present in /etc/ (and it confused me a lot what is obsolete and what is actual, what i can remove/uninstall). Due fact that some suricata package is part of Foris updater package list, removing was working only partially (later i found i have to remove some manually, some via luci and finally uncheck related list in Foris and wait for pkgupdate process to hit in. After that i have to check that list again , run pkgupdate again. once done i removed obsolete folders/files and installed Ludus via luci. *
Now TOS seems to be running normally. (yet ludus still take quite a lot of mem/cpu, but at least there are no swapping and cpu peaks)