SQM on turris flent benchmarks?

Dave_Taht · April 16, 2022, 8:48pm

I see from various threads on this forum that sqm (sch_cake) has been shown to work at rates of up to 600mbit bidirectional. I hope that the next generation of turris’s work can scale to a full gbit here, bu:t it would be VERY interesting to know a couple things about current scaling problems, as I think further development work is going to be required to the core “cake” code. We tend to use the flent.org tests to get detailed reports on how and where things are going wrong. Web tests are far too weak to drive gbit networks reliabily enough to see what’s going wrong.

Is there anyone(s) out there that can run one or more strings of flent vs SQM test’s on bufferbloat.net’s behalf? flent is commonly available for linux repositories and can be made to work on OSX, and we maintain a fleet of flent servers throughout the globe. Our most current one was built for starlink testing (it has nothing to do with starlink itself), but can be leveraged for this:

What I’m mostly looking for at the moment is “rtt_fair” results at various rates - both at what can actually be achieved via turris, and above where it begins to fail with and without sqm enabled.

I don’t know if turris supports the “flent” package for openwrt or not(?) that has additional tools we can use to collect data more directly, via flent as well.

flent -x --socket-stats --step-size=.05 -H de.starlink.taht.net -H london.starlink.taht.net -H singapore.starlink.taht.net -H fremont.starlink.taht.net -t your_test_options_bandwidth_etc rtt_fair4be

It is also very possible to setup a flent server on your local network, enable sqm
on it, and just test that. It would be great if someones had that also, as the rtt_fair test is also a really good test of the wifi version of fq_codel.

Dave_Taht · April 16, 2022, 4:44pm

This is a SQM’d result over 5G from my boat in california of that test.

This is a non-sqm’d result

Miserable, isn’t it?

PSA: For anyone interested in making their LTE better, run, not walk, over to the ongoing development of “cake-autorate” here: CAKE /w Adaptive Bandwidth - Installing and Using OpenWrt - OpenWrt Forum

But that’s not the sort of testing I’m trying to get, I’m trying to understand better how >500Mbit networks work.

Anyway, other useful data to collect

tc -s qdisc show for the sqm test

                 Tin 0
  thresh          6Mbit
  target          5.0ms
  interval      100.0ms
  pk_delay        1.4ms
  av_delay        275us
  sp_delay          4us
  backlog            0b
  pkts          3233988
  bytes      3246666847
  way_inds        93643
  way_miss        61270
  way_cols            0
  drops            2958
  marks          291201
  ack_drop            0
  sp_flows            1
  bk_flows            2
  un_flows            0
  max_len          1514
  quantum           300

Anyway you can learn a lot more about the behaviors from the data in the *.flent.gz files, plot them a zillion different ways, and (especially) do comparison plots, but there doesn’t seem to be a way to upload them to the forum…

moeller0 · April 16, 2022, 7:21pm

It is called flent-tools and is installable and seems to work (I get cpu stats from the router).
opkg update ; opkg install flent-tools

I can only run internet tests up to ~110/37 Mbps, while local testing indicates that 55/550 with some tweaking should be possible, so unless I get a faster internet access not much I can offer. But please holler for any specific test you want.

Sorry, out of my league right now.

Dave_Taht · April 16, 2022, 7:27pm

Glad to hear turris also has that package. The sampling routine was rewritten in C however, and may not work properly on a modern tc…

I would appreciate you posting a reference to the good results you get at the speed you run at. I actually forget the command line requirements to also capture cpu_stats and qdisc_stats required…

moeller0 · April 16, 2022, 8:32pm

Since rtt_fair is actually using CS0 for 2 hosts and CS1 for the other two hosts and I use layer_cake, I switched to rtt_fair4be to just compare between the different host locations:

./run-flent -x --ipv4 -l 60 -H de.starlink.taht.net -H london.starlink.taht.net -H singapore.starlink.taht.net -H fremont.starlink.taht.net rtt_fair4be --remote-metadata=root@192.168.42.1 --test-parameter=cpu_stats_hosts=root@192.168.42.1 --step-size=.05 --socket-stats -D . -t IPv4_SQM_cake_layer-cake_LLA-ETH_OH34_U097pct36000of36998K-D090pct105000of116797K_work-horse-eth0_2_TurrisOmnia-TurrisOS.5.3.7-pppoe-wan-eth2.7_2_bridged-BTHH5A-OpenWrt-r18441-ba7cee05ed-Hvt-VDSL100_2_netperf-eu.bufferbloat.net --log-file

I note that london.starlink.taht.net seems to limit each flow to around 0.8 Mbps thoughput… at least from my ISP, when I do an 8 flow rrul_var test I can reach ~6 Mbps with 8 flows

text output:

Starting Flent 2.0.1-git-65cd64f using Python 3.8.10.
Starting rtt_fair4be test. Expected run time: 70 seconds.
Data file written to ./rtt_fair4be-2022-04-16T221508.845943.IPv4_SQM_cake_layer-cake_LLA-ETH_OH34_U097pct36000of36998K-D090pct105000of116797K_work-horse-eth0_2_TurrisOmnia-TurrisOS_5_3_7-pppoe-wan-eth2_7_2_bridged-BTHH5A-OpenWrt-r18441-ba7cee05ed-Hvt-VDSL100_2_net.flent.gz

Summary of rtt_fair4be test run from 2022-04-16 20:15:08.845943
  Title: 'IPv4_SQM_cake_layer-cake_LLA-ETH_OH34_U097pct36000of36998K-D090pct105000of116797K_work-horse-eth0_2_TurrisOmnia-TurrisOS.5.3.7-pppoe-wan-eth2.7_2_bridged-BTHH5A-OpenWrt-r18441-ba7cee05ed-Hvt-VDSL100_2_netperf-eu.bufferbloat.net'

                                                                          avg       median       99th %          # data pts
 Ping (ms) ICMP1 de.starlink.taht.net                          :        19.60        19.20        29.40 ms             1596
 Ping (ms) ICMP2 london.starlink.taht.net                      :        30.40        29.90        40.20 ms             1596
 Ping (ms) ICMP3 singapore.starlink.taht.net                   :       174.00       173.00       184.01 ms             1596
 Ping (ms) ICMP4 fremont.starlink.taht.net                     :       181.00       180.00       190.00 ms             1596
 Ping (ms) avg                                                 :       101.25          N/A          N/A ms             1596
 TCP download BE1 de.starlink.taht.net                         :        59.50        60.14        97.09 Mbits/s        1596
 TCP download BE2 london.starlink.taht.net                     :         0.75         0.76         0.79 Mbits/s        1596
 TCP download BE3 singapore.starlink.taht.net                  :         4.03         4.57         7.94 Mbits/s        1596
 TCP download BE4 fremont.starlink.taht.net                    :        28.35        28.68        48.31 Mbits/s        1596
 TCP download avg                                              :        23.16          N/A          N/A Mbits/s        1596
 TCP download fairness                                         :         0.49          N/A          N/A Mbits/s        1596
 TCP download sum                                              :        92.63          N/A          N/A Mbits/s        1596
 TCP upload BE1 de.starlink.taht.net                           :        14.60        14.12        31.26 Mbits/s        1596
 TCP upload BE1 de.starlink.taht.net::tcp_cwnd                 :        34.06        31.00        69.72                 829
 TCP upload BE1 de.starlink.taht.net::tcp_delivery_rate        :        14.09        12.10        28.68                 829
 TCP upload BE1 de.starlink.taht.net::tcp_pacing_rate          :        18.35        15.83        37.16                 829
 TCP upload BE1 de.starlink.taht.net::tcp_rtt                  :        26.10        25.33        36.16                 829
 TCP upload BE1 de.starlink.taht.net::tcp_rtt_var              :         1.38         1.07         4.87                 829
 TCP upload BE2 london.starlink.taht.net                       :         0.47         0.39         0.80 Mbits/s        1596
 TCP upload BE2 london.starlink.taht.net::tcp_cwnd             :         1.28         1.00         2.00                 846
 TCP upload BE2 london.starlink.taht.net::tcp_delivery_rate    :         0.47         0.38         0.77                 845
 TCP upload BE2 london.starlink.taht.net::tcp_pacing_rate      :         0.56         0.45         0.92                 845
 TCP upload BE2 london.starlink.taht.net::tcp_rtt              :        31.27        31.04        35.70                 844
 TCP upload BE2 london.starlink.taht.net::tcp_rtt_var          :         1.12         0.80         6.69                 844
 TCP upload BE3 singapore.starlink.taht.net                    :         7.28         8.22        16.41 Mbits/s        1596
 TCP upload BE3 singapore.starlink.taht.net::tcp_cwnd          :       121.33       126.00       185.12                 845
 TCP upload BE3 singapore.starlink.taht.net::tcp_delivery_rate :         7.27         7.60        10.62                 839
 TCP upload BE3 singapore.starlink.taht.net::tcp_pacing_rate   :         8.85         9.25        13.10                 842
 TCP upload BE3 singapore.starlink.taht.net::tcp_rtt           :       190.13       189.16       210.04                 840
 TCP upload BE3 singapore.starlink.taht.net::tcp_rtt_var       :         1.88         1.09         9.78                 840
 TCP upload BE4 fremont.starlink.taht.net                      :         7.91         9.01        15.05 Mbits/s        1596
 TCP upload BE4 fremont.starlink.taht.net::tcp_cwnd            :       137.39       153.00       222.12                 845
 TCP upload BE4 fremont.starlink.taht.net::tcp_delivery_rate   :         7.92         8.77        12.33                 839
 TCP upload BE4 fremont.starlink.taht.net::tcp_pacing_rate     :         9.60        10.64        15.16                 842
 TCP upload BE4 fremont.starlink.taht.net::tcp_rtt             :       198.64       197.59       212.88                 840
 TCP upload BE4 fremont.starlink.taht.net::tcp_rtt_var         :         2.22         1.04        13.40                 840
 TCP upload avg                                                :         7.57          N/A          N/A Mbits/s        1596
 TCP upload fairness                                           :         0.70          N/A          N/A Mbits/s        1596
 TCP upload sum                                                :        30.26          N/A          N/A Mbits/s        1596
 cpu_stats_root@192.168.42.1::load                             :         0.32         0.33         0.78                1237

RTTs look okay, but throughput is decidedly not independent of RTT, sure London seems rate limited, but singapore is also not behaving as expected… mind you in upload direction things look okayish I think.

This is from an otherwise live network wirth loads of gunk happening (including running PAKON).

Dave_Taht · April 16, 2022, 8:32pm

The convergence on the upload is absolutely marvelous, a goal tcp and aqm designers had for decades.

On the download there’s two problems - adjusting for the bandwidth in the wrong place (after the ISP’s router), and… what I think I found in the mikrotik series of tests was that we were hitting cake’s memory limit a bit early in the case of really long rtts. Since the turris has plenty of memory, quintuple that memlimit for another test?

I’m a bit puzzled by your latency spike (and I wish there was a way to post the flent.gz files here) but y’know, this is so massively better than 5g, and we’re used to it.

Dave_Taht · April 16, 2022, 8:36pm

Is ecn on? london shouldn’t be behaving like that…

moeller0 · April 16, 2022, 8:46pm

It takes a while to get there, but once it does it looks nice…

On it, set memlimit 32mb for a factor of 8 increase… let’s see where that goes.

Mmh, that helped singapore in the download, but now fremont took a hit. Upload looks a bit worse, but still pretty acceptable except for london. I guess singapore and fremont suffer from my default interval 100…

As I said non-quiescent network with too much going on…

Good question I need to check on my Linux host, I believe it should be on…

I agree, but the orange line is also crap^W underwhelming in your examples, sure from your location london is far away, but so is de and still de looks much more believable…

Dave_Taht · April 16, 2022, 9:01pm

I rebooted london and disabled ecn neg there. Also rebooted fremont and disabled ecn. Thx for playing!

moeller0 · April 16, 2022, 9:15pm

Starting Flent 2.0.1-git-65cd64f using Python 3.8.10.
Starting rtt_fair4be test. Expected run time: 70 seconds.
Data file written to ./rtt_fair4be-2022-04-16T230921.394857.IPv4_SQM_cake_layer-cake_LLA-ETH_OH34_U097pct36000of36998K-D090pct105000of116797K_work-horse-eth0_2_TurrisOmnia-TurrisOS_5_3_7-pppoe-wan-eth2_7_2_bridged-BTHH5A-OpenWrt-r18441-ba7cee05ed-Hvt-VDSL100_2_net.flent.gz

Summary of rtt_fair4be test run from 2022-04-16 21:09:21.394857
  Title: 'IPv4_SQM_cake_layer-cake_LLA-ETH_OH34_U097pct36000of36998K-D090pct105000of116797K_work-horse-eth0_2_TurrisOmnia-TurrisOS.5.3.7-pppoe-wan-eth2.7_2_bridged-BTHH5A-OpenWrt-r18441-ba7cee05ed-Hvt-VDSL100_2_netperf-eu.bufferbloat.net'

                                                                          avg       median       99th %          # data pts
 Ping (ms) ICMP1 de.starlink.taht.net                          :        19.50        19.20        24.71 ms             1599
 Ping (ms) ICMP2 london.starlink.taht.net                      :        30.20        29.80        36.30 ms             1599
 Ping (ms) ICMP3 singapore.starlink.taht.net                   :       173.00       173.00       180.00 ms             1599
 Ping (ms) ICMP4 fremont.starlink.taht.net                     :       182.00       181.00       188.00 ms             1599
 Ping (ms) avg                                                 :       101.17          N/A          N/A ms             1599
 TCP download BE1 de.starlink.taht.net                         :        33.38        32.20        50.21 Mbits/s        1599
 TCP download BE2 london.starlink.taht.net                     :        31.42        30.41        46.13 Mbits/s        1599
 TCP download BE3 singapore.starlink.taht.net                  :        20.64        24.16        32.22 Mbits/s        1599
 TCP download BE4 fremont.starlink.taht.net                    :         4.53         4.90         6.76 Mbits/s        1599
 TCP download avg                                              :        22.49          N/A          N/A Mbits/s        1599
 TCP download fairness                                         :         0.79          N/A          N/A Mbits/s        1599
 TCP download sum                                              :        89.97          N/A          N/A Mbits/s        1599
 TCP upload BE1 de.starlink.taht.net                           :         9.60         9.15        16.95 Mbits/s        1599
 TCP upload BE1 de.starlink.taht.net::tcp_cwnd                 :        22.49        21.00        39.67                 834
 TCP upload BE1 de.starlink.taht.net::tcp_delivery_rate        :         9.19         8.55        15.80                 833
 TCP upload BE1 de.starlink.taht.net::tcp_pacing_rate          :        11.88        11.14        20.12                 834
 TCP upload BE1 de.starlink.taht.net::tcp_rtt                  :        26.41        25.97        36.60                 828
 TCP upload BE1 de.starlink.taht.net::tcp_rtt_var              :         1.48         1.28         4.19                 828
 TCP upload BE2 london.starlink.taht.net                       :         9.23         8.89        17.36 Mbits/s        1599
 TCP upload BE2 london.starlink.taht.net::tcp_cwnd             :        29.50        28.00        51.00                 836
 TCP upload BE2 london.starlink.taht.net::tcp_delivery_rate    :         8.91         8.41        14.73                 835
 TCP upload BE2 london.starlink.taht.net::tcp_pacing_rate      :        11.56        10.79        19.33                 835
 TCP upload BE2 london.starlink.taht.net::tcp_rtt              :        36.24        35.45        45.72                 830
 TCP upload BE2 london.starlink.taht.net::tcp_rtt_var          :         1.33         1.12         4.23                 830
 TCP upload BE3 singapore.starlink.taht.net                    :         6.05         6.76         9.12 Mbits/s        1599
 TCP upload BE3 singapore.starlink.taht.net::tcp_cwnd          :        98.50       108.00       141.50                 851
 TCP upload BE3 singapore.starlink.taht.net::tcp_delivery_rate :         6.05         6.66         8.42                 846
 TCP upload BE3 singapore.starlink.taht.net::tcp_pacing_rate   :         7.39         8.08        10.59                 849
 TCP upload BE3 singapore.starlink.taht.net::tcp_rtt           :       185.87       185.18       199.77                 848
 TCP upload BE3 singapore.starlink.taht.net::tcp_rtt_var       :         2.05         1.12        10.84                 848
 TCP upload BE4 fremont.starlink.taht.net                      :         5.88         6.74         9.80 Mbits/s        1599
 TCP upload BE4 fremont.starlink.taht.net::tcp_cwnd            :        97.06       102.00       147.57                 844
 TCP upload BE4 fremont.starlink.taht.net::tcp_delivery_rate   :         5.86         6.19         8.94                 839
 TCP upload BE4 fremont.starlink.taht.net::tcp_pacing_rate     :         7.35         7.66        14.48                 841
 TCP upload BE4 fremont.starlink.taht.net::tcp_rtt             :       189.10       188.34       202.46                 838
 TCP upload BE4 fremont.starlink.taht.net::tcp_rtt_var         :         2.09         1.22        13.03                 838
 TCP upload avg                                                :         7.69          N/A          N/A Mbits/s        1599
 TCP upload fairness                                           :         0.95          N/A          N/A Mbits/s        1599
 TCP upload sum                                                :        30.76          N/A          N/A Mbits/s        1599
 cpu_stats_root@192.168.42.1::load                             :         0.36         0.33         0.75                1246

Now with my ingress shaper reduced to 100Mbps (from 105 before) and the nodes rebooted. Upload is still excellent, download looks also better for some stretches… London has recovered and is now similarly well to de which is expected after all both are close by.

Fremont shows some odd download rate spikes…

moeller0 · April 16, 2022, 9:17pm

I can send files via other ways, but I am not sure that will help right now.

Dave_Taht · April 16, 2022, 9:22pm

Fremont is the most busy dc I’m in. I just disabled ecn, and switched to sch_fq. Not trusting my cluster any at the moment… I saw some signs dctcp style ecn was on on my dallas server a while back…

moeller0 · April 17, 2022, 10:25am

Quick thought, I was using the ingress keyword which results in equalising ingress rates, not egress rates of the shaper and is known to result in unequal egress rates as it essentially distrubutes capacity by responsiveness…
Will repeat the test with egress-mode instead…

moeller0 · April 18, 2022, 9:59pm

Not sure what you changed, but even with ingress I get now:

user@work-horse:~/CODE/flent$ ./run-flent -x --ipv4 -l 120 -H de.starlink.taht.net -H london.starlink.taht.net -H singapore.starlink.taht.net -H fremont.starlink.taht.net rtt_fair4be --remote-metadata=root@192.168.42.1 --test-parameter=cpu_stats_hosts=root@192.168.42.1 --step-size=.05 --socket-stats -D . -t ingress_IPv4_SQM_cake_layer-cake_LLA-ETH_OH34_U097pct36000of36998K-D090pct105000of116797K_work-horse-eth0_2_TurrisOmnia-TurrisOS.5.3.7-pppoe-wan-eth2.7_2_bridged-BTHH5A-OpenWrt-r18441-ba7cee05ed-Hvt-VDSL100_2_netperf-eu.bufferbloat.net --log-file
Starting Flent 2.0.1-git-65cd64f using Python 3.8.10.
Starting rtt_fair4be test. Expected run time: 130 seconds.
Data file written to ./rtt_fair4be-2022-04-18T234831.603700.ingress_IPv4_SQM_cake_layer-cake_LLA-ETH_OH34_U097pct36000of36998K-D090pct105000of116797K_work-horse-eth0_2_TurrisOmnia-TurrisOS_5_3_7-pppoe-wan-eth2_7_2_bridged-BTHH5A-OpenWrt-r18441-ba7cee05ed-Hvt-VDSL1.flent.gz

Summary of rtt_fair4be test run from 2022-04-18 21:48:31.603700
  Title: 'ingress_IPv4_SQM_cake_layer-cake_LLA-ETH_OH34_U097pct36000of36998K-D090pct105000of116797K_work-horse-eth0_2_TurrisOmnia-TurrisOS.5.3.7-pppoe-wan-eth2.7_2_bridged-BTHH5A-OpenWrt-r18441-ba7cee05ed-Hvt-VDSL100_2_netperf-eu.bufferbloat.net'

                                                                          avg       median       99th %          # data pts
 Ping (ms) ICMP1 de.starlink.taht.net                          :        18.80        18.40        23.90 ms             2992
 Ping (ms) ICMP2 london.starlink.taht.net                      :        29.70        29.30        34.61 ms             2992
 Ping (ms) ICMP3 singapore.starlink.taht.net                   :       172.00       171.00       178.00 ms             2992
 Ping (ms) ICMP4 fremont.starlink.taht.net                     :       187.00       186.00       192.00 ms             2992
 Ping (ms) avg                                                 :       101.88          N/A          N/A ms             2992
 TCP download BE1 de.starlink.taht.net                         :        28.51        27.24        47.82 Mbits/s        2992
 TCP download BE2 london.starlink.taht.net                     :        26.17        25.49        43.18 Mbits/s        2992
 TCP download BE3 singapore.starlink.taht.net                  :        17.59        18.99        27.44 Mbits/s        2992
 TCP download BE4 fremont.starlink.taht.net                    :        17.72        18.64        28.77 Mbits/s        2992
 TCP download avg                                              :        22.50          N/A          N/A Mbits/s        2992
 TCP download fairness                                         :         0.95          N/A          N/A Mbits/s        2992
 TCP download sum                                              :        89.99          N/A          N/A Mbits/s        2992
 TCP upload BE1 de.starlink.taht.net                           :         9.29         8.93        17.65 Mbits/s        2992
 TCP upload BE1 de.starlink.taht.net::tcp_cwnd                 :        21.62        21.00        37.00                1655
 TCP upload BE1 de.starlink.taht.net::tcp_delivery_rate        :         8.89         8.44        15.54                1654
 TCP upload BE1 de.starlink.taht.net::tcp_pacing_rate          :        11.51        10.98        19.82                1654
 TCP upload BE1 de.starlink.taht.net::tcp_rtt                  :        26.27        25.77        36.07                1649
 TCP upload BE1 de.starlink.taht.net::tcp_rtt_var              :         1.47         1.24         4.54                1649
 TCP upload BE2 london.starlink.taht.net                       :         8.96         8.69        15.01 Mbits/s        2992
 TCP upload BE2 london.starlink.taht.net::tcp_cwnd             :        27.84        27.00        46.49                1652
 TCP upload BE2 london.starlink.taht.net::tcp_delivery_rate    :         8.68         8.34        14.51                1651
 TCP upload BE2 london.starlink.taht.net::tcp_pacing_rate      :        11.14        10.63        18.49                1652
 TCP upload BE2 london.starlink.taht.net::tcp_rtt              :        35.50        34.89        43.74                1648
 TCP upload BE2 london.starlink.taht.net::tcp_rtt_var          :         1.22         1.02         4.17                1648
 TCP upload BE3 singapore.starlink.taht.net                    :         6.08         6.67         9.11 Mbits/s        2992
 TCP upload BE3 singapore.starlink.taht.net::tcp_cwnd          :        99.85       105.00       144.00                1659
 TCP upload BE3 singapore.starlink.taht.net::tcp_delivery_rate :         6.06         6.45         8.58                1654
 TCP upload BE3 singapore.starlink.taht.net::tcp_pacing_rate   :         7.37         7.82        10.51                1656
 TCP upload BE3 singapore.starlink.taht.net::tcp_rtt           :       188.18       187.42       201.17                1649
 TCP upload BE3 singapore.starlink.taht.net::tcp_rtt_var       :         1.78         1.14        11.64                1649
 TCP upload BE4 fremont.starlink.taht.net                      :         6.55         7.19         9.51 Mbits/s        2992
 TCP upload BE4 fremont.starlink.taht.net::tcp_cwnd            :       114.80       124.00       151.00                1663
 TCP upload BE4 fremont.starlink.taht.net::tcp_delivery_rate   :         6.53         7.04         8.62                1657
 TCP upload BE4 fremont.starlink.taht.net::tcp_pacing_rate     :         8.23         8.58        16.65                1660
 TCP upload BE4 fremont.starlink.taht.net::tcp_rtt             :       201.36       200.80       212.85                1655
 TCP upload BE4 fremont.starlink.taht.net::tcp_rtt_var         :         1.50         0.97         8.30                1655
 TCP upload avg                                                :         7.72          N/A          N/A Mbits/s        2992
 TCP upload fairness                                           :         0.97          N/A          N/A Mbits/s        2992
 TCP upload sum                                                :        30.88          N/A          N/A Mbits/s        2992
 cpu_stats_root@192.168.42.1::load                             :         0.32         0.33         0.75                2369

I clearly have some cyclic gunk going on at an ~30 seconds interval, which I should look at later, but the rtt fairness is much better. With RTT being close to 1:10 rate differences being smaller than 1:2 seems pretty OK, given that this is at default interval of 100ms, which is acting against both singapore and fremont from my location.

P.S.: There are still odd rate spikes for fremont which I do not know what to make of.

Dave_Taht · April 18, 2022, 10:19pm

It’s kind of hard to complain. I’m puzzled over fremont’s behavior also. Try dallas?

Do you have the ack-filter on?

For reference, if your household can stand it, a test with sqm off would be “educational”

moeller0 · April 19, 2022, 7:08am

Will do tonight.

No, with only a mild 100/36 asymmetry and a nominally non-bursty link I leave ACKs as they are.

Can do tonight once everybody else stopped using the network.

moeller0 · April 19, 2022, 9:27pm

No SQM:

moeller@work-horse:~/CODE/flent$ ./run-flent -x --ipv4 -l 120 -H de.starlink.taht.net -H london.starlink.taht.net -H singapore.starlink.taht.net -H fremont.starlink.taht.net rtt_fair4be --remote-metadata=root@192.168.42.1 --test-parameter=cpu_stats_hosts=root@192.168.42.1 --step-size=.05 --socket-stats -D . -t ingress_IPv4_NoSQM_cake_layer-cake_LLA-ETH_OH34_U097pct36000of36998K-D090pct105000of116797K_work-horse-eth0_2_TurrisOmnia-TurrisOS.5.3.7-pppoe-wan-eth2.7_2_bridged-BTHH5A-OpenWrt-r18441-ba7cee05ed-Hvt-VDSL100_2_netperf-eu.bufferbloat.net --log-file
Starting Flent 2.0.1-git-65cd64f using Python 3.8.10.
Starting rtt_fair4be test. Expected run time: 130 seconds.
Data file written to ./rtt_fair4be-2022-04-19T231920.248480.ingress_IPv4_NoSQM_cake_layer-cake_LLA-ETH_OH34_U097pct36000of36998K-D090pct105000of116797K_work-horse-eth0_2_TurrisOmnia-TurrisOS_5_3_7-pppoe-wan-eth2_7_2_bridged-BTHH5A-OpenWrt-r18441-ba7cee05ed-Hvt-VDS.flent.gz

Summary of rtt_fair4be test run from 2022-04-19 21:19:20.248480
  Title: 'ingress_IPv4_NoSQM_cake_layer-cake_LLA-ETH_OH34_U097pct36000of36998K-D090pct105000of116797K_work-horse-eth0_2_TurrisOmnia-TurrisOS.5.3.7-pppoe-wan-eth2.7_2_bridged-BTHH5A-OpenWrt-r18441-ba7cee05ed-Hvt-VDSL100_2_netperf-eu.bufferbloat.net'

                                                                          avg       median       99th %          # data pts
 Ping (ms) ICMP1 de.starlink.taht.net                          :        36.60        38.20        53.82 ms             2931
 Ping (ms) ICMP2 london.starlink.taht.net                      :        47.60        49.20        65.02 ms             2931
 Ping (ms) ICMP3 singapore.starlink.taht.net                   :       190.00       191.00       207.00 ms             2931
 Ping (ms) ICMP4 fremont.starlink.taht.net                     :       191.00       192.00       209.00 ms             2931
 Ping (ms) avg                                                 :       116.30          N/A          N/A ms             2931
 TCP download BE1 de.starlink.taht.net                         :        44.94        48.09        65.22 Mbits/s        2931
 TCP download BE2 london.starlink.taht.net                     :        25.23        27.17        39.78 Mbits/s        2931
 TCP download BE3 singapore.starlink.taht.net                  :        19.34        16.23        69.87 Mbits/s        2931
 TCP download BE4 fremont.starlink.taht.net                    :        13.39        11.72        41.27 Mbits/s        2931
 TCP download avg                                              :        25.73          N/A          N/A Mbits/s        2931
 TCP download fairness                                         :         0.82          N/A          N/A Mbits/s        2931
 TCP download sum                                              :       102.90          N/A          N/A Mbits/s        2931
 TCP upload BE1 de.starlink.taht.net                           :        14.08        14.46        20.12 Mbits/s        2931
 TCP upload BE1 de.starlink.taht.net::tcp_cwnd                 :        48.29        49.00        75.00                1695
 TCP upload BE1 de.starlink.taht.net::tcp_delivery_rate        :        13.79        13.80        19.03                1694
 TCP upload BE1 de.starlink.taht.net::tcp_pacing_rate          :        17.74        17.36        33.62                1695
 TCP upload BE1 de.starlink.taht.net::tcp_rtt                  :        39.24        40.31        54.74                1692
 TCP upload BE1 de.starlink.taht.net::tcp_rtt_var              :         0.93         0.74         4.86                1692
 TCP upload BE2 london.starlink.taht.net                       :         9.49         9.66        15.08 Mbits/s        2931
 TCP upload BE2 london.starlink.taht.net::tcp_cwnd             :        42.11        40.00        77.00                1697
 TCP upload BE2 london.starlink.taht.net::tcp_delivery_rate    :         9.32         9.19        14.39                1696
 TCP upload BE2 london.starlink.taht.net::tcp_pacing_rate      :        11.90        11.59        21.32                1696
 TCP upload BE2 london.starlink.taht.net::tcp_rtt              :        50.27        51.27        65.51                1693
 TCP upload BE2 london.starlink.taht.net::tcp_rtt_var          :         1.25         1.01         4.98                1693
 TCP upload BE3 singapore.starlink.taht.net                    :         2.50         2.58         5.48 Mbits/s        2931
 TCP upload BE3 singapore.starlink.taht.net::tcp_cwnd          :        44.55        40.00        78.00                1720
 TCP upload BE3 singapore.starlink.taht.net::tcp_delivery_rate :         2.49         2.23         4.22                1715
 TCP upload BE3 singapore.starlink.taht.net::tcp_pacing_rate   :         3.19         2.79         6.39                1718
 TCP upload BE3 singapore.starlink.taht.net::tcp_rtt           :       201.28       202.55       219.12                1714
 TCP upload BE3 singapore.starlink.taht.net::tcp_rtt_var       :         3.43         2.87        10.46                1714
 TCP upload BE4 fremont.starlink.taht.net                      :         2.30         2.33         5.38 Mbits/s        2931
 TCP upload BE4 fremont.starlink.taht.net::tcp_cwnd            :        44.15        41.00        82.00                1727
 TCP upload BE4 fremont.starlink.taht.net::tcp_delivery_rate   :         2.29         2.09         4.14                1721
 TCP upload BE4 fremont.starlink.taht.net::tcp_pacing_rate     :         2.95         2.62         6.16                1724
 TCP upload BE4 fremont.starlink.taht.net::tcp_rtt             :       217.27       218.09       237.47                1722
 TCP upload BE4 fremont.starlink.taht.net::tcp_rtt_var         :         3.66         3.02        10.61                1722
 TCP upload avg                                                :         7.09          N/A          N/A Mbits/s        2931
 TCP upload fairness                                           :         0.67          N/A          N/A Mbits/s        2931
 TCP upload sum                                                :        28.37          N/A          N/A Mbits/s        2931
 cpu_stats_root@192.168.42.1::load                             :         0.26         0.25         0.67                2292

Note what My ISP does to my latency under load, and how much worst this looks compared to the competent AQM/traffic shaper that is cake.

Dave_Taht · April 20, 2022, 1:34am

Base on that last result… your ISP is actually doing a pretty good job at defaults. Most end-users (not gamers) would be perfectly content with only the 60ms latency and jitter you are showing, as it’s kilometers better than most dsl and cable. But since we both have OCD and can do much better for all uses and users and all workloads, we sally forth, with cake.

Thx VERY much for testing this on the turris for me. I’m still hoping other turris users running at > 500Mbit can chip in, which was my original question!

Also thx for validating my new cloud!

Dave_Taht · April 20, 2022, 1:36am

in terms of moving variables around, could you re-test with cake 100mbit with the default memlimit? I’m pretty sure at this point that its too low from various other tests, but…

moeller0 · April 20, 2022, 6:37am

Yes they have come a long way… this is a VDSL2 link with bidirectional vectoring and G.INP retransmissions. But I am using my own modem and router so can not fully appreciate the experience of the bulk of my ISPs customers who likely use the all in one modem router offered by the ISP.

Again, yes. Since it is not that hard to overload/saturate my link I really appreciate the IP and flow isolation cake’s scheduler offers…

+1; over here the plan now is to convert the bulk of the links to FTTH by 2030, though that is more aspirational than reliable. So I guess if nobody beats me to it, I might be able to post such a result within the next 8 years

My pleasure, thanks for sharing.

I can and will do tonight, but when I had a look at cake’s reported memory usage (by manually calling tc -s qdisc on a second terminal during the test) it did not seem to get close to the default 4MB.