SQM on turris flent benchmarks?

Omnia hardware Omnia HW tweaks
21

Reference memlimit 32mb 105/36

run-flent -x --ipv4 -l 60 -H de.starlink.taht.net -H london.starlink.taht.net -H singapore.starlink.taht.net -H fremont.starlink.taht.net rtt_fair4be --remote-metadata=root@192.168.42.1 --test-parameter=cpu_stats_hosts=root@192.168.42.1 --step-size=.05 --socket-stats -D . -t memlimit32mb_IPv4_SQM_cake_layer-cake_LLA-ETH_OH34_U097pct36000of36998K-D090pct105000of116797K_work-horse-eth0_2_TurrisOmnia-TurrisOS.5.3.7-pppoe-wan-eth2.7_2_bridged-BTHH5A-OpenWrt-r18441-ba7cee05ed-Hvt-VDSL100_2_xxx.starlink.taht.net --log-file

Starting Flent 2.0.1-git-65cd64f using Python 3.8.10.
Starting rtt_fair4be test. Expected run time: 70 seconds.
Data file written to ./rtt_fair4be-2022-04-21T203810.994215.memlimit32mb_IPv4_SQM_cake_layer-cake_LLA-ETH_OH34_U097pct36000of36998K-D090pct105000of116797K_work-horse-eth0_2_TurrisOmnia-TurrisOS_5_3_7-pppoe-wan-eth2_7_2_bridged-BTHH5A-OpenWrt-r18441-ba7cee05ed-Hvt-.flent.gz

Summary of rtt_fair4be test run from 2022-04-21 18:38:10.994215
  Title: 'memlimit32mb_IPv4_SQM_cake_layer-cake_LLA-ETH_OH34_U097pct36000of36998K-D090pct105000of116797K_work-horse-eth0_2_TurrisOmnia-TurrisOS.5.3.7-pppoe-wan-eth2.7_2_bridged-BTHH5A-OpenWrt-r18441-ba7cee05ed-Hvt-VDSL100_2_xxx.starlink.taht.net'

                                                                          avg       median       99th %          # data pts
 Ping (ms) ICMP1 de.starlink.taht.net                          :        19.30        19.10        26.61 ms             1604
 Ping (ms) ICMP2 london.starlink.taht.net                      :        31.60        31.40        37.70 ms             1604
 Ping (ms) ICMP3 singapore.starlink.taht.net                   :       172.00       172.00       179.00 ms             1604
 Ping (ms) ICMP4 fremont.starlink.taht.net                     :       183.00       182.00       189.00 ms             1604
 Ping (ms) avg                                                 :       101.47          N/A          N/A ms             1604
 TCP download BE1 de.starlink.taht.net                         :        37.21        36.22        56.90 Mbits/s        1604
 TCP download BE2 london.starlink.taht.net                     :        34.12        34.30        46.01 Mbits/s        1604
 TCP download BE3 singapore.starlink.taht.net                  :         3.95         4.29         7.09 Mbits/s        1604
 TCP download BE4 fremont.starlink.taht.net                    :        19.67        21.34        34.80 Mbits/s        1604
 TCP download avg                                              :        23.74          N/A          N/A Mbits/s        1604
 TCP download fairness                                         :         0.76          N/A          N/A Mbits/s        1604
 TCP download sum                                              :        94.95          N/A          N/A Mbits/s        1604
 TCP upload BE1 de.starlink.taht.net                           :         9.73         9.59        22.44 Mbits/s        1604
 TCP upload BE1 de.starlink.taht.net::tcp_cwnd                 :        22.61        22.00        39.00                 854
 TCP upload BE1 de.starlink.taht.net::tcp_delivery_rate        :         9.30         8.70        15.50                 853
 TCP upload BE1 de.starlink.taht.net::tcp_pacing_rate          :        12.06        11.39        20.40                 853
 TCP upload BE1 de.starlink.taht.net::tcp_rtt                  :        26.31        25.80        36.13                 851
 TCP upload BE1 de.starlink.taht.net::tcp_rtt_var              :         1.47         1.27         4.38                 851
 TCP upload BE2 london.starlink.taht.net                       :         9.30         9.11        16.58 Mbits/s        1604
 TCP upload BE2 london.starlink.taht.net::tcp_cwnd             :        30.45        29.00        53.00                 852
 TCP upload BE2 london.starlink.taht.net::tcp_delivery_rate    :         8.98         8.44        14.59                 851
 TCP upload BE2 london.starlink.taht.net::tcp_pacing_rate      :        11.70        10.91        20.25                 851
 TCP upload BE2 london.starlink.taht.net::tcp_rtt              :        37.27        36.49        46.68                 848
 TCP upload BE2 london.starlink.taht.net::tcp_rtt_var          :         1.26         1.03         4.46                 848
 TCP upload BE3 singapore.starlink.taht.net                    :         5.74         6.37        10.10 Mbits/s        1604
 TCP upload BE3 singapore.starlink.taht.net::tcp_cwnd          :        92.40        97.00       140.42                 859
 TCP upload BE3 singapore.starlink.taht.net::tcp_delivery_rate :         5.73         6.02         8.41                 854
 TCP upload BE3 singapore.starlink.taht.net::tcp_pacing_rate   :         7.00         7.32        10.68                 857
 TCP upload BE3 singapore.starlink.taht.net::tcp_rtt           :       184.02       183.31       200.00                 855
 TCP upload BE3 singapore.starlink.taht.net::tcp_rtt_var       :         2.11         1.22        12.11                 855
 TCP upload BE4 fremont.starlink.taht.net                      :         5.90         6.73         9.52 Mbits/s        1604
 TCP upload BE4 fremont.starlink.taht.net::tcp_cwnd            :        95.01       103.00       137.00                 857
 TCP upload BE4 fremont.starlink.taht.net::tcp_delivery_rate   :         5.89         6.46         8.49                 852
 TCP upload BE4 fremont.starlink.taht.net::tcp_pacing_rate     :         7.36         7.97        14.42                 854
 TCP upload BE4 fremont.starlink.taht.net::tcp_rtt             :       184.54       183.87       197.84                 851
 TCP upload BE4 fremont.starlink.taht.net::tcp_rtt_var         :         1.84         1.10        11.00                 851
 TCP upload avg                                                :         7.67          N/A          N/A Mbits/s        1604
 TCP upload fairness                                           :         0.94          N/A          N/A Mbits/s        1604
 TCP upload sum                                                :        30.67          N/A          N/A Mbits/s        1604
 cpu_stats_root@192.168.42.1::load                             :         0.36         0.33         0.79                1248

rtt_fair4be_-_memlimit32mb_IPv4_SQM_cake_layer-cake_LLA-ETH_OH34_U097pct36000of36998K-D090pct105000of116797K_work-horse-eth0_2_TurrisOmnia-TurrisOS.5.3.7-pppoe-wan-eth2.7_2_bridged-BTHH5A-OpenWrt-r18441-ba7cee05ed-Hvt-VDSL100_2_xxx.starlink.taht.net
rtt_fair4be_-_memlimit32mb_IPv4_SQM_cake_layer-cake_LLA-ETH_OH34_U097pct36000of36998K-D090pct105000of116797K_work-horse-eth0_2_TurrisOmnia-TurrisOS.5.3.7-pppoe-wan-eth2.7_2_bridged-BTHH5A-OpenWrt-r18441-ba7cee05ed-Hvt-VDSL100_2_xxx.starlink.taht.net1349×605 165 KB

And now with the OpenWrt default of 4mb per direction:

user@work-horse:~/CODE/flent$ ./run-flent -x --ipv4 -l 60 -H de.starlink.taht.net -H london.starlink.taht.net -H singapore.starlink.taht.net -H fremont.starlink.taht.net rtt_fair4be --remote-metadata=root@192.168.42.1 --test-parameter=cpu_stats_hosts=root@192.168.42.1 --step-size=.05 --socket-stats -D . -t memlimit4mb_IPv4_SQM_cake_layer-cake_LLA-ETH_OH34_U097pct36000of36998K-D090pct105000of116797K_work-horse-eth0_2_TurrisOmnia-TurrisOS.5.3.7-pppoe-wan-eth2.7_2_bridged-BTHH5A-OpenWrt-r18441-ba7cee05ed-Hvt-VDSL100_2_xxx.starlink.taht.net --log-file
Starting Flent 2.0.1-git-65cd64f using Python 3.8.10.
Starting rtt_fair4be test. Expected run time: 70 seconds.
Data file written to ./rtt_fair4be-2022-04-21T205112.644552.memlimit4mb_IPv4_SQM_cake_layer-cake_LLA-ETH_OH34_U097pct36000of36998K-D090pct105000of116797K_work-horse-eth0_2_TurrisOmnia-TurrisOS_5_3_7-pppoe-wan-eth2_7_2_bridged-BTHH5A-OpenWrt-r18441-ba7cee05ed-Hvt-V.flent.gz

Summary of rtt_fair4be test run from 2022-04-21 18:51:12.644552
  Title: 'memlimit4mb_IPv4_SQM_cake_layer-cake_LLA-ETH_OH34_U097pct36000of36998K-D090pct105000of116797K_work-horse-eth0_2_TurrisOmnia-TurrisOS.5.3.7-pppoe-wan-eth2.7_2_bridged-BTHH5A-OpenWrt-r18441-ba7cee05ed-Hvt-VDSL100_2_xxx.starlink.taht.net'

                                                                          avg       median       99th %          # data pts
 Ping (ms) ICMP1 de.starlink.taht.net                          :        19.30        18.90        25.60 ms             1631
 Ping (ms) ICMP2 london.starlink.taht.net                      :        30.90        30.60        37.60 ms             1631
 Ping (ms) ICMP3 singapore.starlink.taht.net                   :       172.00       172.00       178.00 ms             1631
 Ping (ms) ICMP4 fremont.starlink.taht.net                     :       183.00       182.00       189.00 ms             1631
 Ping (ms) avg                                                 :       101.30          N/A          N/A ms             1631
 TCP download BE1 de.starlink.taht.net                         :        37.17        36.36        55.30 Mbits/s        1631
 TCP download BE2 london.starlink.taht.net                     :        33.21        33.68        48.78 Mbits/s        1631
 TCP download BE3 singapore.starlink.taht.net                  :         4.13         4.30         7.42 Mbits/s        1631
 TCP download BE4 fremont.starlink.taht.net                    :        19.49        19.94        42.91 Mbits/s        1631
 TCP download avg                                              :        23.50          N/A          N/A Mbits/s        1631
 TCP download fairness                                         :         0.77          N/A          N/A Mbits/s        1631
 TCP download sum                                              :        94.00          N/A          N/A Mbits/s        1631
 TCP upload BE1 de.starlink.taht.net                           :         9.68         9.23        18.78 Mbits/s        1631
 TCP upload BE1 de.starlink.taht.net::tcp_cwnd                 :        22.50        21.00        39.73                 828
 TCP upload BE1 de.starlink.taht.net::tcp_delivery_rate        :         9.23         8.69        15.79                 828
 TCP upload BE1 de.starlink.taht.net::tcp_pacing_rate          :        11.92        11.31        21.14                 828
 TCP upload BE1 de.starlink.taht.net::tcp_rtt                  :        26.36        25.79        36.15                 826
 TCP upload BE1 de.starlink.taht.net::tcp_rtt_var              :         1.38         1.16         4.69                 826
 TCP upload BE2 london.starlink.taht.net                       :         9.32         8.98        19.14 Mbits/s        1631
 TCP upload BE2 london.starlink.taht.net::tcp_cwnd             :        30.38        29.00        49.00                 831
 TCP upload BE2 london.starlink.taht.net::tcp_delivery_rate    :         9.02         8.54        14.58                 830
 TCP upload BE2 london.starlink.taht.net::tcp_pacing_rate      :        11.62        10.91        19.67                 830
 TCP upload BE2 london.starlink.taht.net::tcp_rtt              :        37.16        36.64        47.22                 829
 TCP upload BE2 london.starlink.taht.net::tcp_rtt_var          :         1.23         0.99         4.54                 829
 TCP upload BE3 singapore.starlink.taht.net                    :         5.66         6.28         9.91 Mbits/s        1631
 TCP upload BE3 singapore.starlink.taht.net::tcp_cwnd          :        94.52        99.00       146.57                 844
 TCP upload BE3 singapore.starlink.taht.net::tcp_delivery_rate :         5.65         5.92         8.33                 839
 TCP upload BE3 singapore.starlink.taht.net::tcp_pacing_rate   :         6.87         7.16        10.40                 842
 TCP upload BE3 singapore.starlink.taht.net::tcp_rtt           :       191.22       190.57       205.30                 838
 TCP upload BE3 singapore.starlink.taht.net::tcp_rtt_var       :         2.09         1.20        14.72                 838
 TCP upload BE4 fremont.starlink.taht.net                      :         6.18         6.95        10.18 Mbits/s        1631
 TCP upload BE4 fremont.starlink.taht.net::tcp_cwnd            :        98.33       107.00       143.62                 839
 TCP upload BE4 fremont.starlink.taht.net::tcp_delivery_rate   :         6.16         6.78         8.71                 834
 TCP upload BE4 fremont.starlink.taht.net::tcp_pacing_rate     :         7.73         8.32        15.17                 836
 TCP upload BE4 fremont.starlink.taht.net::tcp_rtt             :       182.40       181.54       197.98                 833
 TCP upload BE4 fremont.starlink.taht.net::tcp_rtt_var         :         1.85         1.10        12.08                 833
 TCP upload avg                                                :         7.71          N/A          N/A Mbits/s        1631
 TCP upload fairness                                           :         0.95          N/A          N/A Mbits/s        1631
 TCP upload sum                                                :        30.84          N/A          N/A Mbits/s        1631
 cpu_stats_root@192.168.42.1::load                             :         0.39         0.40         0.90                1275

rtt_fair4be_-_memlimit4mb_IPv4_SQM_cake_layer-cake_LLA-ETH_OH34_U097pct36000of36998K-D090pct105000of116797K_work-horse-eth0_2_TurrisOmnia-TurrisOS.5.3.7-pppoe-wan-eth2.7_2_bridged-BTHH5A-OpenWrt-r18441-ba7cee05ed-Hvt-VDSL100_2_xxx.starlink.taht.net
rtt_fair4be_-_memlimit4mb_IPv4_SQM_cake_layer-cake_LLA-ETH_OH34_U097pct36000of36998K-D090pct105000of116797K_work-horse-eth0_2_TurrisOmnia-TurrisOS.5.3.7-pppoe-wan-eth2.7_2_bridged-BTHH5A-OpenWrt-r18441-ba7cee05ed-Hvt-VDSL100_2_xxx.starlink.taht.net1349×605 168 KB

1 Like
22

In Download direction I see some degradation, upload however was/is a bit crummy tonight… I guess I will increase the ingress memlimit for the future… thanks.

P.S. Since the question came up, all tests via an ubuntu 20LTS host with on-board braodcom gigabit ethernet NIC. That host was controlled via ssh from a different wired computer.

1 Like
23

Well, darn it, singapore shouldn’t look like that. I’ll poke into it. thx for (inadvertently) testing out my cluster at these rtts!

1 Like
24

Still wanna test 1Gbps+, @Dave_Taht ?

25

very much so! Everything except singapore seem capable of it, but in part by spreading the load worldwide, we get better data and stress out the servers less. I have other boxes up (atlanta, mumbai) that are worth testing from europe, also.

1 Like
26

Cool. So I hope to get my colleagues to test it next week.

27

I don’t know much about the turris. Does the ethernet driver have BQL and fq_codel on? It’s the default in openwrt, but a lot of devices now-adays do offloads. ?

 tc -s qdisc show

So an rtt_fair result like this would be useful on a gbit-symmetric link for reference, then one with sqm on, taking a look at the cpu usage for both. IF you have working BQL, cake at line rate (no bandwidth parameter) would be interesting as well. After that tracing some cpu bottlenecks would start to be useful.

I’m pretty sure from looking at this data (so far) that the default openwrt memlimit for fq_codel and cake is too low for these rates - the linux default of 32MB we tested to 10Gbit on other networks, but we didn’t look hard enough at the long-rtt cases at the openwrt patched default of 4Mb.

1 Like
28

This is new tools for me and I do not have as deep knowledge as you guys but I made a couple of test runs on my Turris MOX to night.
I’m located in sweden and have a 1G uplink from MOX

labuser@lab1:~/flent$ flent -x --socket-stats --step-size=.05 -H de.starlink.taht.net -H london.starlink.taht.net -H singapore.starlink.taht.net -H netperf-eu.bufferbloat.net -t rtt_fair4be-SQM_fq_codel_simple.qos-upload850000_download850000_1Gfibre_MOX rtt_fair4be
Started Flent 1.3.2 using Python 3.8.10.
Starting rtt_fair4be test. Expected run time: 70 seconds.
Data file written to ./rtt_fair4be-2022-04-21T213249.605471.rtt_fair4be-SQM_fq_codel_simple_qos-upload850000_download850000_1Gfibre_MOX.flent.gz.
Summary of rtt_fair4be test run from 2022-04-21 19:32:49.605471
  Title: 'rtt_fair4be-SQM_fq_codel_simple.qos-upload850000_download850000_1Gfibre_MOX'
                                                                          avg       median          # data pts
 Ping (ms) ICMP1 de.starlink.taht.net                          :        36.69        35.90 ms             1398
 Ping (ms) ICMP2 london.starlink.taht.net                      :        37.17        36.40 ms             1398
 Ping (ms) ICMP3 singapore.starlink.taht.net                   :       258.17       257.00 ms             1398
 Ping (ms) ICMP4 netperf-eu.bufferbloat.net                    :         7.56         6.77 ms             1398
 Ping (ms) avg                                                 :        84.90          N/A ms             1398
 TCP download BE1 de.starlink.taht.net                         :        16.99        16.62 Mbits/s        1398
 TCP download BE2 london.starlink.taht.net                     :        16.03        11.66 Mbits/s        1398
 TCP download BE3 singapore.starlink.taht.net                  :        16.57         5.01 Mbits/s        1398
 TCP download BE4 netperf-eu.bufferbloat.net                   :       211.36       203.50 Mbits/s        1398
 TCP download avg                                              :        65.24          N/A Mbits/s        1398
 TCP download fairness                                         :         0.37          N/A Mbits/s        1398
 TCP download sum                                              :       260.95          N/A Mbits/s        1398
 TCP upload BE1 de.starlink.taht.net                           :        62.83        68.66 Mbits/s        1398
 TCP upload BE1 de.starlink.taht.net::tcp_cwnd                 :       205.25       190.00                 580
 TCP upload BE1 de.starlink.taht.net::tcp_delivery_rate        :        62.35        57.85                 579
 TCP upload BE1 de.starlink.taht.net::tcp_pacing_rate          :        78.38        72.13                 580
 TCP upload BE1 de.starlink.taht.net::tcp_rtt                  :        36.91        36.38                 580
 TCP upload BE1 de.starlink.taht.net::tcp_rtt_var              :         0.78         0.45                 580
 TCP upload BE2 london.starlink.taht.net                       :        87.41        79.88 Mbits/s        1398
 TCP upload BE2 london.starlink.taht.net::tcp_cwnd             :       292.67       245.50                 582
 TCP upload BE2 london.starlink.taht.net::tcp_delivery_rate    :        87.65        75.07                 581
 TCP upload BE2 london.starlink.taht.net::tcp_pacing_rate      :       109.57        92.84                 582
 TCP upload BE2 london.starlink.taht.net::tcp_rtt              :        37.51        36.85                 580
 TCP upload BE2 london.starlink.taht.net::tcp_rtt_var          :         0.67         0.38                 580
 TCP upload BE3 singapore.starlink.taht.net                    :        12.96        13.21 Mbits/s        1398
 TCP upload BE3 singapore.starlink.taht.net::tcp_cwnd          :       293.20       276.00                 597
 TCP upload BE3 singapore.starlink.taht.net::tcp_delivery_rate :        13.01        12.28                 592
 TCP upload BE3 singapore.starlink.taht.net::tcp_pacing_rate   :        16.73        15.10                 594
 TCP upload BE3 singapore.starlink.taht.net::tcp_rtt           :       258.57       257.99                 592
 TCP upload BE3 singapore.starlink.taht.net::tcp_rtt_var       :         1.89         0.65                 592
 TCP upload BE4 netperf-eu.bufferbloat.net                     :       392.23       395.09 Mbits/s        1398
 TCP upload BE4 netperf-eu.bufferbloat.net::tcp_cwnd           :       359.41       357.00                 575
 TCP upload BE4 netperf-eu.bufferbloat.net::tcp_delivery_rate  :       387.82       386.62                 575
 TCP upload BE4 netperf-eu.bufferbloat.net::tcp_pacing_rate    :       511.19       513.45                 575
 TCP upload BE4 netperf-eu.bufferbloat.net::tcp_rtt            :        10.04         9.21                 575
 TCP upload BE4 netperf-eu.bufferbloat.net::tcp_rtt_var        :         0.31         0.21                 575
 TCP upload avg                                                :       138.86          N/A Mbits/s        1398
 TCP upload fairness                                           :         0.47          N/A Mbits/s        1398
 TCP upload sum                                                :       555.43          N/A Mbits/s        1398

Screenshot from 2022-04-21 22-33-06
Screenshot from 2022-04-21 22-33-061466×580 243 KB

1 Like
30

thank you very much for the new data. I figured things would deteriorate at higher rates, but not by that much. What happens with no sqm on at all?

  1. could you use fremont.starlink.taht.net rather than netperf-eu?

  2. I wish there was a way to post the flent.gz files also. As much as I like and prefer to look at the long form plot, the summary plots we get from flent-gui, especially the cdfs and totals plots are often useful.

1 Like
31

Glad I can help.
As your host set did not deliver anywhere near the throughput of >500 so that was why I used the netperf-eu host.
But I will rerun with your host set shortly.

1 Like
32

EDIT:not relevant test as I ran over wifi (not turris wifi, external AP)

labuser@lab1:~/flent$ flent -x --socket-stats --step-size=.05 -H de.starlink.taht.net -H london.starlink.taht.net -H singapore.starlink.taht.net -H fremont.starlink.taht.net -t rtt_fair4be-SQM_fq_codel_simple.qos-upload850000_download850000_1Gfibre_MOX rtt_fair4be
Started Flent 1.3.2 using Python 3.8.10.
Starting rtt_fair4be test. Expected run time: 70 seconds.
Data file written to ./rtt_fair4be-2022-04-21T221617.611439.rtt_fair4be-SQM_fq_codel_simple_qos-upload850000_download850000_1Gfibre_MOX.flent.gz.

Summary of rtt_fair4be test run from 2022-04-21 20:16:17.611439
  Title: 'rtt_fair4be-SQM_fq_codel_simple.qos-upload850000_download850000_1Gfibre_MOX'

                                                                          avg       median          # data pts
 Ping (ms) ICMP1 de.starlink.taht.net                          :       114.46       101.00 ms             1400
 Ping (ms) ICMP2 london.starlink.taht.net                      :       115.47       101.00 ms             1400
 Ping (ms) ICMP3 singapore.starlink.taht.net                   :       336.37       322.00 ms             1400
 Ping (ms) ICMP4 fremont.starlink.taht.net                     :       235.67       218.00 ms             1400
 Ping (ms) avg                                                 :       200.49          N/A ms             1400
 TCP download BE1 de.starlink.taht.net                         :        36.41        36.44 Mbits/s        1400
 TCP download BE2 london.starlink.taht.net                     :        19.74        19.26 Mbits/s        1400
 TCP download BE3 singapore.starlink.taht.net                  :        17.34        19.37 Mbits/s        1400
 TCP download BE4 fremont.starlink.taht.net                    :        15.33        18.21 Mbits/s        1400
 TCP download avg                                              :        22.20          N/A Mbits/s        1400
 TCP download fairness                                         :         0.88          N/A Mbits/s        1400
 TCP download sum                                              :        88.82          N/A Mbits/s        1400
 TCP upload BE1 de.starlink.taht.net                           :        54.35        55.88 Mbits/s        1400
 TCP upload BE1 de.starlink.taht.net::tcp_cwnd                 :       696.33       410.00                 543
 TCP upload BE1 de.starlink.taht.net::tcp_delivery_rate        :        53.62        44.62                 542
 TCP upload BE1 de.starlink.taht.net::tcp_pacing_rate          :        70.40        58.18                 542
 TCP upload BE1 de.starlink.taht.net::tcp_rtt                  :       125.65       113.31                 542
 TCP upload BE1 de.starlink.taht.net::tcp_rtt_var              :         2.48         2.06                 542
 TCP upload BE2 london.starlink.taht.net                       :        43.88        49.09 Mbits/s        1400
 TCP upload BE2 london.starlink.taht.net::tcp_cwnd             :       569.10       409.50                 546
 TCP upload BE2 london.starlink.taht.net::tcp_delivery_rate    :        42.60        41.04                 545
 TCP upload BE2 london.starlink.taht.net::tcp_pacing_rate      :        57.14        52.53                 545
 TCP upload BE2 london.starlink.taht.net::tcp_rtt              :       126.22       114.28                 545
 TCP upload BE2 london.starlink.taht.net::tcp_rtt_var          :         2.58         2.28                 545
 TCP upload BE3 singapore.starlink.taht.net                    :         5.18         5.35 Mbits/s        1400
 TCP upload BE3 singapore.starlink.taht.net::tcp_cwnd          :       161.21       169.50                 546
 TCP upload BE3 singapore.starlink.taht.net::tcp_delivery_rate :         5.12         5.20                 540
 TCP upload BE3 singapore.starlink.taht.net::tcp_pacing_rate   :         6.59         6.49                 543
 TCP upload BE3 singapore.starlink.taht.net::tcp_rtt           :       362.46       350.33                 543
 TCP upload BE3 singapore.starlink.taht.net::tcp_rtt_var       :         7.26         4.55                 543
 TCP upload BE4 fremont.starlink.taht.net                      :        22.61        26.31 Mbits/s        1400
 TCP upload BE4 fremont.starlink.taht.net::tcp_cwnd            :       466.21       326.00                 550
 TCP upload BE4 fremont.starlink.taht.net::tcp_delivery_rate   :        22.56        19.09                 547
 TCP upload BE4 fremont.starlink.taht.net::tcp_pacing_rate     :        29.20        23.31                 548
 TCP upload BE4 fremont.starlink.taht.net::tcp_rtt             :       246.76       234.06                 547
 TCP upload BE4 fremont.starlink.taht.net::tcp_rtt_var         :         4.63         3.02                 547
 TCP upload avg                                                :        31.51          N/A Mbits/s        1400
 TCP upload fairness                                           :         0.73          N/A Mbits/s        1400
 TCP upload sum                                                :       126.02          N/A Mbits/s        1400
33

yes, both fremont and singapores behaviors are puzzling. I just made a configuration change to fremont a second ago… this second run is worse than your first…

1 Like
34

hahah , sorry I screwed up, that test ran over wifi…
I will re run now

35 
labuser@lab1:~/flent$ flent -x --socket-stats --step-size=.05 -H de.starlink.taht.net -H london.starlink.taht.net -H singapore.starlink.taht.net -H fremont.starlink.taht.net -t rtt_fair4be-SQM_fq_codel_simple.qos-upload850000_download850000_1Gfibre_MOX rtt_fair4be
Started Flent 1.3.2 using Python 3.8.10.
Starting rtt_fair4be test. Expected run time: 70 seconds.
Data file written to ./rtt_fair4be-2022-04-21T222602.107032.rtt_fair4be-SQM_fq_codel_simple_qos-upload850000_download850000_1Gfibre_MOX.flent.gz.

Summary of rtt_fair4be test run from 2022-04-21 20:26:02.107032
  Title: 'rtt_fair4be-SQM_fq_codel_simple.qos-upload850000_download850000_1Gfibre_MOX'

                                                                          avg       median          # data pts
 Ping (ms) ICMP1 de.starlink.taht.net                          :        36.36        35.70 ms             1400
 Ping (ms) ICMP2 london.starlink.taht.net                      :        36.86        36.20 ms             1400
 Ping (ms) ICMP3 singapore.starlink.taht.net                   :       257.95       257.00 ms             1400
 Ping (ms) ICMP4 fremont.starlink.taht.net                     :       158.58       158.00 ms             1400
 Ping (ms) avg                                                 :       122.44          N/A ms             1400
 TCP download BE1 de.starlink.taht.net                         :        39.70        37.43 Mbits/s        1400
 TCP download BE2 london.starlink.taht.net                     :        28.21        28.41 Mbits/s        1400
 TCP download BE3 singapore.starlink.taht.net                  :        24.21        13.81 Mbits/s        1400
 TCP download BE4 fremont.starlink.taht.net                    :        25.03         9.67 Mbits/s        1400
 TCP download avg                                              :        29.29          N/A Mbits/s        1400
 TCP download fairness                                         :         0.96          N/A Mbits/s        1400
 TCP download sum                                              :       117.15          N/A Mbits/s        1400
 TCP upload BE1 de.starlink.taht.net                           :        69.15        75.19 Mbits/s        1400
 TCP upload BE1 de.starlink.taht.net::tcp_cwnd                 :       226.32       223.00                 551
 TCP upload BE1 de.starlink.taht.net::tcp_delivery_rate        :        68.84        68.83                 551
 TCP upload BE1 de.starlink.taht.net::tcp_pacing_rate          :        87.20        85.05                 551
 TCP upload BE1 de.starlink.taht.net::tcp_rtt                  :        36.74        36.04                 551
 TCP upload BE1 de.starlink.taht.net::tcp_rtt_var              :         0.71         0.39                 551
 TCP upload BE2 london.starlink.taht.net                       :       121.83       120.08 Mbits/s        1400
 TCP upload BE2 london.starlink.taht.net::tcp_cwnd             :       409.68       379.00                 553
 TCP upload BE2 london.starlink.taht.net::tcp_delivery_rate    :       121.81       112.94                 552
 TCP upload BE2 london.starlink.taht.net::tcp_pacing_rate      :       154.13       140.49                 553
 TCP upload BE2 london.starlink.taht.net::tcp_rtt              :        37.62        37.01                 553
 TCP upload BE2 london.starlink.taht.net::tcp_rtt_var          :         0.56         0.32                 553
 TCP upload BE3 singapore.starlink.taht.net                    :        50.93        50.07 Mbits/s        1400
 TCP upload BE3 singapore.starlink.taht.net::tcp_cwnd          :      1154.41      1206.00                 551
 TCP upload BE3 singapore.starlink.taht.net::tcp_delivery_rate :        50.69        53.63                 546
 TCP upload BE3 singapore.starlink.taht.net::tcp_pacing_rate   :        64.79        66.53                 548
 TCP upload BE3 singapore.starlink.taht.net::tcp_rtt           :       259.02       258.17                 546
 TCP upload BE3 singapore.starlink.taht.net::tcp_rtt_var       :         1.67         0.51                 546
 TCP upload BE4 fremont.starlink.taht.net                      :        53.18        49.56 Mbits/s        1400
 TCP upload BE4 fremont.starlink.taht.net::tcp_cwnd            :       805.76       660.50                 554
 TCP upload BE4 fremont.starlink.taht.net::tcp_delivery_rate   :        53.38        47.58                 551
 TCP upload BE4 fremont.starlink.taht.net::tcp_pacing_rate     :        78.91        58.86                 553
 TCP upload BE4 fremont.starlink.taht.net::tcp_rtt             :       159.55       158.46                 552
 TCP upload BE4 fremont.starlink.taht.net::tcp_rtt_var         :         1.31         0.48                 552
 TCP upload avg                                                :        73.77          N/A Mbits/s        1400
 TCP upload fairness                                           :         0.87          N/A Mbits/s        1400
 TCP upload sum                                                :       295.09          N/A Mbits/s        1400

Screenshot from 2022-04-21 22-30-49
Screenshot from 2022-04-21 22-30-491463×577 322 KB

1 Like
36

one of my goals is indeed to get good reference results for the turris wifi, also. could you clearly mark in the title what is under test? is this with or without sqm?

1 Like
37

Ok I tried to be as clear as possibel in the title, with sqm and the corresponding conf.
But maybe there is more to it than my simple conf in the luci gui.
What more detailes do you like to see?

1 Like
38

It could use to be shorter. :sunny: Honestly though this is a great and useful amount amount of detail.

SQM_fq_codel_simple.qos-upload850000_download850000_1Gfibre_MOX

Can you turn sqm off and try once more?

NOSQM_1Gfibre_MOX

1 Like
39

Ok, good
Yes, but it will have to wait untill tomorrow.

1 Like
40

test run with no sqm
I will be happy to send you the gz files if you want, just send me a DM with the detailes where to send them

flent -4 -x --socket-stats --step-size=.05 -H de.starlink.taht.net -H london.starlink.taht.net -H singapore.starlink.taht.net -H fremont.starlink.taht.net -t NOSQM_1Gfibre_MOX rtt_fair4be
Started Flent 1.3.2 using Python 3.8.10.
Starting rtt_fair4be test. Expected run time: 70 seconds.
Data file written to ./rtt_fair4be-2022-04-22T221504.882205.NOSQM_1Gfibre_MOX.flent.gz.

Summary of rtt_fair4be test run from 2022-04-22 20:15:04.882205
  Title: 'NOSQM_1Gfibre_MOX'

                                                                          avg       median          # data pts
 Ping (ms) ICMP1 de.starlink.taht.net                          :        35.91        35.40 ms             1400
 Ping (ms) ICMP2 london.starlink.taht.net                      :        36.41        35.90 ms             1400
 Ping (ms) ICMP3 singapore.starlink.taht.net                   :       257.99       257.00 ms             1400
 Ping (ms) ICMP4 fremont.starlink.taht.net                     :       158.40       158.00 ms             1400
 Ping (ms) avg                                                 :       122.18          N/A ms             1400
 TCP download BE1 de.starlink.taht.net                         :        44.92        38.84 Mbits/s        1400
 TCP download BE2 london.starlink.taht.net                     :        34.18        32.43 Mbits/s        1400
 TCP download BE3 singapore.starlink.taht.net                  :        21.36        14.72 Mbits/s        1400
 TCP download BE4 fremont.starlink.taht.net                    :         7.19         5.23 Mbits/s        1400
 TCP download avg                                              :        26.91          N/A Mbits/s        1400
 TCP download fairness                                         :         0.78          N/A Mbits/s        1400
 TCP download sum                                              :       107.65          N/A Mbits/s        1400
 TCP upload BE1 de.starlink.taht.net                           :       111.00       104.69 Mbits/s        1400
 TCP upload BE1 de.starlink.taht.net::tcp_cwnd                 :       356.93       314.00                 454
 TCP upload BE1 de.starlink.taht.net::tcp_delivery_rate        :       110.55        98.43                 454
 TCP upload BE1 de.starlink.taht.net::tcp_pacing_rate          :       142.27       120.49                 454
 TCP upload BE1 de.starlink.taht.net::tcp_rtt                  :        36.28        35.72                 454
 TCP upload BE1 de.starlink.taht.net::tcp_rtt_var              :         0.49         0.22                 454
 TCP upload BE2 london.starlink.taht.net                       :       100.53       106.82 Mbits/s        1400
 TCP upload BE2 london.starlink.taht.net::tcp_cwnd             :       322.75       324.00                 451
 TCP upload BE2 london.starlink.taht.net::tcp_delivery_rate    :       100.52       100.05                 451
 TCP upload BE2 london.starlink.taht.net::tcp_pacing_rate      :       124.46       122.22                 451
 TCP upload BE2 london.starlink.taht.net::tcp_rtt              :        36.64        36.18                 451
 TCP upload BE2 london.starlink.taht.net::tcp_rtt_var          :         0.42         0.20                 451
 TCP upload BE3 singapore.starlink.taht.net                    :        44.99        44.39 Mbits/s        1400
 TCP upload BE3 singapore.starlink.taht.net::tcp_cwnd          :      1068.12       938.00                 461
 TCP upload BE3 singapore.starlink.taht.net::tcp_delivery_rate :        45.63        41.06                 457
 TCP upload BE3 singapore.starlink.taht.net::tcp_pacing_rate   :        61.41        54.49                 459
 TCP upload BE3 singapore.starlink.taht.net::tcp_rtt           :       259.28       258.42                 459
 TCP upload BE3 singapore.starlink.taht.net::tcp_rtt_var       :         2.02         0.39                 459
 TCP upload BE4 fremont.starlink.taht.net                      :        34.29        35.01 Mbits/s        1400
 TCP upload BE4 fremont.starlink.taht.net::tcp_cwnd            :       473.96       433.50                 464
 TCP upload BE4 fremont.starlink.taht.net::tcp_delivery_rate   :        34.54        31.66                 462
 TCP upload BE4 fremont.starlink.taht.net::tcp_pacing_rate     :        43.60        38.69                 463
 TCP upload BE4 fremont.starlink.taht.net::tcp_rtt             :       158.79       158.16                 463
 TCP upload BE4 fremont.starlink.taht.net::tcp_rtt_var         :         1.01         0.32                 463
 TCP upload avg                                                :        72.70          N/A Mbits/s        1400
 TCP upload fairness                                           :         0.83          N/A Mbits/s        1400
 TCP upload sum                                                :       290.81          N/A Mbits/s        1400

Screenshot from 2022-04-22 22-18-52
Screenshot from 2022-04-22 22-18-521440×569 300 KB

1 Like
41

Thanks for writing up the report on all Your effort, @Dave_Taht!

https://www.patreon.com/posts/state-of-fq-and-65554532?utm_medium=social&utm_source=twitter&utm_campaign=postshare

2 Likes