So DOCSIS networks use a shaper to make sure no user ever exceeds its contracted rate (except temporarily for power boost like systems, but that is part of the contract). This shaper incidentally only takes the full ethernet frame with frame check sequence into account, see http://www.cablelabs.com/specification/docsis-3-0-mac-and-upper-layer-protocols-interface-specification/:
“C.2.2.7.2 Maximum Sustained Traffic Rate 632 This parameter is the rate parameter R of a token-bucket-based rate limit for packets. R is expressed in bits per second, and MUST take into account all MAC frame data PDU of the Service Flow from the byte following the MAC header HCS to the end of the CRC, including every PDU in the case of a Concatenated MAC Frame. This parameter is applied after Payload Header Suppression; it does not include the bytes suppressed for PHS. The number of bytes forwarded (in bytes) is limited during any time interval T by Max(T), as described in the expression: Max(T) = T * (R / 8) + B, (1) where the parameter B (in bytes) is the Maximum Traffic Burst Configuration Setting (refer to Annex C.2.2.7.3). NOTE: This parameter does not limit the instantaneous rate of the Service Flow. The specific algorithm for enforcing this parameter is not mandated here. Any implementation which satisfies the above equation is conformant. In particular, the granularity of enforcement and the minimum implemented value of this parameter are vendor specific. The CMTS SHOULD support a granularity of at most 100 kbps. The CM SHOULD support a granularity of at most 100 kbps. NOTE: If this parameter is omitted or set to zero, then there is no explicitly-enforced traffic rate maximum. This field specifies only a bound, not a guarantee that this rate is available.”
So in essence DOCSIS users need to only account for 18 Bytes of ethernet overhead in both ingress and egress directions under non-congested conditions.
BUT the bigger problem on DOCSIS networks are, a) this shaping approach is actually insufficient as it does not really account for the actual DOCSIS wire-sizes of the packets (smaller packets have a higher DOCSIS cost and that is not reflected by the shaper) and b) end users typically have no precise information about the actually configured bandwidth of the shaper. On a positive note many cable ISPs seem to provision higher bandwidths than expected from looking at the contracts. On a less positive note und congestion all bets are off ;). That said, no shaper will work that well under congestion, it is jus t that the DOCSIS shaper, by not properly accounting for the on-the-wire-size will have no real chance of fairly distributing the reduced available bandwidth under congestion (which mostly is a theoretical problem, good DOCSIS ISPs will split their nodes if congestion happens too often, but I digress).
Back to your observation, on systems with precisely defined (guaranteed) bandwidth like VDSL2, it often is possible to reduce/eradicate bufferbloat with the shaper set at 100% of the link rate, IF all overhead is accounted correctly, with DOCSIS this has not yet been observed. I note that your 28 is actually too high (the real per packet overhead is 18), and setting the overhead too high effectively lowers the shaper bandwidth which will almost always help against buffer bloat…
In your case, if you see a step in buffer bloat rating by going from obehead 18 to overhead 17 then I would argue that your on to something
Thanks @moeller0 with your fantastically good and detailed explanations .
I have been reading up and testing some extra arguments to add to the “Advanced option string to pass to the ingress/egress queueing disciplines” fields. These can be beneficial to some users but these are not a magic bullet to magically increase your internet connection speed.
Other then that, I am busy with some RL stuff. When I have the time I will expand and reformat the OP and write a wiki article about SQM.
I @iddqd I added a bit about DOCSIS to https://lede-project.org/docs/howto/sqm#sqmlink_layer_adaptation_tab which I consider to be one of the canonical references for sqm scripts nowadays. This still has no information about adding things to the advanced qdisc option fields, but that is very much on purpose, as these are truly dangerous (one typo or un-supported keyword and sqm will more or less silently fail to instantiate, which is quite irritating for experts, let alone new-comers). The intermediate goal is to add explicit controls for cake’s advanced options to sqm-scripts and the GUI. That said, I like your instructions, including the instructions about the avanced qdisc options, as these should be documented somewhere, just not at the canonical “official” descriptions. I also believe that turrisOS and lede ship slightly different versions of sch_cake and the tc binary, so it makes sense to keep dedicated instructions for each “distribution”.
Best Regards & Thanks for your great documentation (I have pointed several people to that post of yours saving a ton of time in the process.)
First of all a generous thank you for your excellent explanations of the topic ! Have your hints obeyed and the values look already really good, stand now however before the question what is now in my case the correct value for the Overhead !? ISP is Tele Columbus (cable) with 200Mbit / 8Mbit. As you’ve described, your recommendation was 18 bytes. In addition, I have only one field in sqm linkage in Omnia. Do you have a recommendation?
Go to the GUI’s “Link Layer Adaptation” tab, check the “Show Advanced Linklayer Options, (only needed if MTU > 1500). Advanced options will only be used as long as this box is checked.” checkbox and select cake as “Which linklayer adaptation mechanism to use; for testing only”. Also add 64 in the tcMPU field.
That would be more or less what I would recommend, but you are starting from a pretty good baseline, so please do not expect much behavioral changes from those small configuration tweaks.
how might i make a specific internal ip’s packets high priority? say my computer ip, or game console ip? (ps: i wouldn’t know where to start in terms of settings since there is no dedicated device prioritization section in luci )
i hear ya, right now i have the latency decently low and stable. now when one of my family members decides they want to stream videos or use netflix that stable ping becomes tripled and even higher at times. and that’s where i want to see if there is anything i can do in luci to make the voip/ or gaming packets the highest priority
yes. when they start video streaming i notice my voip starts to degrade or cut off for a split second. and when in a pvp game the shots don’t register as instantly when they are video streaming… seems as if the video packets are taking a front seat and getting the priority over the game. in other words the shots feel “heavy” and delayed when they are streaming and when no one is on the internet the shots feel instant “light” snappy.
Well, there are at some issues at play here:
a) Currently ingress shaping does not account for the dropped packets aggressively enough, resulting in less drop feedback for the senders. (Upstream) Cake has an experimental ingress keyword with which t attempts to drop more aggressively. To test for this you might be available to set the ingress shaper to 50% of the line-rate so that sqm dropps more aggressively to begin with.
b) Many newer “streaming sources” actually do not stream at the intended bandwidth but put large bursts at maximum available bandwidth followed by periods of quiet. Making it somewhat unruly for a shaper to deal with.
c) Some “streamers” actually started to use TCP protocols that use queueing delay as signal to reduce the sending bandwidth instead of dropped packets, assuming that unmanaged queues show a large increase in transmission latency on congestion. But since sqm explicitly tries to keep the encountered delay short these senders often do not seem to get the congestion signal and keep “spamming” the downstream (as the queue backspills into the ISP upstream equipment), and if they drown out/delay the latency critical packets already of that stage where sqm has no way to selectively prioritize them.
It is not guaranteed that any of these issues are actually causing your problem though…
very insightful. thank you for replying as well. although i more or less understand what you are saying, wondering if you could elaborate on the part where you say “(Upstream) Cake has an experimental ingress keyword with which t attempts to drop more aggressively. To test for this you might be available to set the ingress shaper to 50% of the line-rate so that sqm dropps more aggressively to begin with.”
that part where you say experimental ingress keyword , what do you mean by that? i know there are the “dangerous advanced settings” on qos sqm. are you referring those when advanced options are expanded?
i am referring to the queue discipline TAB and under that is another tab for (Show and Use Advanced Configuration. Advanced options will only be used as long as this box is checked.)
also when i use the show and use advanced configuration etc settings that include that"nat dual-dsthost" input, i get ping spikes upwards to 200/400ms every so often . when i disable the advanced configuration the ping doesn’t spike that high. any insite on as to why it may do that?
The upstream cake source (https://github.com/dtaht/sch_cake) has a development branch called cobalt (https://github.com/dtaht/sch_cake/tree/cobalt)) and there exists the ingress keyword, so if you build your own module from source and also build the matching tc (from https://github.com/dtaht/tc-adv) you could try the ingress keyword yourself. Now if you have tc and sch_cake that know how to interpret the ingress keyword you would add this to the advanced option string, please note that as far as I can tell the sch_cake and tc from the turris repository do not (yet?) support the ingress keyword, so this information I gave is not immediately actionable.
Also could you remind me again what link technology you are usng, dsl, cable or fiber (I am sure you told me already, but I follow slightly more threads than I can comfortably keep all the details for straight)
thanks again for your replies! before i go ahead and post those quick question, what is this chaos calmer ive heard about? can it help with maintaining/ lowering ping and can that be installed on turris omnia as simply as say that sched cake mod thing i installed for cake?
Chaos Calmer is the codename for 15.05.1 branch of OpenWRT (and also the name for a nice cocktail). Turris OS is based of of it but with parts of LEDE added.
gotcha. how can i go about getting qdisc pfifo added to the list of qdiscs like piece of cake/layer of cake etc?
i hear sfq and pfifo_fast is a decent combination for packets to get out quickly (mind you i have very little knowledge on sqm and qos /qdiscs things… so maybe im speaking on some old qdisc and you guys can point me in the right direction… im interested in getting my gaming or voip device’s packets out the quickest so that device’s ping can be a little lower
Ah, sorry for being to terse, please use ssh to log into your router (see https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html for a windows ssh client). Then execute the parts contained in the double quotes, but without the quotes themselves, and copy the output of those commands from ssh and post it here:
“cat /etc/config/sqm” this just dumps your current configuration for sqm-scripts
“tc -s qdisc” this will show statistics for your qdisc, as above, reboot your router and just run a quick speedtest and then execute the command. Keep your router running and once you notice that the latency spikes have (just) occurred again repeat “tc -s qdisc”. Also since you are on cable/DOCSIS, what is the precise modem you use (there have been reports that intel puma6 based modems create latency spikes just by them selves in that case sqm-scripts will never be able to control bufferbloat 100%)?
! Have your hints obeyed and the values look already really good, stand now however before the question what is now in my case the correct value for the Overhead !? ISP is Tele Columbus (cable) with 200Mbit / 8Mbit. As you’ve described, your recommendation was 18 bytes. In addition, I have only one field in sqm linkage in Omnia. Do you have a recommendation?
