How to use the cake queue management system on the Turris Omnia

Thanks you for your technical insight and always helping people in the forum :slight_smile:

@moeller0, I totally second @iddqd thank you!

And @iddqd, thank you for the great cake write-up! Without it I would never even have attempted to configure cake in any form.

I will try out layer_cake this evening.

@moeller0,

You were right. I switched over to layer_cake without a hitch. So, out of curiosity, what is the difference between piece_of-cake and layer_cake?

Back to my VoIP issue:

Hoping that my packets would arrive appropriately marked on the [Queue Discipline] tab I set:
“Squash DSCP on inbound packets (ingress):” to “DO NOT SQUASH”
“Ignore DSCP on ingress:” to “Allow”

Would that have been appropriate?

Unfortunately, I see no DSCP marking so I suspect that in practice neither of those settings actually matter.

OTOH, my VoIP box is the sole device connected to my TO’s eth2 port. Is there some means by which I can add appropriate DSCP markings on ingress?

Glad it worked!

Basically piece-of_cake instantiates a single tin (a tin is a priority tier in cake parlance) shaper, while layer_cake insatantiates a 3 tin shaper. The idea is that there is one tin for background traffic (CS1-marked) that will give way to the other two tiers. There is one tin for most besteffort traffic, and there is one tin (with the smallest bandwidth) for high priority markings. Ideally the VoIP packets you want expedited are marked appropriately (cake’s hight priority tin is based on those DSCPs that typically are used for e.g. VoIP packets).

That depends, if you trust your upstream to send you correctly marked packets that sounds reasonable (but you do not need to trust, you can use wireshark/tcpdump and create an actual packet dump and measure/control that the upstream does the right thing) Also cake will try not to be gamed, if all packets are marked high-priority, cake will only expedite a small percentage of those ans delegate most back to the best effort category, but I digress.

Well, if your ISP zeros the DSCP bits of incoming packets, then the priretisation will not work for inbound packets…

You could try that, but it is very unlikely that you will manage to have this re-mapping happening before cake gets hold of incoing packets, so this will be rather complicated. What about you simply try how well sqm-scripts and layer_cake.qos perform without additional twaeking? With a bit of luck it will be good enough (as they say, perfect is the enemy of good :wink: )

Best Regards

1 Like

@moeller0,

Thanks for your continuing guidance. I am in process of “cutting the cord”. As you may have surmised, as part of that effor, I am replacing Comcast’s phone service with my own VoIP box.

Yesterday I replaced Comcast internet with Google fiber. Of course I expected much higher, nearly symmetric bandwidth. What I was not expecting was the better than an order of magnitude drop in latency. Given this new state of affairs I am happy to follow your advice re not fiddling further with packet processing.

Out of curiosity, if the stack provides only DSCP mark squashing rather than any actual packet marking, how do sqm-scripts and layer_cake.qos choose into what tin to place a packer? Does it depend on packets arriving with trustable DSCP marks? Is it then your expectation that, over time, such marks will become the norm?

1 Like

So currently most sqm scripts only look at the DSCPs.

Well, expectation is a bit strong, so rather hope. One option would be to use connection tracking to basically copy the DSCPs from outgoing packets to incoming packets of the same flow, that would not put blind trust in incoming packets, since in theory the outgoing DSPCs can be controlled, configured and confirmed by the user. But that is not going to happen anytime soon. In lieu of that one can only do (repeated) tests of ones ingress packet markings and if those make sense avoid the DSCP zeroing/squashing.

Best Regards

A question, before I actively try this out: what is the rationale of setting 90% of the actual uplink/downlink? (I have 100Mbit up/50 down, so I guess I should put in 90/45… the real question is why is this needed).

First let me start with a quick reminder how sqm-scripts achieves its goal of keeping the latency under load increase (aka bufferbloat) und control. The typical problem is that some network components have buffers that are not well matched with either their ingress or egress bandwidth. Say at 1Mbps it takes one second to empty a buffer of size 1Mb (Megabit = 10^6 bits), while at 100Mbps that same buffer will be emptied in 10 milliseconds. Especially unmanaged buffers tend to be sub-optimally sized for almost all traffic. This problem becomes especially dire at network nodes that have asymmetrical ingress and egress bandwidth. E.g. a DSLAM/MSAN might talk to the ISP network via Gigabitethernet but say only with 16Mbps ADSL to the customer, no single statically sized non-managed buffer will work well for both directions, and since the size of the buffer, if too small, will reduce total achievable bandwidth it is quite common for networking equipment to be over-buffered. This is also true for home equipment (CPE customer premise equipment), think a 100Mbps capable VDSL modem connected to a 376Kbps ADSL1 line… Now what sqm-scripts does is to create a managed buffer that will not show this effect.
Let’s look at ingress (internet download direction) and egress separately (internet upload direction).
For egress the goal typically is not to fill the CPE’s (csable modem or XDSL-modem) over-sized and under-managed buffers. To achieve that goal it is sufficient to never send more than what can be sent over the uplink, that means after accounting for all per packet- and encapsulation overhead it is possible to set sqm’s egress shaper to 100 of the relevant egress bandwidth. Now figuring out that relevant egress bandwidth exactly is quite involved and sometimes even impossible, typically the overheads and encapsulation do add up to <10% so 90% of line rste often works, and the recommended 90% of achievable good put as measured by speedtests is almost guarateed to work well.
For ingress things get a bit more complicated. The challenge here is that the real bottleneck link/buffer typically is not after sqm’s controlled buffers, but before, so any packet reaching sqm will already have passed the true bottleneck link. And that means that if packets rush in too fast they will fill up the upstream DSLAM/MSAN?CMTS buffers that we can not really control well. It turns out that if we say set the ingress shaper to 10% of the link we almost never see the upstream buffer filling up, and if we set the ingress shapers to 100% we basically have not control over buffer bloat. Empirically shaping ingress to 85-90% works relatively well. But please note that even with those shaper settings one can occasionally still run into situations where the upstream buffers fill. So in essence in the download direction shaping is not really that strict and more approximate (mind you it still works pretty well). And that means that you simply need to figure out which trade-off between bufferbloat-avoidance and bandwidth sacrifice you are comfortable with. So do not take 90% as Gospel, but rather an initial value to use as the starting point of a few iterative tests to select the shaper percentage that you personally are happy with given your personal evaluation of the acceptable latency under load increase.

So hopefully that was not too long.

Tl;dr: shaping to 90% of link/contract guaranteed bandwidths simply works well for many users. Egress shaping with proper accounting for encapsulation and overheads works well up to 100%, ingress shaping works better at lower shaping rates.

Please note that in most cases proper accounting for the link layer characteristics will make the shaper work better and will allow to set the shaper closer to the real bottleneck Best Re.

Best Regards

5 Likes

hello, i am very lost when reading this forum topic. although im trying to catch on to this whole cake thing. Now, im looking to change my qdisc to Cake. although it is not listing in my drop down menu. so i read on this topic to find out you have to install cake. where would someone clueless like myself go in the LuCI menu of TurrisOmnia and install cake? that way i may use piece of cake qos. thanks!

I can imagine since there is a lot of jargon present in this topic. You made me read my own post again and notice that it is not clear how to install everything. The line I presented (opkg update && opkg install sqm-scripts luci-app-sqm kmod-sched-cake ) is something you type into a terminal and that isn’t something you can easily do with LuCi. So, how to install cake and SQM QoS by only using LuCi?

  1. Go to System -> Software and click on update lists
  2. In the text field after “Filter” type “kmod-sched-cake” and press "Find package)
  3. Go to the “Available packages” tab listed under “Status”, find kmod-sched-cake and press “install”
  4. Do steps 2 and 3 for luci-app-sqm.
  5. Now, in LuCi, under the tab “Network” there should be a SQM QoS entry and you should be able to follow my guide.

awesome i found the download. now i don’t see it in the drop down still. forgive me but should that be an indicator that i need to reboot the router?

edit: nvm looking on the LuCI interface it asks you to reboot when installing new qdisc. will report back if the changes make any difference for me :slight_smile:

went ahead and did pretty much everything you listed to the T (cake, piece of cake"). although i do not see a difference in what i was doing before… what exactly are these settings suppose to accomplish or do in layman’s terms? thank you for replying earlier as well!

what im doing is trying to fix my connection, in game as after installing turris omnia router in my home i have noticed my connection in game being poor and achieving a higher ping than what i was having before. my connection use to be fine when using a cisco router, and now im starting to notice lag/ping spikes after using the turris omnia as my router . just trying to find a way to fix this and im not quite sure bandwidth prioritization is the way to go here?

Let me chime in here; sqm-scripts goal is to reduce unwanted latency under load increases caused by over-sized and under-managed buffers (aka bufferbloat). The way this works is twofold, 1) we need to make sure that we get the bottleneck of the link under our control, and 2) we use a smart way to manage the buffer space on that artificial bottleneck.

For traffic flowing from the internal network to the internet (colloquially uploads) or egress traffic (from the point of view of the router’s WAN interface) your omnia sits just in front of the bottleneck link (which is typically a cable/docsis link, an ADSL/VDSL link, and for a few lucky ones a fiber link). It is the buffers in the cable/vdsl/fiber “modems” that are often causing unnecessary latency under load that we try to manage. The trick is simply to never allow more data towards the modem than the modem can send upstream in the given time window, which makes sure the modems buffer never fills up and can not introduce unwanted latency. If we manage to account for the full on-the-wire-size of the shaped traffic this will work well with shaping rates set up to 100% of the true botttleneck link bandwidth (but you need to know all the characteristics of your link to pull that off, so it is still recommended to start out with 85 to 95% of link bandwidth…)

For traffic flowing from the internet into the internal network (colloquially downloads) or ingress traffic (from the WAN interface’s vantage point) things are slightly more complicated. The true bottleneck link sits upstream of us, so we are at the wrong end of it; the mismanaged buffer are located in upstream devices (like the CMTS, MSAN/DSLAM, or even higher level elements with instantiated traffic shaping). So any traffic coming in too fast will overrun that upstream buffers and cause the phenotype of bufferbloat, all we can do is to create an artificial bottleneck on our side of the link (and that is what sqm-scripts does). The slower that artificial bottleneck is compare to the true bottleneck the more effective and snappy our ingress de-bloating is going to work. That in essence means that one needs to test with a few different shaper settings to figure out the trade-off between bandwidth sacrificied and snappiness onder load retained one likes, the recommended 85-90% of true bottleneck bandwidth simply are value ranges that seem to work well enough for a number of people.

Once this is done sqm-script relies on fq_codel or cake to manage the local buffers well. Both as usually configured will try to separate the traffic into separate independent flows, with a flow ideally representing all the traffic from say internal machine A, port N to external machine B port M. That way sqm can give individual flows independent slow-down signals (either by dropping a packet or marking it according to the explicit congestion notification (ECN) method; that way sqm scripts will try to make sure that no flow gets more than its fair share of the link bandwidth. Fq_codel and cake also make sure that the queue for each of the flows never exceeds a certain threshold (both do allow occasional transient excursions).
Let’s call this per-flow-fairness for this post.

Okay, with that overview of of the way, let’s try to get to what cake brings to the table in addition to that. Typically home routers need to perform network address translation (NAT) since IPv4 Addresses are a scarce resource and most ISP will only give out one IPv4 address per customer, NAT essentially is a way of sharing that address between all your internal networks’ computers. A consequence of this NAT is that packets arriving at the WAN interface will always only have the router’ external IPv4 address. Now cake, if run on the computer that performs the NAT, has a way to inquire the kernel about which internal address:port tuple corresponds to a given NATed external address:port tuple. This functionality is controlled by the “nat” keyword that @iddqd recommends to set in his instructions.
And now we get the next trick that cake can do for us: higher level flow isolation. For example cake, unlike fq_codel, can actually first fairly share bandwidth between all active (with nat internal) IP addresses and will also give the above described per-flow-fairness for all flows going to (or) from a given address. The dual-XXXhost options control which address cake is looking at, src for src address and dst for destination address. If used as described in the first posting cake will essentially isolate traffic by internal host IP.

That mode should n essence isolate your gaming traffic from computer A from all the traffic from the other computers in your internal network. That is, there are limits to this, as all computers will be served too many active internal IP addresses will still introduce potentially unwanted latency to your gaming traffic.

sqm-scripts also allows to use DSCP markings to sort packets into different priority bins, which might speed things up a bit, but it is quite difficult to actually mark egress traffic accordingly (only few applications allow to specify the DSCP markings for outgoing traffic) and actually nasty to do the same for incoming traffic (also often ISPs manipulate DSCPs for their own purposes, so there is not even a guarantee that incoming traffic has meaningful DSCP markings; the option is to set those markings by script before packets are passed to cake but again that is quite involved and tricky). If you should want to try that select layer_cake.qos instead of piece_of_cake.qos.

My recommendation still would be first to use piece_of_cake and try to make sure that it is properly configured and operational and only if this empirically is shown to be insufficient I would try to go for explicit markings.

Best Regards

P.S.: Discourse just noticed that I supply 31% of this thread and asked me whether I would want to give others to possibility to voice their opinions. So @iddqd if you feel I am intruding into your thread, please let me know and I will try to cut back a bit.

6 Likes

@moeller0, you are definitely not intruding. You are helping people and even helping me understand the technical intricacies of cake and SQM QoS. You are much appreciated!

1 Like

majority of that went over my head, although i appreciate the time you took to respond. wish i knew what you guys were talking about that way i could use this router to its full potential. i initially bought it because they spoke of security highly. but i have no idea how to configure anything in the LuCI interface so how can i even tap into the security features it offers lol. on top of that i notice my connection becoming lesser of that to my previous router and idk why :frowning:
boy oh boy what did i get myself into with this thing lol

Sorry, maybe I should have started by recommending to read https://lede-project.org/docs/howto/sqm which hopefully helps a bit in understanding the sqm-scripts configuration. But maybe I should just stop to overwhelm you with unrequested information and simply wait for explicit questions, which I am happy to reply to :wink:

i do have a question, when i have my ethernet cable hooked up to my computer, and run a speed test im only getting about 50/60 mb download when i have a 100 mb connection. but on wifi im getting 100+ on speed test. and this has been this way out the box, as soon as i installed the turris in my home… now when i turned on that SQM my speed test reads about 70/80. i do not understand why this is

Could you please post the result of executing the following on the command line* of your omnia:

cat /etc/config/sqm
tc -d qdisc

I will probably ask for more outputs later, but let’s try this step by step…
The first will show the exact configuration you are using and the second will give an overview of installed shapers, both are pieces of information to get started in finding the cause of your issues, I hope.

Best Regards

*) You might need to install putty (if you are using windows, see http://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html) and then connect to 192.168.1.1 (assuming you did not renumber your omnia from the default); use root as username and give the password you use to log into the omnia’s web interface. On Macosx or Linux simply open a terminal window and type “ssh root@192.168.1.1”

thank you for replying, i have since disconnected the turris omnia and reconnected my old router… connection has improved and ping is stable and low as it should be. not planning on keeping omnia but thank you guys for your advice and input. sucks its not so user friendly for someone like me but hopefully what was mentioned by the users and or mods can be useful to others who may be more familiar with how to intricately set everything up. have been going back and forth with tech support for weeks and barely found this forum last weekend. but thank you and appreciate the time of those who shared their input but this router just isn’t for me. cheers !

The LEDE manual at: https://lede-project.org/docs/howto/sqm suggests to use link layer adaptation if you are behind a docsis cable modem. But the first post doesn’t. That is use “Ethernet with overhead” and then set 28 bytes.
But, but, if I set this all I do go from a rating of F on bufferbloat to A. Although only after a reboot. :slight_smile: