You may want to put this in the doc wiki once you’re done with all the bits.
Thanks! Been waiting for a simple TO cake guide.
@einar great idea! I will make a wiki entry when the guide is finalized.
Finished writing the manual configuration part. Feedback is welcomed!
Great work!
I am not sure whether the old advise to use 8 for VDSL overhead still applies. Recent cake, has learned to de-account for those parts of an ethernet header the kernel adds automatically. And that means those 14 bytes needs to be accounted again, and if sqm is instantiated on a pppoe interface we also need to account for the 8 byte ppp/pppoe overhead, so for my VDSL2/PPPoE/VLAN link I use overhead 34 on the pppoe-wan interface:
PPP: 2 Byte PPP + 6 Byte PPPoE = 8
ETHERNET: 4 Byte Frame Check Sequence (FCS) + 6 (dest MAC) + 6 (src MAC) + 2 (ethertype) + 4 Byte VLAN = 22
VDSL2PTM: 1 Byte Start of Frame (S), 1 Byte End of Frame (Ck), 2 Byte TC-CRC (PTM-FCS), = 4
Total: = 8 +22 + 4 = 34
The old recommendation of 8 came from instantiating sqm on an “raw” ethN interface with silently accounted 14 bytes and assuming neither VLAN nor PPP/PPoE: (14) + 4 Byte Frame Check Sequence (FCS) + 4 Byte VDLS2/PTM = (14) + 8 = effective 22 Byte. Cake’s ability to undo the silent overhead accounting makes things much cleaner to document and explain.
Please note that “tc -s qdisc show dev $YOURINTERFACE” will (if using fq_codel at least) report a variable maxpacket, that shows the size of a packet as seen on that interface (that number actually already adds the sqm overhead). To give an example: with overhead 34 on ethN one can expect to see “maxpacket 1534”, but with the same settings instantiated on a ppp interface on top of the same ethN (say pppoe-wan) one only will see “maxpacket 1526”. The reason for that is quite simple, the pppoe interfaces do not “see” the 8 byte PPPoE overhead, so the maximum packet size reaching that interface will be 1492 and 1492+34=1526. And this is the reason here to specify 34 in the first place for pppoe-wan, instead of the true overhead of 26 we would spacify on ethN, weneed to account for the PPPoE overhead “manually”
.
1 Like
i tried using small packets to get the most amplification for the overhead-error
like iperf -c <ip> -b1m -P10 -l100 (iperf2)
with cake limit at 80% line rate (8Mbit/s)
i had to set overhead 68 (on pppoe-wan) to get steady ping-results.
verified with 50byte packets -> no change
might there be some error in the code that makes up for the “double accounting”?
Also: also on dtag vdsl2/pppoe/vlan on the omnia via bridged vmg1312
A2: eqdisc_opts 'nat dual-srchost diffserv4 rtt 60ms ptm overhead 68'
1 Like
Hi Fuller,
if you believe the problem to be related to cake’s overhead accounting feel free to set the $LLAM to “tc_stab”, that way you can also configure the MPU, the minimal size of a packet, which in all likelihood is >= 64 bytes on ethernet. On pppoe-wan the kernel will not automatically add 14 bytes so you should be okay. The MPU question is more relevant, but also massively unclear, VDSL2 can send frames <64B unlike ethernet, but whether it actually does or does not escapes my understanding. I assume that in most conditions if passed a padded ethernet frame VDSL2 will simply pass on that frame including padding, as otherwise it would need to calculate a new FCS at the vdsl-sender and the vdsl-receiver would need to undo this and padd the packets again, in essence that indicates that all transports using ethernet frames might want mpu set to 64, bytes.
Regarding using cake’s “ptm” setting, please don’t. This is a misfeature, rather go and statically discount your bandwidth by 100-100*64/65 = 1.54% instead of incurring additional computations at each packets that also seem to be less precise.
BUT on DTAG links your problem might actually not be the VDSL2 leg, but rather the BRAS/BNG level shaper that DTAG employs. You would need to configure your sqm-instance to do the right thing for that shaper (as far as I know it uses double VLAN tagging, but not sure about the FCS) also that shaper’s actual bandwidth is not communicated to end users so it requires careful testing, if you use your modems sync bandwidth you certainly will shape too high which will show especially when testing with small packets…
Best Regards
will do
ok, does not seem to make a difference …
i specifically set egress-bw to 80% to avoid running into this
that is, if the overhead calculation would be correct.
sry i did not make clear that with the tested overhead 68 i can set it to 9,8Mbit/s with no effect on latency.
it just seems odd to be exacly double of what your calculation yields.
also the test is unidirectional, udp, v4, all offloads on omnia are disabled.
cheers 
If the overhead is set too high, this effectively just decreases the shaper bandwidth (in a packetsize dependent way, but when testing with fixed packet sizes like you do you can ignore that aspect). So setting the overhead higher will allow you to set a higher shaper bandwidth as due to overhead accounting sqm/cake will not really admit that many bytes to the wire. If the 9.8Mbps is actually measured goodput, then my argument does not apply ;).
BTW, on a DTAG VDSL2@BRAS link (50/10Mbps) I have set egress shaping to 9545Kbps of a sync of 10048K as otherwise the BRAS shaper would cause additional bufferbloat.
According to http://www.onlinekosten.de/forum/showthread.php?p=2380059#post2380059 DTAG does throttle @BNG to Download: 48.400 Kibps 48400 1024/1000 = 49561.6Kbps (expected: 51.300 kbit/s) Upload: 9.470 Kibps 94701024/1000 = 9697.28Kbps (expected: 10.000 kbit/s) (this is for BNG not BRAS so I can not directly check that) In that light I expect an egress shaper with correct per-packet-overead configuration to still show signs of bufferbloat at shaper rates > 9470. I do have no independent confirmation of the DTAG shaper rates, so all of the above might be logically wrong and/or besides the point.
1 Like
am aware of that. again the testing was done at “safe” speed of 8Mbit/s (cake-limit that is)
i was not!
at around overhead 50 the flood with 100byte (udp-payload size) packets does not clogg the uplink, but the 50byte test does kill it completely. (at “safe” speed)
at overhead 68 there is no difference by packet size anymore and i then probed max-tx-speed, which led to 9,8Mbit/s (cake-limit).
did not even think about tc_stab vs cake linklayer adaption thou …
this got out of hand quickly 
Hi fuller,
oops, thanks for the corrections, I clearly did not understand where the 50 bytes came in (iperf2 question those 50 bytes are iperf payload or total packet size before ethernet padding?)
Testing tc_stab versus cake as $LLAM would be quite an interesting test (one reason why sqm-scripts exposes multiple methods here is exactly so they can be checked against each other). Should you find time to perform this test I would be grateful if you could post it here.
ok, with linklayer_adaptation_mechanism 'tc_stab' the full overhead is “seen” by cake.
with this and setting tcMPU '64' and overhead '34' sqm-scripts seem to get it “right” for my connection.
the qdisc now show as:
... diffserv4 dual-dsthost nat rtt 60.0ms noatm overhead 14
and ... overhead 26 for egress
max_len 1536 for both
after re-realising that the “50 byte” udp-payload expands to a 78 byte ip-packet (i.e. i will likely not run into padding), i tried with iperf -l10
and am now equally unsure, as i see a different effect:
something upstream of the omnia consistently starts dropping packets above 11kpps, latency and droprate rise steadily and the line reconnects at around 20kpps.
so, my previous results may also reflect that …
more research is obviously mandated 
This seems odd, if you explicitly requested 34, why are you a) not getting 34 and b) why fifferent values of !34 for both directions? What is your /etc/config/sqm looking like?
with cake-overhead its always max_len 1492
i now use
config queue
option interface 'pppoe-wan'
option enabled '1'
option script 'piece_of_cake.qos'
option qdisc_advanced '1'
option squash_dscp '0'
option squash_ingress '0'
option ingress_ecn 'ECN'
option egress_ecn 'NOECN'
option qdisc 'cake'
option qdisc_really_really_advanced '1'
option iqdisc_opts 'nat dual-dsthost diffserv4 rtt 60ms'
option eqdisc_opts 'nat dual-srchost diffserv4 rtt 60ms'
option download '47000'
option debug_logging '0'
option verbosity '10'
option linklayer 'ethernet'
option linklayer_advanced '1'
option upload '9000'
option overhead '34'
option tcMTU '2047'
option tcTSIZE '128'
option tcMPU '64'
option linklayer_adaptation_mechanism 'tc_stab'
this manages a flood with ack-size packets iperf ... -l12 -b1m -P10 with little loss, whereas i see massive ping-loss when using cake-overhead .
I believe cake’s overhead accounting does not change skb->len (which tc-stab does) so the maxlen reports the “payload” length, So in your case that is just 1500 - 8 bytes PPPoE overhead (which I am sure you knew )
The config seems fine, except maybe you should consider to set squash_dscp unless you trust your incoming packets to be marked correctly…
Argh, there was a small time window when the was some API breakage (sch_cake and iproute2’s need to match) that actually might affect parameter passing from tc to cake, this might be a symptom of that. I guess I will need to run a few tests to figure out whether I can reproduce the failure…
If somebody more have issues with SQM + 6in4 (IPv6 tunnel): ethtool -K eth1 gro off
More info:
Thanks for the great how to. I now am happily up and running with cake.
One question: I have a Ooma Telo VoIP box plugged into eth2 / lan4 on the TO that connects to the internet. Is there anything further that I need to configure so as to ensure lowest latency for my VoIP packets?
Assuming your VoIP box uses standard DSCP markings you might want to use layer_cake.qos instead of the recomended piece_of_cake.qos. Layer_cake will try to expedite some DSCP markings over others and typical VoIP markings should be in the set of prioretized DSCPs…
Best Regards
1 Like
@moeller0, i really knew nothing about setting up any flavor of cake. And honestly I do not yet grasp the distinction between simple cake, layer_cake.qos and piece_of_cake.qos.
I was immensely grateful for the work you and @iddqd put into documenting how to setup piece_of_cake.qos. If I attempt to setup layer_cake.qos how different will it be? Are there any important gotchas of which I need to be aware?
Oh, thank @iddqd, he did all the heavy lifting, I was just playing the “peanut gallery”. But all you need to do to get layer-cake instead of piece_of_cake is:
"Select layer_cake.qos under Queue setup script."
if you use the GUI or with the editor approach:
replace "option script ‘piece_of_cake.qos’ " with "option script ‘layer_cake.qos’ ". That should do the trick, assuming you have configured piece_of_cake.qos as described above.
Best Regards