[TOS6.0] Bottleneck of circa 9,5Mbit/s on egress traffic only

Software SW bugs discussion
#1

Hi,

I am pulling the hair trying to understand why I have upload speed limit of around Upload: 9.40 Mbit/s on device itself and devices behind AP. My network is not so complex but I use some VLANS. I have two devices; two Omnias one configured as main router and second one as dumb AP. On my main device (Omnia router) I am able to somewhat saturate the FTTH connection I have from ISP:

root@router:~# speedtest-cli --simple --secure
Ping: 15.074 ms
Download: 373.01 Mbit/s
Upload: 172.42 Mbit/s

Lets assume its as in the contract 400Mb/200Mb.

But on my other Omnia (the Access Point Omnia and devices connected to it) this looks like this:

root@ap:~# speedtest-cli --simple --secure
Ping: 24.601 ms
Download: 345.70 Mbit/s
Upload: 9.55 Mbit/s

Only egress direction is somehow limited to circa 10Mbit/s. And thats the weirdest part. Its the same for clients connected to AP via WiFi and via cable. I am using HBL TOS6.0 on both of them to be able to configure trunk port with LuCi.

Trunk with few VLANS on the devices; router’s tagged port is configured on lan4 and on the access point on eth2 (eth2/wan port added to br-lan to use it as dumb AP)

I checked negotiated speed between devices its Full duplex gigabit. There were 2 FCS_errors so I already ordered new ethernet patch-cords shielded cat6 cuz it might be the cables. But I doubt that.

I am willing to provide detailed configuration but could somebody point me into direction on what that might be. I am out of ideas here. Thanks!

P.S. There is one cake SQM configured on router WAN port to ged rid of bufferbloat and thats it.

#2

I thought there was a post stating that TOS6 has still issues with VLANs that result in performance drops? (Even though I have no VLANs in use that issue so far kept me from switching, so I have no first-hand TOS6 experience).

Maybe also post the output of tc -s qdisc and tc -d qdisc from both omnias just to double check that there is no stray unaccounted for traffic shaping going on?

#3
router 
root@router:~# tc -s qdisc
qdisc noqueue 0: dev lo root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc mq 0: dev eth0 root
 Sent 1190151081 bytes 1120981 pkt (dropped 0, overlimits 0 requeues 51)
 backlog 0b 0p requeues 51
qdisc fq_codel 0: dev eth0 parent :8 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :7 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :6 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :5 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :4 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :3 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :2 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 1190151081 bytes 1120981 pkt (dropped 0, overlimits 0 requeues 51)
 backlog 0b 0p requeues 51
  maxpacket 1496 drop_overlimit 0 new_flow_count 60 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc mq 0: dev eth1 root
 Sent 6456027717 bytes 5618267 pkt (dropped 0, overlimits 0 requeues 304)
 backlog 0b 0p requeues 304
qdisc fq_codel 0: dev eth1 parent :8 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth1 parent :7 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth1 parent :6 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth1 parent :5 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth1 parent :4 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth1 parent :3 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth1 parent :2 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth1 parent :1 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 6456027717 bytes 5618267 pkt (dropped 0, overlimits 0 requeues 304)
 backlog 0b 0p requeues 304
  maxpacket 12176 drop_overlimit 0 new_flow_count 933 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc mq 0: dev eth2 root
 Sent 1789099671 bytes 4776844 pkt (dropped 0, overlimits 0 requeues 148)
 backlog 0b 0p requeues 148
qdisc fq_codel 0: dev eth2 parent :8 limit 10240p flows 1024 quantum 1514 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth2 parent :7 limit 10240p flows 1024 quantum 1514 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth2 parent :6 limit 10240p flows 1024 quantum 1514 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth2 parent :5 limit 10240p flows 1024 quantum 1514 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth2 parent :4 limit 10240p flows 1024 quantum 1514 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth2 parent :3 limit 10240p flows 1024 quantum 1514 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth2 parent :2 limit 10240p flows 1024 quantum 1514 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth2 parent :1 limit 10240p flows 1024 quantum 1514 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 1789099671 bytes 4776844 pkt (dropped 0, overlimits 0 requeues 148)
 backlog 0b 0p requeues 148
  maxpacket 1496 drop_overlimit 0 new_flow_count 213 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev lan0 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev lan1 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev lan2 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev lan3 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev lan4 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-lan root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-lan.1337 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-lan.10 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-lan.20 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-lan.30 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth2.101 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev home0 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev home1 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev guest0 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev iot0 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev guest1 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev iot1 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc fq_codel 0: dev dns0 root refcnt 2 limit 10240p flows 1024 quantum 1500
 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 2063887 bytes 8606 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc cake 8009: dev pppoe-wan root refcnt 2 bandwidth 204800Kbit diffserv4 d
ual-srchost nat nowash ack-filter split-gso rtt 50ms noatm overhead 46
 Sent 640451156 bytes 855994 pkt (dropped 4504, overlimits 728355 requeues 0)

 backlog 0b 0p requeues 0
 memory used: 328384b of 5000Kb
 capacity estimate: 204800Kbit
 min/max network layer size:           36 /    1474
 min/max overhead-adjusted size:       82 /    1520
 average network hdr offset:            0

                   Bulk  Best Effort        Video        Voice
  thresh      12800Kbit   204800Kbit   102400Kbit    51200Kbit
  target          2.5ms        2.5ms        2.5ms        2.5ms
  interval         50ms         50ms         50ms         50ms
  pk_delay          0us          9us         14us         12us
  av_delay          0us          4us          4us          2us
  sp_delay          0us          3us          3us          2us
  backlog            0b           0b           0b           0b
  pkts                0       860015          256          227
  bytes               0    640556732        19456        55260
  way_inds            0         4281            0            0
  way_miss            0         6050          256           20
  way_cols            0            0            0            0
  drops               0            0            0            0
  marks               0            0            0            0
  ack_drop            0         4504            0            0
  sp_flows            0            3            1            0
  bk_flows            0            1            0            0
  un_flows            0            0            0            0
  max_len             0         1474           76          576
  quantum           390         1514         1514         1514

qdisc ingress ffff: dev pppoe-wan parent ffff:fff1 ----------------
 Sent 2572943035 bytes 2023218 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev docker0 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc fq_codel 0: dev tun_turris root refcnt 2 limit 10240p flows 1024 quantu
m 1500 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 750695 bytes 2971 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc cake 800a: dev ifb4pppoe-wan root refcnt 2 bandwidth 409600Kbit diffser
v4 dual-dsthost nat wash ingress no-ack-filter split-gso rtt 50ms noatm overh
ead 46
 Sent 2569274452 bytes 2020723 pkt (dropped 2495, overlimits 3073258 requeues
 0)
 backlog 0b 0p requeues 0
 memory used: 1860Kb of 10000Kb
 capacity estimate: 409600Kbit
 min/max network layer size:           30 /    1474
 min/max overhead-adjusted size:       76 /    1520
 average network hdr offset:            0

                   Bulk  Best Effort        Video        Voice
  thresh      25600Kbit   409600Kbit   204800Kbit   102400Kbit
  target          2.5ms        2.5ms        2.5ms        2.5ms
  interval         50ms         50ms         50ms         50ms
  pk_delay         36us         15us          0us          7us
  av_delay          7us          7us          0us          0us
  sp_delay          3us          3us          0us          0us
  backlog            0b           0b           0b           0b
  pkts            31565      1991613            0           40
  bytes        31218848   2541716647            0         7540
  way_inds            0        17978            0            0
  way_miss         1204         8296            0           34
  way_cols            0            0            0            0
  drops               0         2495            0            0
  marks               0            0            0            0
  ack_drop            0            0            0            0
  sp_flows            0            3            0            1
  bk_flows            0            0            0            0
  un_flows            0            0            0            0
  max_len          1474         1474            0          576
  quantum           781         1514         1514         1514

root@router:~# tc -d qdisc                                                  
qdisc noqueue 0: dev lo root refcnt 2
qdisc mq 0: dev eth0 root
qdisc fq_codel 0: dev eth0 parent :8 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc fq_codel 0: dev eth0 parent :7 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc fq_codel 0: dev eth0 parent :6 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc fq_codel 0: dev eth0 parent :5 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc fq_codel 0: dev eth0 parent :4 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc fq_codel 0: dev eth0 parent :3 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc fq_codel 0: dev eth0 parent :2 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc mq 0: dev eth1 root
qdisc fq_codel 0: dev eth1 parent :8 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc fq_codel 0: dev eth1 parent :7 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc fq_codel 0: dev eth1 parent :6 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc fq_codel 0: dev eth1 parent :5 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc fq_codel 0: dev eth1 parent :4 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc fq_codel 0: dev eth1 parent :3 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc fq_codel 0: dev eth1 parent :2 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc fq_codel 0: dev eth1 parent :1 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc mq 0: dev eth2 root
qdisc fq_codel 0: dev eth2 parent :8 limit 10240p flows 1024 quantum 1514 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc fq_codel 0: dev eth2 parent :7 limit 10240p flows 1024 quantum 1514 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc fq_codel 0: dev eth2 parent :6 limit 10240p flows 1024 quantum 1514 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc fq_codel 0: dev eth2 parent :5 limit 10240p flows 1024 quantum 1514 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc fq_codel 0: dev eth2 parent :4 limit 10240p flows 1024 quantum 1514 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc fq_codel 0: dev eth2 parent :3 limit 10240p flows 1024 quantum 1514 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc fq_codel 0: dev eth2 parent :2 limit 10240p flows 1024 quantum 1514 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc fq_codel 0: dev eth2 parent :1 limit 10240p flows 1024 quantum 1514 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc noqueue 0: dev lan0 root refcnt 2
qdisc noqueue 0: dev lan1 root refcnt 2
qdisc noqueue 0: dev lan2 root refcnt 2
qdisc noqueue 0: dev lan3 root refcnt 2
qdisc noqueue 0: dev lan4 root refcnt 2
qdisc noqueue 0: dev br-lan root refcnt 2
qdisc noqueue 0: dev br-lan.1337 root refcnt 2
qdisc noqueue 0: dev br-lan.10 root refcnt 2
qdisc noqueue 0: dev br-lan.20 root refcnt 2
qdisc noqueue 0: dev br-lan.30 root refcnt 2
qdisc noqueue 0: dev eth2.101 root refcnt 2
qdisc noqueue 0: dev home0 root refcnt 2
qdisc noqueue 0: dev home1 root refcnt 2
qdisc noqueue 0: dev guest0 root refcnt 2
qdisc noqueue 0: dev iot0 root refcnt 2
qdisc noqueue 0: dev guest1 root refcnt 2
qdisc noqueue 0: dev iot1 root refcnt 2
qdisc fq_codel 0: dev dns0 root refcnt 2 limit 10240p flows 1024 quantum 1500
 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc cake 8009: dev pppoe-wan root refcnt 2 bandwidth 204800Kbit diffserv4 d
ual-srchost nat nowash ack-filter split-gso rtt 50ms noatm overhead 46
qdisc ingress ffff: dev pppoe-wan parent ffff:fff1 ----------------
qdisc noqueue 0: dev docker0 root refcnt 2
qdisc fq_codel 0: dev tun_turris root refcnt 2 limit 10240p flows 1024 quantu
m 1500 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc cake 800a: dev ifb4pppoe-wan root refcnt 2 bandwidth 409600Kbit diffser
v4 dual-dsthost nat wash ingress no-ack-filter split-gso rtt 50ms noatm overh
ead 46
#4
access point 
root@ap:~# tc -s qdisc
qdisc noqueue 0: dev lo root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc mq 0: dev eth0 root
 Sent 1808083 bytes 21907 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc fq_codel 0: dev eth0 parent :8 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :7 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :6 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :5 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :4 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :3 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :2 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 1808083 bytes 21907 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 90 drop_overlimit 0 new_flow_count 1 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc mq 0: dev eth1 root
 Sent 15544004 bytes 24553 pkt (dropped 0, overlimits 0 requeues 14)
 backlog 0b 0p requeues 14
qdisc fq_codel 0: dev eth1 parent :8 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth1 parent :7 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth1 parent :6 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth1 parent :5 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth1 parent :4 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth1 parent :3 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth1 parent :2 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth1 parent :1 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 15544004 bytes 24553 pkt (dropped 0, overlimits 0 requeues 14)
 backlog 0b 0p requeues 14
  maxpacket 4488 drop_overlimit 0 new_flow_count 321 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc mq 0: dev eth2 root
 Sent 679624 bytes 3275 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc fq_codel 0: dev eth2 parent :8 limit 10240p flows 1024 quantum 1514 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth2 parent :7 limit 10240p flows 1024 quantum 1514 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth2 parent :6 limit 10240p flows 1024 quantum 1514 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth2 parent :5 limit 10240p flows 1024 quantum 1514 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth2 parent :4 limit 10240p flows 1024 quantum 1514 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth2 parent :3 limit 10240p flows 1024 quantum 1514 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth2 parent :2 limit 10240p flows 1024 quantum 1514 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth2 parent :1 limit 10240p flows 1024 quantum 1514 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 679624 bytes 3275 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 1394 drop_overlimit 0 new_flow_count 10 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev lan0 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev lan1 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev lan2 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev lan3 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev lan4 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-lan root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-lan.30 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-lan.10 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-lan.1337 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-lan.20 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev home1 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev home0 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev guest1 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev guest0 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev iot0 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev iot1 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev docker0 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
root@ap:~# tc -d qdisc                                                      
qdisc noqueue 0: dev lo root refcnt 2
qdisc mq 0: dev eth0 root
qdisc fq_codel 0: dev eth0 parent :8 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc fq_codel 0: dev eth0 parent :7 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc fq_codel 0: dev eth0 parent :6 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc fq_codel 0: dev eth0 parent :5 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc fq_codel 0: dev eth0 parent :4 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc fq_codel 0: dev eth0 parent :3 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc fq_codel 0: dev eth0 parent :2 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc mq 0: dev eth1 root
qdisc fq_codel 0: dev eth1 parent :8 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc fq_codel 0: dev eth1 parent :7 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc fq_codel 0: dev eth1 parent :6 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc fq_codel 0: dev eth1 parent :5 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc fq_codel 0: dev eth1 parent :4 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc fq_codel 0: dev eth1 parent :3 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc fq_codel 0: dev eth1 parent :2 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc fq_codel 0: dev eth1 parent :1 limit 10240p flows 1024 quantum 1522 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc mq 0: dev eth2 root
qdisc fq_codel 0: dev eth2 parent :8 limit 10240p flows 1024 quantum 1514 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc fq_codel 0: dev eth2 parent :7 limit 10240p flows 1024 quantum 1514 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc fq_codel 0: dev eth2 parent :6 limit 10240p flows 1024 quantum 1514 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc fq_codel 0: dev eth2 parent :5 limit 10240p flows 1024 quantum 1514 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc fq_codel 0: dev eth2 parent :4 limit 10240p flows 1024 quantum 1514 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc fq_codel 0: dev eth2 parent :3 limit 10240p flows 1024 quantum 1514 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc fq_codel 0: dev eth2 parent :2 limit 10240p flows 1024 quantum 1514 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc fq_codel 0: dev eth2 parent :1 limit 10240p flows 1024 quantum 1514 tar
get 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc noqueue 0: dev lan0 root refcnt 2
qdisc noqueue 0: dev lan1 root refcnt 2
qdisc noqueue 0: dev lan2 root refcnt 2
qdisc noqueue 0: dev lan3 root refcnt 2
qdisc noqueue 0: dev lan4 root refcnt 2
qdisc noqueue 0: dev br-lan root refcnt 2
qdisc noqueue 0: dev br-lan.30 root refcnt 2
qdisc noqueue 0: dev br-lan.10 root refcnt 2
qdisc noqueue 0: dev br-lan.1337 root refcnt 2
qdisc noqueue 0: dev br-lan.20 root refcnt 2
qdisc noqueue 0: dev home1 root refcnt 2
qdisc noqueue 0: dev home0 root refcnt 2
qdisc noqueue 0: dev guest1 root refcnt 2
qdisc noqueue 0: dev guest0 root refcnt 2
qdisc noqueue 0: dev iot0 root refcnt 2
qdisc noqueue 0: dev iot1 root refcnt 2
qdisc noqueue 0: dev docker0 root refcnt 2
#5

Thanks, no unexpected traffic shaper apparent, as expected, but it would have been easy to remedy. Sorry for the distraction.

#6

I just made a few measurements and somehow the upload on AP was good around 160Mbits for some time but after a reboot its back to 9,5Mbit limit. On the router directly connected to ISP the problem doesn’t exist. Weird stuff

#7

Can you try changing tagged port from lan4 to any other, I suspect 2nd eth(eth0/lan4) has problem with vlan, afaik openwrt not officially supporing 2nd eth.

#8

Thanks for the idea and interest I will try that in a while.

Edit: I used lan3 as the trunk port but still same problem:

Upload: 9.48 Mbit/s

Edit2: But you gave me an idea for test. I created tagged vlan interface on my laptop and connected to that trunk port and the speed was as expected. So it might give some idea that is indeed in my access point’s (so Omnia) VLAN implementation.

#9

I just replaced cables with cat 6 shielded patch-cords. The FCS errors are gone from the interface but still speed is around 9,50Mbit/s…

#10

@hagrid will you be able to reproduce it in lab env?

#11

Sure. Can you send me a drawing of your network infrastructure and configurations of network on both devices?

#12

I will do that when I am back on site.

Edit: @hagrid I’ve send it to you via PM