Only one CPU core is used during download/upload. I have a 1 Gbit connection and this causes the CPU to go to 100%, but only one of the cores. The other one is idling at 5-8%…
Why aren’t both cores utilized during download/upload?
Here is a video of htop during download:
You do not show which process is maxing the CPU. Is it Pakon or SQM or something else? Also, go to Setup and check the option to show detailed CPU usage. That will then show grey parts of the progress bar where the CPU is blocked by I/O.
Sorry, where can I find the option to show detailed CPU usage?
htop doesn’t show which process is maxing the CPU, unfortunately.
Likely interrupt and/or softinterrupt processing. Under F2 Setup → Display Options you can check "Detailed CPU time (System/IO-Wait/Hard-IRQ/Soft-IRQ/Steal/Guest) " which will show soft interrupts in pink, I bet these make up a big chunk on the loaded CPU…
Also please post the output of:
cat /proc/interrupts
and
cat /proc/softirqs
to see how these are distributed over the CPUs…
EDIT: fixed unfortunate typos in both commands, it turns out neither interrups nor softirq is the correct name…
I suspect this workload case isn’t worth splitting over the two CPUs. (i.e. it wouldn’t get faster)
You can try (installing and) running program irqbalance. It could help distributing the load between the two CPUs (if it is the I/O-based load).
root@turris:~# cat /proc/interrups
cat: can’t open ‘/proc/interrups’: No such file or directory
root@turris:~# cat /proc/softirq
cat: can’t open ‘/proc/softirq’: No such file or directory
With Detailed CPU time enabled:
This is all weird. Your htop looks too new (see the I/O tab which is a very recent addition). Are you running HBL or some other experimental branch?
Also, I have both /proc/interrupts and /proc/softirq files on my Omnia.
Are you using an Omnia or a Mox?
| Device | Turris Omnia |
|---|---|
| Serial number | 51539647XXX |
| reForis version | 1.4.1 |
| Turris OS version | 6.2.4 |
| Turris OS branch | HBS |
| Kernel version | 5.15.94 |
Sorry this is my typo should be cat /proc/interrupts (note that I initially missed a “t”)…
Same for cat /proc/softirq probably should be cat /proc/softirqs
As I expected one CPU pegged by softirqs, you could try to enable packet steering, but that will by default not do the best possible on dual core CPUs IIRC.
root@turris:~# cat /proc/interrupts
CPU0 CPU1
25: 0 0 GIC-0 27 Edge gt
26: 2331242 2159290 GIC-0 29 Edge twd
27: 0 0 MPIC 5 Level armada_370_xp_per_cpu_tick
28: 0 0 MPIC 3 Level arm-pmu
29: 2160332 0 GIC-0 34 Level mv64xxx_i2c
30: 25 0 GIC-0 44 Level ttyS0
41: 0 0 GIC-0 41 Level f1020300.watchdog
45: 0 0 GIC-0 96 Level f1020300.watchdog
47: 59354740 0 MPIC 10 Level eth1
48: 20452519 0 MPIC 12 Level eth2
49: 0 0 GIC-0 50 Level ehci_hcd:usb1
50: 0 0 GIC-0 51 Level f1090000.crypto
51: 0 0 GIC-0 52 Level f1090000.crypto
52: 0 0 GIC-0 53 Level f10a3800.rtc
53: 0 0 GIC-0 58 Level ahci-mvebu[f10a8000.sata]
54: 38573 0 GIC-0 57 Level mmc0
55: 0 0 GIC-0 48 Level xhci-hcd:usb2
56: 0 0 GIC-0 49 Level xhci-hcd:usb4
58: 2381737 0 GIC-0 61 Level pcie0.0
59: 0 0 MPIC SOC 4 Level pcie0.0
60: 0 0 GIC-0 65 Level pcie1.0
61: 0 0 MPIC SOC 5 Level pcie1.0
62: 0 0 GIC-0 102 Level pcie2.0
63: 0 0 MPIC SOC 15 Level pcie2.0
64: 1 0 mvebu-rp 0 Edge pciehp
65: 1 0 mvebu-rp 0 Edge pciehp
66: 0 0 mvebu-rp 0 Edge pciehp
67: 2 0 GIC-0 54 Level f1060800.xor
68: 2 0 GIC-0 97 Level f1060900.xor
69: 20 0 f1018140.gpio 13 Level mv88e6xxx-f1072004.mdio-mii:10
73: 0 0 mv88e6xxx-g1 3 Edge mv88e6xxx-f1072004.mdio-mii:10-g1-atu-prob
75: 0 0 mv88e6xxx-g1 5 Edge mv88e6xxx-f1072004.mdio-mii:10-g1-vtu-prob
77: 0 20 mv88e6xxx-g1 7 Edge mv88e6xxx-f1072004.mdio-mii:10-g2
79: 0 7 mv88e6xxx-g2 0 Edge mv88e6xxx-1:00
80: 0 0 mv88e6xxx-g2 1 Edge mv88e6xxx-1:01
81: 0 9 mv88e6xxx-g2 2 Edge mv88e6xxx-1:02
82: 0 1 mv88e6xxx-g2 3 Edge mv88e6xxx-1:03
83: 0 3 mv88e6xxx-g2 4 Edge mv88e6xxx-1:04
94: 0 0 mv88e6xxx-g2 15 Edge mv88e6xxx-f1072004.mdio-mii:10-watchdog
95: 1 0 f1018140.gpio 14 Level 8-0071
97: 1583208 0 MPIC MSI 1048576 Edge mt7915e
98: 2381737 0 mvebu-INTx 0 Level ath9k
IPI0: 0 1 CPU wakeup interrupts
IPI1: 0 0 Timer broadcast interrupts
IPI2: 6753 6165 Rescheduling interrupts
IPI3: 414996 1719782 Function call interrupts
IPI4: 0 0 CPU stop interrupts
IPI5: 0 0 IRQ work interrupts
IPI6: 0 0 completion interrupts
Err: 0
root@turris:~# cat /proc/softirqs
CPU0 CPU1
HI: 0 0
TIMER: 879766 204268
NET_TX: 307245 796
NET_RX: 79367464 247331
BLOCK: 8141 7978
IRQ_POLL: 0 0
TASKLET: 5495265 530533
SCHED: 1685374 1420909
HRTIMER: 0 0
RCU: 572918 473446
@moeller0 updated with Preformatted text
Please use the Preformatted text button for computer output or put your text between two rows of three backticks:
```
your text
iiii iiii
```
which will then use fixed width font resulting in the expected akignment
your text
iiii iiii
root@turris:~# cat /proc/interrupts
CPU0 CPU1
25: 0 0 GIC-0 27 Edge gt
26: 2078632 1922058 GIC-0 29 Edge twd
27: 0 0 MPIC 5 Level armada_370_xp_per_cpu_tick
28: 0 0 MPIC 3 Level arm-pmu
29: 1952852 0 GIC-0 34 Level mv64xxx_i2c
30: 25 0 GIC-0 44 Level ttyS0
41: 0 0 GIC-0 41 Level f1020300.watchdog
45: 0 0 GIC-0 96 Level f1020300.watchdog
47: 58111138 0 MPIC 10 Level eth1
48: 19474190 0 MPIC 12 Level eth2
49: 0 0 GIC-0 50 Level ehci_hcd:usb1
50: 0 0 GIC-0 51 Level f1090000.crypto
51: 0 0 GIC-0 52 Level f1090000.crypto
52: 0 0 GIC-0 53 Level f10a3800.rtc
53: 0 0 GIC-0 58 Level ahci-mvebu[f10a8000.sata]
54: 38573 0 GIC-0 57 Level mmc0
55: 0 0 GIC-0 48 Level xhci-hcd:usb2
56: 0 0 GIC-0 49 Level xhci-hcd:usb4
58: 2128249 0 GIC-0 61 Level pcie0.0
59: 0 0 MPIC SOC 4 Level pcie0.0
60: 0 0 GIC-0 65 Level pcie1.0
61: 0 0 MPIC SOC 5 Level pcie1.0
62: 0 0 GIC-0 102 Level pcie2.0
63: 0 0 MPIC SOC 15 Level pcie2.0
64: 1 0 mvebu-rp 0 Edge pciehp
65: 1 0 mvebu-rp 0 Edge pciehp
66: 0 0 mvebu-rp 0 Edge pciehp
67: 2 0 GIC-0 54 Level f1060800.xor
68: 2 0 GIC-0 97 Level f1060900.xor
69: 14 0 f1018140.gpio 13 Level mv88e6xxx-f1072004.mdio-mii:10
73: 0 0 mv88e6xxx-g1 3 Edge mv88e6xxx-f1072004.mdio-mii:10-g1-atu-prob
75: 0 0 mv88e6xxx-g1 5 Edge mv88e6xxx-f1072004.mdio-mii:10-g1-vtu-prob
77: 0 14 mv88e6xxx-g1 7 Edge mv88e6xxx-f1072004.mdio-mii:10-g2
79: 0 1 mv88e6xxx-g2 0 Edge mv88e6xxx-1:00
80: 0 0 mv88e6xxx-g2 1 Edge mv88e6xxx-1:01
81: 0 9 mv88e6xxx-g2 2 Edge mv88e6xxx-1:02
82: 0 1 mv88e6xxx-g2 3 Edge mv88e6xxx-1:03
83: 0 3 mv88e6xxx-g2 4 Edge mv88e6xxx-1:04
94: 0 0 mv88e6xxx-g2 15 Edge mv88e6xxx-f1072004.mdio-mii:10-watchdog
95: 1 0 f1018140.gpio 14 Level 8-0071
97: 1432236 0 MPIC MSI 1048576 Edge mt7915e
98: 2128249 0 mvebu-INTx 0 Level ath9k
IPI0: 0 1 CPU wakeup interrupts
IPI1: 0 0 Timer broadcast interrupts
IPI2: 6353 5783 Rescheduling interrupts
IPI3: 385746 1621331 Function call interrupts
IPI4: 0 0 CPU stop interrupts
IPI5: 0 0 IRQ work interrupts
IPI6: 0 0 completion interrupts
Err: 0
root@turris:~# cat /proc/softirqs
CPU0 CPU1
HI: 0 0
TIMER: 779309 184141
NET_TX: 303044 789
NET_RX: 77253117 234871
BLOCK: 8141 7978
IRQ_POLL: 0 0
TASKLET: 4967234 504658
SCHED: 1487078 1257448
HRTIMER: 0 0
RCU: 522277 431451
So yes, you have most interrupt and softinterrupt processing on CPU0… Not sure which can be moved around, but I would try irqbalance, as @peci1 proposed, and then have this run for some time and repost /proc/interrupts again…
I have the same behavior since TOS 6. And performance degradation.
And it’s “normal” if we see an old post Turris 6.0 - download speed capped at 860 Mbit/s
Not sure if there was another post related to that.
MultiCPU DSA didn’t work yet maybe.
I’m on TOS 6.3 and still the same. Just tested with a speedtest and only CPU0 fully used.
My softirqs:
CPU0 CPU1
HI: 0 0
TIMER: 3549072 1905011
NET_TX: 1135809 1
NET_RX: 40123884 212887
BLOCK: 61396 63295
IRQ_POLL: 0 0
TASKLET: 11681753 79
SCHED: 8576086 7521782
HRTIMER: 61 5
RCU: 1646367 1541770
Are you running irqbalance?
More details on the vague feeling: Omnia’s interfaces are single-queue. (By my memory and by ethtool output.)
No didn’t tried yet. There is caution on irqbalance page : 2core targets, outside of benchmarking alone, there may be performance losses.
Maybe it was the same on TOS 5…didn’t checked that when I was on it.
I’ll try two heavy transferts on two computers at same time to see how are CPUs.
I’have a 2.5G sfp wan.
I think you need to activate receive packet steering, and maybe even edit the script that configures packet steering. In the past the issue was that packet steering tried to move packet processing (mostly done in softinterrupt context) to all CPUs that are not actually handling the ethernet interrupts. On a dual core router that means that e.g. when using SQM both shaper instances will run on CPU1, which is only a slight improvement over having both shapers and the ethernet interrupts on CPU0… IMHO the solution for dual core routers is to allow packet steering onto all CPUs.
I think the original design was motivated by a redhat howto, that IMHO targeted big multi-core computers with fast interfaces, where a) interrupt processing is a more noticeable load in itself and b) where there are enough CPUs to distribute the softinterrupt processing over. So I argue that the default packet steering strategy is not optimized for dual core routers…
Mmmh, looking in TOS6.2.4’s /etc/hotplug.d/net/20-smp-packet-steering it seems that the exclusion of interrupt processing CPUs is not happening anymore, so if you have not done so already please try to enable packet steering (in the Network->Interfaces page on the Global network options tab) and redo your test.
Side-note, I seem to recall that the omnia’s SoC can not move some interrupts around at all, so some will stick to CPU0…
I tried it now. Still 100% on CPU0, nothing on CPU1. Only diff is now I only get like 460 Mbit/s instead of 860 Mbit/s download. 400 Mbit/s slower download speed. Upload still around 940 Mbit/s.
Is this deterministic? That is if you disable packet-steering again do you see ~800 Mbps downloads again?