I’ve had Omnia for years and never had a problem. A couple weeks ago it started slowing down for no apparent reason. No config changes, only the automatic updates. It’s best demonstrated with ping:
64 bytes from _gateway (172.31.174.1): icmp_seq=20 ttl=64 time=129 ms
64 bytes from _gateway (172.31.174.1): icmp_seq=21 ttl=64 time=114 ms
64 bytes from _gateway (172.31.174.1): icmp_seq=22 ttl=64 time=135 ms
64 bytes from _gateway (172.31.174.1): icmp_seq=23 ttl=64 time=74.1 ms
64 bytes from _gateway (172.31.174.1): icmp_seq=24 ttl=64 time=120 ms
64 bytes from _gateway (172.31.174.1): icmp_seq=25 ttl=64 time=248 ms
64 bytes from _gateway (172.31.174.1): icmp_seq=26 ttl=64 time=280 ms
64 bytes from _gateway (172.31.174.1): icmp_seq=27 ttl=64 time=353 ms
64 bytes from _gateway (172.31.174.1): icmp_seq=28 ttl=64 time=151 ms
<Turris power cycle>
64 bytes from _gateway (172.31.174.1): icmp_seq=61 ttl=64 time=0.695 ms
64 bytes from _gateway (172.31.174.1): icmp_seq=62 ttl=64 time=0.424 ms
64 bytes from _gateway (172.31.174.1): icmp_seq=63 ttl=64 time=0.409 ms
64 bytes from _gateway (172.31.174.1): icmp_seq=64 ttl=64 time=0.507 ms
64 bytes from _gateway (172.31.174.1): icmp_seq=65 ttl=64 time=0.504 ms
64 bytes from _gateway (172.31.174.1): icmp_seq=66 ttl=64 time=0.474 ms
64 bytes from _gateway (172.31.174.1): icmp_seq=67 ttl=64 time=0.532 ms
64 bytes from _gateway (172.31.174.1): icmp_seq=68 ttl=64 time=0.586 ms
This is a ping over Ethernet, not Wifi.
It seems to slow down gradually and after a week or two it gets to be unusable and has to be restarted.
Simple reboot from the shell helps too but not sure if it’s as efficient as a power cycle.
It’s not overheating and it’s not overloaded with traffic, no idea what’s causing it. Any idea what it could be? HW problem? SW problem? Any solution to try?
do you use SW updates? which TOS version do you have?
Hi yes it’s auto patching and it’s up to date at 6.2.4.
any non-standard SW running on it?
n8mahr
March 5, 2023, 11:54am
5
have you checked ram usage? maybe some software is flooding it? or, since you mentioned “for years” - check the file system of the mmc…
Hi guys,
no nonstandard software, it’s just a router doing the router thing.
the memory doesn’t seem to be overused, the procs sizes are ok.
root@turris's password:
Warning: Changes performed using anything other than
official web interface reForis are not covered by
Turris support team unless instructed!
______ _ ____ _____
/_ __/_ ____________(_)____ / __ \/ ___/
/ / / / / / ___/ ___/ / ___/ / / / /\__
/ / / /_/ / / / / / (__ ) / /_/ /___/ /
/_/ \__,_/_/ /_/ /_/____/ \____//____/
-----------------------------------------------------
TurrisOS 6.2.4, Turris Omnia
-----------------------------------------------------
root@turris:~# ps uax
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 1452 928 ? Ss Mar05 0:06 /sbin/procd
root 2 0.0 0.0 0 0 ? S Mar05 0:00 [kthreadd]
root 3 0.0 0.0 0 0 ? I< Mar05 0:00 [rcu_gp]
root 4 0.0 0.0 0 0 ? I< Mar05 0:00 [rcu_par_gp]
root 5 0.0 0.0 0 0 ? I< Mar05 0:00 [slub_flushwq]
root 6 0.0 0.0 0 0 ? I< Mar05 0:00 [netns]
root 10 0.0 0.0 0 0 ? I< Mar05 0:00 [mm_percpu_wq]
root 11 0.0 0.0 0 0 ? S Mar05 0:00 [rcu_tasks_trace]
root 12 18.4 0.0 0 0 ? R Mar05 709:55 [ksoftirqd/0]
root 13 0.2 0.0 0 0 ? I Mar05 11:02 [rcu_sched]
root 14 0.0 0.0 0 0 ? S Mar05 0:00 [migration/0]
root 15 0.0 0.0 0 0 ? S Mar05 0:00 [cpuhp/0]
root 16 0.0 0.0 0 0 ? S Mar05 0:00 [cpuhp/1]
root 17 0.0 0.0 0 0 ? S Mar05 0:00 [migration/1]
root 18 0.0 0.0 0 0 ? S Mar05 0:06 [ksoftirqd/1]
root 21 0.0 0.0 0 0 ? S Mar05 0:00 [kdevtmpfs]
root 22 0.0 0.0 0 0 ? I< Mar05 0:00 [inet_frag_wq]
root 24 0.0 0.0 0 0 ? S Mar05 0:00 [oom_reaper]
root 25 0.0 0.0 0 0 ? I< Mar05 0:00 [writeback]
root 26 0.0 0.0 0 0 ? S Mar05 0:36 [kcompactd0]
root 38 0.0 0.0 0 0 ? I< Mar05 0:00 [pencrypt_serial]
root 39 0.0 0.0 0 0 ? I< Mar05 0:00 [pdecrypt_serial]
root 40 0.0 0.0 0 0 ? I< Mar05 0:00 [cryptd]
root 57 0.0 0.0 0 0 ? I< Mar05 0:00 [kblockd]
root 58 0.0 0.0 0 0 ? I< Mar05 0:00 [blkcg_punt_bio]
root 59 0.0 0.0 0 0 ? I< Mar05 0:00 [ata_sff]
root 60 0.0 0.0 0 0 ? S Mar05 0:00 [watchdogd]
root 89 0.0 0.0 0 0 ? S Mar05 0:00 [kswapd0]
root 91 0.0 0.0 0 0 ? I< Mar05 0:00 [kthrotld]
root 132 0.0 0.0 0 0 ? S Mar05 0:00 [irq/64-pciehp]
root 138 0.0 0.0 0 0 ? S Mar05 0:00 [irq/65-pciehp]
root 144 0.0 0.0 0 0 ? S Mar05 0:00 [irq/66-pciehp]
root 211 0.0 0.0 0 0 ? I< Mar05 0:00 [iscsi_eh]
root 216 0.0 0.0 0 0 ? I< Mar05 0:00 [iscsi_conn_clea]
root 219 0.0 0.0 0 0 ? I< Mar05 0:00 [nvme-wq]
root 222 0.0 0.0 0 0 ? I< Mar05 0:00 [nvme-reset-wq]
root 223 0.0 0.0 0 0 ? I< Mar05 0:00 [nvme-delete-wq]
root 238 0.0 0.0 0 0 ? S Mar05 0:00 [scsi_eh_0]
root 239 0.0 0.0 0 0 ? I< Mar05 0:00 [scsi_tmf_0]
root 243 0.0 0.0 0 0 ? S Mar05 0:00 [scsi_eh_1]
root 244 0.0 0.0 0 0 ? I< Mar05 0:00 [scsi_tmf_1]
root 258 0.0 0.0 0 0 ? S Mar05 0:00 [spi0]
root 492 0.0 0.0 0 0 ? I< Mar05 0:00 [kworker/u5:0-btrfs-worker-high]
root 493 0.0 0.0 0 0 ? I< Mar05 0:00 [sdhci]
root 494 0.0 0.0 0 0 ? S Mar05 0:00 [irq/54-mmc0]
root 501 0.0 0.0 0 0 ? S Mar05 0:00 [irq/50-f1090000]
root 502 0.0 0.0 0 0 ? S Mar05 0:00 [irq/51-f1090000]
root 519 0.0 0.0 0 0 ? I< Mar05 0:00 [mld]
root 520 0.0 0.0 0 0 ? I< Mar05 0:00 [ipv6_addrconf]
root 524 0.0 0.0 0 0 ? I< Mar05 0:00 [dsa_ordered]
root 540 0.0 0.0 0 0 ? S Mar05 0:00 [irq/69-mv88e6xx]
root 541 0.0 0.0 0 0 ? S Mar05 0:00 [irq/77-mv88e6xx]
root 542 0.0 0.0 0 0 ? S Mar05 0:00 [irq/94-mv88e6xx]
root 543 0.0 0.0 0 0 ? S Mar05 0:00 [irq/73-mv88e6xx]
root 544 0.0 0.0 0 0 ? S Mar05 0:00 [irq/75-mv88e6xx]
root 566 0.0 0.0 0 0 ? I< Mar05 0:00 [mmc_complete]
root 583 0.0 0.0 0 0 ? S Mar05 0:00 [irq/79-mv88e6xx]
root 587 0.0 0.0 0 0 ? S Mar05 0:00 [irq/80-mv88e6xx]
root 591 0.0 0.0 0 0 ? S Mar05 0:00 [irq/81-mv88e6xx]
root 595 0.0 0.0 0 0 ? S Mar05 0:00 [irq/82-mv88e6xx]
root 599 0.0 0.0 0 0 ? S Mar05 0:00 [irq/83-mv88e6xx]
root 609 0.0 0.0 0 0 ? I< Mar05 0:00 [btrfs-worker]
root 610 0.0 0.0 0 0 ? I< Mar05 0:00 [btrfs-worker-hi]
root 611 0.0 0.0 0 0 ? I< Mar05 0:00 [btrfs-delalloc]
root 612 0.0 0.0 0 0 ? I< Mar05 0:00 [btrfs-flush_del]
root 613 0.0 0.0 0 0 ? I< Mar05 0:00 [btrfs-cache]
root 614 0.0 0.0 0 0 ? I< Mar05 0:00 [btrfs-fixup]
root 615 0.0 0.0 0 0 ? I< Mar05 0:00 [btrfs-endio]
root 616 0.0 0.0 0 0 ? I< Mar05 0:00 [btrfs-endio-met]
root 617 0.0 0.0 0 0 ? I< Mar05 0:00 [btrfs-endio-met]
root 618 0.0 0.0 0 0 ? I< Mar05 0:00 [btrfs-endio-rai]
root 619 0.0 0.0 0 0 ? I< Mar05 0:00 [btrfs-rmw]
root 620 0.0 0.0 0 0 ? I< Mar05 0:00 [btrfs-endio-wri]
root 621 0.0 0.0 0 0 ? I< Mar05 0:00 [btrfs-freespace]
root 622 0.0 0.0 0 0 ? I< Mar05 0:00 [btrfs-delayed-m]
root 623 0.0 0.0 0 0 ? I< Mar05 0:00 [btrfs-readahead]
root 624 0.0 0.0 0 0 ? I< Mar05 0:00 [btrfs-qgroup-re]
root 625 0.0 0.0 0 0 ? S Mar05 0:01 [btrfs-cleaner]
root 626 0.0 0.0 0 0 ? S Mar05 0:18 [btrfs-transacti]
root 628 0.0 0.0 0 0 ? I< Mar05 0:00 [kworker/1:1H-kblockd]
root 633 0.0 0.0 0 0 ? I< Mar05 0:00 [uas]
root 681 0.0 0.0 0 0 ? I< Mar05 0:00 [kworker/u5:1-btrfs-worker-high]
ubus 967 0.0 0.0 1068 628 ? S Mar05 0:01 /sbin/ubusd
root 968 0.0 0.0 732 484 ttyS0 Ss+ Mar05 0:00 /sbin/askfirst /bin/ash --login
root 1023 0.0 0.0 1636 1172 ? S Mar05 0:12 /usr/sbin/haveged -F -w 1024 -d 32 -i 32 -v 1
root 1202 0.0 0.0 0 0 ? S Mar05 0:00 [irq/95-8-0071]
root 1376 0.0 0.0 0 0 ? I< Mar05 0:00 [cfg80211]
root 1417 0.0 0.0 0 0 ? I< Mar05 0:00 [ath10k_wq]
root 1418 0.0 0.0 0 0 ? I< Mar05 0:00 [ath10k_aux_wq]
root 1419 0.0 0.0 0 0 ? I< Mar05 0:00 [ath10k_tx_compl]
root 1752 0.0 0.0 1232 40 ? Ss Mar05 0:03 watchdog -T 30 -t 5 /dev/watchdog
root 1781 0.0 0.0 1912 980 ? S Mar05 0:00 /sbin/rpcd -s /var/run/ubus/ubus.sock -t 30
root 1842 0.0 0.0 0 0 ? I< Mar05 0:00 [kworker/1:2H]
root 2265 0.1 0.1 5048 2080 ? S Mar05 6:13 /usr/sbin/hostapd -s -g /var/run/hostapd/global
root 2266 0.0 0.1 5000 2080 ? S Mar05 0:06 /usr/sbin/wpa_supplicant -n -s -g /var/run/wpa_supplicant/global
kresd 2275 0.6 0.8 40536 17136 ? S< Mar07 5:47 /usr/sbin/kresd --noninteractive -c /tmp/kresd.config /tmp/kresd
root 2327 0.0 0.0 1708 1064 ? S Mar05 0:33 /sbin/netifd
root 2381 0.0 0.0 1320 704 ? S Mar05 1:20 /usr/sbin/odhcpd
nobody 2458 0.0 0.0 816 512 ? S Mar05 0:00 /usr/sbin/atd -f
root 2858 0.0 0.1 3508 3000 ? S Mar05 0:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
root 2902 0.0 0.3 13072 7940 ? Sl Mar05 0:20 /usr/sbin/syslog-ng --foreground
mosquit+ 3952 0.1 0.1 3816 2176 ? S Mar05 6:57 mosquitto -c /var/etc/fosquitto.generated.conf
root 4005 0.1 0.6 18612 14428 ? Sl Mar05 6:23 /usr/bin/python3 /usr/bin/foris-ws -a none --host 127.0.0.1 --port 9080 mqtt --mqtt-host localhost --mqtt-port 11883 --mqtt-passwd-file /etc/fosquitto/credentials.plain
root 4099 4.6 1.2 31768 26164 ? Sl Mar05 177:25 /usr/bin/python3 /usr/bin/foris-controller -b openwrt -C /var/run/foris-controller-client.sock mqtt --host localhost --port 11883 --passwd-file /etc/fosquitto/credentials.pl
root 4155 0.0 0.1 5436 4012 ? S Mar05 0:23 /usr/sbin/lighttpd -D -f /etc/lighttpd/lighttpd.conf
mosquit+ 4213 0.0 0.1 3888 3176 ? S Mar05 3:37 mosquitto -c /etc/mosquitto/mosquitto.conf
root 4652 0.0 0.0 1316 672 ? S Mar05 0:34 /usr/sbin/umdns
root 4724 0.0 0.0 1672 872 ? S Mar05 0:01 /usr/sbin/crond -n -P -m off -s
root 4751 0.4 0.9 23136 19300 ? S Mar05 16:22 /usr/bin/python3 /usr/bin/dev-detect-daemon
root 4820 0.0 0.7 23116 15624 ? Sl Mar05 0:49 /usr/bin/python3 /usr/bin/foris-controller -b openwrt -C /var/run/foris-controller-client.sock mqtt --host localhost --port 11883 --passwd-file /etc/fosquitto/credentials.pl
root 4960 0.0 0.7 24252 15868 ? Sl Mar05 0:49 /usr/bin/python3 /usr/bin/foris-controller -b openwrt -C /var/run/foris-controller-client.sock mqtt --host localhost --port 11883 --passwd-file /etc/fosquitto/credentials.pl
root 5452 0.0 0.0 1232 548 ? S< Mar05 0:01 /usr/sbin/ntpd -n -N -S /usr/sbin/ntpd-hotplug -p ntp.nic.cz -p 0.openwrt.pool.ntp.org -p 1.openwrt.pool.ntp.org -p 2.openwrt.pool.ntp.org -p 3.openwrt.pool.ntp.org -p 217.3
root 6144 0.2 1.0 30240 22060 ? Sl Mar05 9:56 /usr/bin/python3 /usr/bin/foris-controller -b openwrt -C /var/run/foris-controller-client.sock mqtt --host localhost --port 11883 --passwd-file /etc/fosquitto/credentials.pl
root 6454 0.0 0.0 1120 824 ? S Mar05 1:33 /usr/sbin/pppd nodetach ipparam wan ifname pppoe-wan lcp-echo-interval 1 lcp-echo-failure 5 lcp-echo-adaptive +ipv6 set AUTOIPV6=1 set PEERDNS=0 nodefaultroute usepeerdns ma
root 6620 0.0 0.0 876 572 ? S Mar05 0:01 odhcp6c -s /lib/netifd/dhcpv6.script -Nforce -P60 -U -t120 pppoe-wan
dnsmasq 6987 0.0 0.0 2680 1360 ? S Mar05 0:03 /usr/sbin/dnsmasq -C /var/etc/dnsmasq.conf.cfg01411c -k -x /var/run/dnsmasq/dnsmasq.cfg01411c.pid
root 6990 0.0 0.0 2648 860 ? S Mar05 0:00 /usr/sbin/dnsmasq -C /var/etc/dnsmasq.conf.cfg01411c -k -x /var/run/dnsmasq/dnsmasq.cfg01411c.pid
root 21438 0.0 0.0 0 0 ? I< 09:17 0:00 [kworker/0:2H-mmc_complete]
root 25901 82.8 0.2 6352 5236 ? R 10:18 178:29 /usr/sbin/openvpn --syslog openvpn(server_turris) --status /var/run/openvpn.server_turris.status --cd /var/etc --config openvpn-server_turris.conf --up /usr/libexec/openvpn-
root 26229 0.3 0.2 4920 4360 ? S 10:18 0:43 /usr/sbin/snmpd -Lf /dev/null -f -r
root 28842 0.0 0.0 0 0 ? I< 12:12 0:00 [kworker/0:0H-mmc_complete]
root 29683 0.0 0.0 0 0 ? I 13:08 0:02 [kworker/u4:1-events_unbound]
root 29846 0.0 0.0 0 0 ? I 13:26 0:00 [kworker/1:0-events_power_efficient]
root 29934 3.6 0.0 0 0 ? I 13:38 0:33 [kworker/0:0-events]
root 30014 3.7 0.0 0 0 ? I 13:43 0:22 [kworker/0:1-events]
root 30051 0.0 0.0 0 0 ? I 13:47 0:00 [kworker/1:2-events_power_efficient]
root 30055 0.0 0.0 0 0 ? I 13:48 0:00 [kworker/u4:2-events_unbound]
root 30056 2.1 0.0 0 0 ? I 13:48 0:06 [kworker/0:2-events]
root 30103 0.0 0.0 0 0 ? I 13:52 0:00 [kworker/1:1-events_power_efficient]
root 30106 1.5 0.1 3520 2952 ? Ss 13:53 0:00 sshd: root@pts/0
root 30110 1.2 0.0 2224 1960 pts/0 Ss 13:53 0:00 -bash
root 30118 0.3 0.0 0 0 ? I 13:53 0:00 [kworker/u4:0-events_unbound]
root 30119 0.0 0.0 1548 628 pts/0 R+ 13:53 0:00 ps uax
root@turris:~# free
total used free shared buff/cache available
Mem: 2064520 180496 1650332 26320 233692 1742292
Swap: 0 0 0
Here’s a Grafana chart for ping RTT, from an internal host. In the middle is a soft reboot of the turris router, no other changes to the network…
mawt
April 2, 2023, 12:45pm
7
I’m also experiencing this. It’s very noticable when I get packet loss in games. I did a fresh upgrade to Windows 11 last weekend and was able to rule out my desktop and now I found this post.
It seems you max out your CPU via OpenVPN?
OpenVPN has nothing to do with this, I tried.
The latency doesn’t recover when I kill OpenVPN.
OpenVPN runs all the time even right after the reboot with quite a constant traffic. Yet it still takes a couple of days for the latency problems to show up.
It’s unlikely to be that.
1 Like
tac2
April 3, 2023, 6:26pm
10
Strange indeed.
What does dmesg reveal after your system slowed to a crawl?
