Network slows down over time - Turris must be rebooted to recover

mludvig · March 1, 2023, 11:42pm

I’ve had Omnia for years and never had a problem. A couple weeks ago it started slowing down for no apparent reason. No config changes, only the automatic updates. It’s best demonstrated with ping:

64 bytes from _gateway (172.31.174.1): icmp_seq=20 ttl=64 time=129 ms
64 bytes from _gateway (172.31.174.1): icmp_seq=21 ttl=64 time=114 ms
64 bytes from _gateway (172.31.174.1): icmp_seq=22 ttl=64 time=135 ms
64 bytes from _gateway (172.31.174.1): icmp_seq=23 ttl=64 time=74.1 ms
64 bytes from _gateway (172.31.174.1): icmp_seq=24 ttl=64 time=120 ms
64 bytes from _gateway (172.31.174.1): icmp_seq=25 ttl=64 time=248 ms
64 bytes from _gateway (172.31.174.1): icmp_seq=26 ttl=64 time=280 ms
64 bytes from _gateway (172.31.174.1): icmp_seq=27 ttl=64 time=353 ms
64 bytes from _gateway (172.31.174.1): icmp_seq=28 ttl=64 time=151 ms

<Turris power cycle>

64 bytes from _gateway (172.31.174.1): icmp_seq=61 ttl=64 time=0.695 ms
64 bytes from _gateway (172.31.174.1): icmp_seq=62 ttl=64 time=0.424 ms
64 bytes from _gateway (172.31.174.1): icmp_seq=63 ttl=64 time=0.409 ms
64 bytes from _gateway (172.31.174.1): icmp_seq=64 ttl=64 time=0.507 ms
64 bytes from _gateway (172.31.174.1): icmp_seq=65 ttl=64 time=0.504 ms
64 bytes from _gateway (172.31.174.1): icmp_seq=66 ttl=64 time=0.474 ms
64 bytes from _gateway (172.31.174.1): icmp_seq=67 ttl=64 time=0.532 ms
64 bytes from _gateway (172.31.174.1): icmp_seq=68 ttl=64 time=0.586 ms

This is a ping over Ethernet, not Wifi.

It seems to slow down gradually and after a week or two it gets to be unusable and has to be restarted.

Simple reboot from the shell helps too but not sure if it’s as efficient as a power cycle.

It’s not overheating and it’s not overloaded with traffic, no idea what’s causing it. Any idea what it could be? HW problem? SW problem? Any solution to try?

fantomas · March 2, 2023, 6:22pm

do you use SW updates? which TOS version do you have?

mludvig · March 2, 2023, 6:56pm

Hi yes it’s auto patching and it’s up to date at 6.2.4.

fantomas · March 5, 2023, 11:44am

any non-standard SW running on it?

n8mahr · March 5, 2023, 11:54am

have you checked ram usage? maybe some software is flooding it? or, since you mentioned “for years” - check the file system of the mmc…

mludvig · March 8, 2023, 2:50am

Hi guys,

no nonstandard software, it’s just a router doing the router thing.
the memory doesn’t seem to be overused, the procs sizes are ok.

root@turris's password: 
 Warning: Changes performed using anything other than
 official web interface reForis are not covered by
 Turris support team unless instructed!
      ______                _         ____  _____
     /_  __/_  ____________(_)____   / __ \/ ___/
      / / / / / / ___/ ___/ / ___/  / / / /\__ 
     / / / /_/ / /  / /  / (__  )  / /_/ /___/ / 
    /_/  \__,_/_/  /_/  /_/____/   \____//____/  
                                             
 -----------------------------------------------------
 TurrisOS 6.2.4, Turris Omnia
 -----------------------------------------------------
root@turris:~# ps uax
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0   1452   928 ?        Ss   Mar05   0:06 /sbin/procd
root         2  0.0  0.0      0     0 ?        S    Mar05   0:00 [kthreadd]
root         3  0.0  0.0      0     0 ?        I<   Mar05   0:00 [rcu_gp]
root         4  0.0  0.0      0     0 ?        I<   Mar05   0:00 [rcu_par_gp]
root         5  0.0  0.0      0     0 ?        I<   Mar05   0:00 [slub_flushwq]
root         6  0.0  0.0      0     0 ?        I<   Mar05   0:00 [netns]
root        10  0.0  0.0      0     0 ?        I<   Mar05   0:00 [mm_percpu_wq]
root        11  0.0  0.0      0     0 ?        S    Mar05   0:00 [rcu_tasks_trace]
root        12 18.4  0.0      0     0 ?        R    Mar05 709:55 [ksoftirqd/0]
root        13  0.2  0.0      0     0 ?        I    Mar05  11:02 [rcu_sched]
root        14  0.0  0.0      0     0 ?        S    Mar05   0:00 [migration/0]
root        15  0.0  0.0      0     0 ?        S    Mar05   0:00 [cpuhp/0]
root        16  0.0  0.0      0     0 ?        S    Mar05   0:00 [cpuhp/1]
root        17  0.0  0.0      0     0 ?        S    Mar05   0:00 [migration/1]
root        18  0.0  0.0      0     0 ?        S    Mar05   0:06 [ksoftirqd/1]
root        21  0.0  0.0      0     0 ?        S    Mar05   0:00 [kdevtmpfs]
root        22  0.0  0.0      0     0 ?        I<   Mar05   0:00 [inet_frag_wq]
root        24  0.0  0.0      0     0 ?        S    Mar05   0:00 [oom_reaper]
root        25  0.0  0.0      0     0 ?        I<   Mar05   0:00 [writeback]
root        26  0.0  0.0      0     0 ?        S    Mar05   0:36 [kcompactd0]
root        38  0.0  0.0      0     0 ?        I<   Mar05   0:00 [pencrypt_serial]
root        39  0.0  0.0      0     0 ?        I<   Mar05   0:00 [pdecrypt_serial]
root        40  0.0  0.0      0     0 ?        I<   Mar05   0:00 [cryptd]
root        57  0.0  0.0      0     0 ?        I<   Mar05   0:00 [kblockd]
root        58  0.0  0.0      0     0 ?        I<   Mar05   0:00 [blkcg_punt_bio]
root        59  0.0  0.0      0     0 ?        I<   Mar05   0:00 [ata_sff]
root        60  0.0  0.0      0     0 ?        S    Mar05   0:00 [watchdogd]
root        89  0.0  0.0      0     0 ?        S    Mar05   0:00 [kswapd0]
root        91  0.0  0.0      0     0 ?        I<   Mar05   0:00 [kthrotld]
root       132  0.0  0.0      0     0 ?        S    Mar05   0:00 [irq/64-pciehp]
root       138  0.0  0.0      0     0 ?        S    Mar05   0:00 [irq/65-pciehp]
root       144  0.0  0.0      0     0 ?        S    Mar05   0:00 [irq/66-pciehp]
root       211  0.0  0.0      0     0 ?        I<   Mar05   0:00 [iscsi_eh]
root       216  0.0  0.0      0     0 ?        I<   Mar05   0:00 [iscsi_conn_clea]
root       219  0.0  0.0      0     0 ?        I<   Mar05   0:00 [nvme-wq]
root       222  0.0  0.0      0     0 ?        I<   Mar05   0:00 [nvme-reset-wq]
root       223  0.0  0.0      0     0 ?        I<   Mar05   0:00 [nvme-delete-wq]
root       238  0.0  0.0      0     0 ?        S    Mar05   0:00 [scsi_eh_0]
root       239  0.0  0.0      0     0 ?        I<   Mar05   0:00 [scsi_tmf_0]
root       243  0.0  0.0      0     0 ?        S    Mar05   0:00 [scsi_eh_1]
root       244  0.0  0.0      0     0 ?        I<   Mar05   0:00 [scsi_tmf_1]
root       258  0.0  0.0      0     0 ?        S    Mar05   0:00 [spi0]
root       492  0.0  0.0      0     0 ?        I<   Mar05   0:00 [kworker/u5:0-btrfs-worker-high]
root       493  0.0  0.0      0     0 ?        I<   Mar05   0:00 [sdhci]
root       494  0.0  0.0      0     0 ?        S    Mar05   0:00 [irq/54-mmc0]
root       501  0.0  0.0      0     0 ?        S    Mar05   0:00 [irq/50-f1090000]
root       502  0.0  0.0      0     0 ?        S    Mar05   0:00 [irq/51-f1090000]
root       519  0.0  0.0      0     0 ?        I<   Mar05   0:00 [mld]
root       520  0.0  0.0      0     0 ?        I<   Mar05   0:00 [ipv6_addrconf]
root       524  0.0  0.0      0     0 ?        I<   Mar05   0:00 [dsa_ordered]
root       540  0.0  0.0      0     0 ?        S    Mar05   0:00 [irq/69-mv88e6xx]
root       541  0.0  0.0      0     0 ?        S    Mar05   0:00 [irq/77-mv88e6xx]
root       542  0.0  0.0      0     0 ?        S    Mar05   0:00 [irq/94-mv88e6xx]
root       543  0.0  0.0      0     0 ?        S    Mar05   0:00 [irq/73-mv88e6xx]
root       544  0.0  0.0      0     0 ?        S    Mar05   0:00 [irq/75-mv88e6xx]
root       566  0.0  0.0      0     0 ?        I<   Mar05   0:00 [mmc_complete]
root       583  0.0  0.0      0     0 ?        S    Mar05   0:00 [irq/79-mv88e6xx]
root       587  0.0  0.0      0     0 ?        S    Mar05   0:00 [irq/80-mv88e6xx]
root       591  0.0  0.0      0     0 ?        S    Mar05   0:00 [irq/81-mv88e6xx]
root       595  0.0  0.0      0     0 ?        S    Mar05   0:00 [irq/82-mv88e6xx]
root       599  0.0  0.0      0     0 ?        S    Mar05   0:00 [irq/83-mv88e6xx]
root       609  0.0  0.0      0     0 ?        I<   Mar05   0:00 [btrfs-worker]
root       610  0.0  0.0      0     0 ?        I<   Mar05   0:00 [btrfs-worker-hi]
root       611  0.0  0.0      0     0 ?        I<   Mar05   0:00 [btrfs-delalloc]
root       612  0.0  0.0      0     0 ?        I<   Mar05   0:00 [btrfs-flush_del]
root       613  0.0  0.0      0     0 ?        I<   Mar05   0:00 [btrfs-cache]
root       614  0.0  0.0      0     0 ?        I<   Mar05   0:00 [btrfs-fixup]
root       615  0.0  0.0      0     0 ?        I<   Mar05   0:00 [btrfs-endio]
root       616  0.0  0.0      0     0 ?        I<   Mar05   0:00 [btrfs-endio-met]
root       617  0.0  0.0      0     0 ?        I<   Mar05   0:00 [btrfs-endio-met]
root       618  0.0  0.0      0     0 ?        I<   Mar05   0:00 [btrfs-endio-rai]
root       619  0.0  0.0      0     0 ?        I<   Mar05   0:00 [btrfs-rmw]
root       620  0.0  0.0      0     0 ?        I<   Mar05   0:00 [btrfs-endio-wri]
root       621  0.0  0.0      0     0 ?        I<   Mar05   0:00 [btrfs-freespace]
root       622  0.0  0.0      0     0 ?        I<   Mar05   0:00 [btrfs-delayed-m]
root       623  0.0  0.0      0     0 ?        I<   Mar05   0:00 [btrfs-readahead]
root       624  0.0  0.0      0     0 ?        I<   Mar05   0:00 [btrfs-qgroup-re]
root       625  0.0  0.0      0     0 ?        S    Mar05   0:01 [btrfs-cleaner]
root       626  0.0  0.0      0     0 ?        S    Mar05   0:18 [btrfs-transacti]
root       628  0.0  0.0      0     0 ?        I<   Mar05   0:00 [kworker/1:1H-kblockd]
root       633  0.0  0.0      0     0 ?        I<   Mar05   0:00 [uas]
root       681  0.0  0.0      0     0 ?        I<   Mar05   0:00 [kworker/u5:1-btrfs-worker-high]
ubus       967  0.0  0.0   1068   628 ?        S    Mar05   0:01 /sbin/ubusd
root       968  0.0  0.0    732   484 ttyS0    Ss+  Mar05   0:00 /sbin/askfirst /bin/ash --login
root      1023  0.0  0.0   1636  1172 ?        S    Mar05   0:12 /usr/sbin/haveged -F -w 1024 -d 32 -i 32 -v 1
root      1202  0.0  0.0      0     0 ?        S    Mar05   0:00 [irq/95-8-0071]
root      1376  0.0  0.0      0     0 ?        I<   Mar05   0:00 [cfg80211]
root      1417  0.0  0.0      0     0 ?        I<   Mar05   0:00 [ath10k_wq]
root      1418  0.0  0.0      0     0 ?        I<   Mar05   0:00 [ath10k_aux_wq]
root      1419  0.0  0.0      0     0 ?        I<   Mar05   0:00 [ath10k_tx_compl]
root      1752  0.0  0.0   1232    40 ?        Ss   Mar05   0:03 watchdog -T 30 -t 5 /dev/watchdog
root      1781  0.0  0.0   1912   980 ?        S    Mar05   0:00 /sbin/rpcd -s /var/run/ubus/ubus.sock -t 30
root      1842  0.0  0.0      0     0 ?        I<   Mar05   0:00 [kworker/1:2H]
root      2265  0.1  0.1   5048  2080 ?        S    Mar05   6:13 /usr/sbin/hostapd -s -g /var/run/hostapd/global
root      2266  0.0  0.1   5000  2080 ?        S    Mar05   0:06 /usr/sbin/wpa_supplicant -n -s -g /var/run/wpa_supplicant/global
kresd     2275  0.6  0.8  40536 17136 ?        S<   Mar07   5:47 /usr/sbin/kresd --noninteractive -c /tmp/kresd.config /tmp/kresd
root      2327  0.0  0.0   1708  1064 ?        S    Mar05   0:33 /sbin/netifd
root      2381  0.0  0.0   1320   704 ?        S    Mar05   1:20 /usr/sbin/odhcpd
nobody    2458  0.0  0.0    816   512 ?        S    Mar05   0:00 /usr/sbin/atd -f
root      2858  0.0  0.1   3508  3000 ?        S    Mar05   0:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
root      2902  0.0  0.3  13072  7940 ?        Sl   Mar05   0:20 /usr/sbin/syslog-ng --foreground
mosquit+  3952  0.1  0.1   3816  2176 ?        S    Mar05   6:57 mosquitto -c /var/etc/fosquitto.generated.conf
root      4005  0.1  0.6  18612 14428 ?        Sl   Mar05   6:23 /usr/bin/python3 /usr/bin/foris-ws -a none --host 127.0.0.1 --port 9080 mqtt --mqtt-host localhost --mqtt-port 11883 --mqtt-passwd-file /etc/fosquitto/credentials.plain
root      4099  4.6  1.2  31768 26164 ?        Sl   Mar05 177:25 /usr/bin/python3 /usr/bin/foris-controller -b openwrt -C /var/run/foris-controller-client.sock mqtt --host localhost --port 11883 --passwd-file /etc/fosquitto/credentials.pl
root      4155  0.0  0.1   5436  4012 ?        S    Mar05   0:23 /usr/sbin/lighttpd -D -f /etc/lighttpd/lighttpd.conf
mosquit+  4213  0.0  0.1   3888  3176 ?        S    Mar05   3:37 mosquitto -c /etc/mosquitto/mosquitto.conf
root      4652  0.0  0.0   1316   672 ?        S    Mar05   0:34 /usr/sbin/umdns
root      4724  0.0  0.0   1672   872 ?        S    Mar05   0:01 /usr/sbin/crond -n -P -m off -s
root      4751  0.4  0.9  23136 19300 ?        S    Mar05  16:22 /usr/bin/python3 /usr/bin/dev-detect-daemon
root      4820  0.0  0.7  23116 15624 ?        Sl   Mar05   0:49 /usr/bin/python3 /usr/bin/foris-controller -b openwrt -C /var/run/foris-controller-client.sock mqtt --host localhost --port 11883 --passwd-file /etc/fosquitto/credentials.pl
root      4960  0.0  0.7  24252 15868 ?        Sl   Mar05   0:49 /usr/bin/python3 /usr/bin/foris-controller -b openwrt -C /var/run/foris-controller-client.sock mqtt --host localhost --port 11883 --passwd-file /etc/fosquitto/credentials.pl
root      5452  0.0  0.0   1232   548 ?        S<   Mar05   0:01 /usr/sbin/ntpd -n -N -S /usr/sbin/ntpd-hotplug -p ntp.nic.cz -p 0.openwrt.pool.ntp.org -p 1.openwrt.pool.ntp.org -p 2.openwrt.pool.ntp.org -p 3.openwrt.pool.ntp.org -p 217.3
root      6144  0.2  1.0  30240 22060 ?        Sl   Mar05   9:56 /usr/bin/python3 /usr/bin/foris-controller -b openwrt -C /var/run/foris-controller-client.sock mqtt --host localhost --port 11883 --passwd-file /etc/fosquitto/credentials.pl
root      6454  0.0  0.0   1120   824 ?        S    Mar05   1:33 /usr/sbin/pppd nodetach ipparam wan ifname pppoe-wan lcp-echo-interval 1 lcp-echo-failure 5 lcp-echo-adaptive +ipv6 set AUTOIPV6=1 set PEERDNS=0 nodefaultroute usepeerdns ma
root      6620  0.0  0.0    876   572 ?        S    Mar05   0:01 odhcp6c -s /lib/netifd/dhcpv6.script -Nforce -P60 -U -t120 pppoe-wan
dnsmasq   6987  0.0  0.0   2680  1360 ?        S    Mar05   0:03 /usr/sbin/dnsmasq -C /var/etc/dnsmasq.conf.cfg01411c -k -x /var/run/dnsmasq/dnsmasq.cfg01411c.pid
root      6990  0.0  0.0   2648   860 ?        S    Mar05   0:00 /usr/sbin/dnsmasq -C /var/etc/dnsmasq.conf.cfg01411c -k -x /var/run/dnsmasq/dnsmasq.cfg01411c.pid
root     21438  0.0  0.0      0     0 ?        I<   09:17   0:00 [kworker/0:2H-mmc_complete]
root     25901 82.8  0.2   6352  5236 ?        R    10:18 178:29 /usr/sbin/openvpn --syslog openvpn(server_turris) --status /var/run/openvpn.server_turris.status --cd /var/etc --config openvpn-server_turris.conf --up /usr/libexec/openvpn-
root     26229  0.3  0.2   4920  4360 ?        S    10:18   0:43 /usr/sbin/snmpd -Lf /dev/null -f -r
root     28842  0.0  0.0      0     0 ?        I<   12:12   0:00 [kworker/0:0H-mmc_complete]
root     29683  0.0  0.0      0     0 ?        I    13:08   0:02 [kworker/u4:1-events_unbound]
root     29846  0.0  0.0      0     0 ?        I    13:26   0:00 [kworker/1:0-events_power_efficient]
root     29934  3.6  0.0      0     0 ?        I    13:38   0:33 [kworker/0:0-events]
root     30014  3.7  0.0      0     0 ?        I    13:43   0:22 [kworker/0:1-events]
root     30051  0.0  0.0      0     0 ?        I    13:47   0:00 [kworker/1:2-events_power_efficient]
root     30055  0.0  0.0      0     0 ?        I    13:48   0:00 [kworker/u4:2-events_unbound]
root     30056  2.1  0.0      0     0 ?        I    13:48   0:06 [kworker/0:2-events]
root     30103  0.0  0.0      0     0 ?        I    13:52   0:00 [kworker/1:1-events_power_efficient]
root     30106  1.5  0.1   3520  2952 ?        Ss   13:53   0:00 sshd: root@pts/0
root     30110  1.2  0.0   2224  1960 pts/0    Ss   13:53   0:00 -bash
root     30118  0.3  0.0      0     0 ?        I    13:53   0:00 [kworker/u4:0-events_unbound]
root     30119  0.0  0.0   1548   628 pts/0    R+   13:53   0:00 ps uax

root@turris:~# free
              total        used        free      shared  buff/cache   available
Mem:        2064520      180496     1650332       26320      233692     1742292
Swap:             0           0           0

Here’s a Grafana chart for ping RTT, from an internal host. In the middle is a soft reboot of the turris router, no other changes to the network…

mawt · April 2, 2023, 12:45pm

I’m also experiencing this. It’s very noticable when I get packet loss in games. I did a fresh upgrade to Windows 11 last weekend and was able to rule out my desktop and now I found this post.

moeller0 · April 2, 2023, 1:13pm

mludvig:

root     25901 82.8  0.2   6352  5236 ?        R    10:18 178:29 /usr/sbin/openvpn --syslog openvpn(server_turris) --status /var/run/openvpn.server_turris.status --cd /var/etc --config openvpn-server_turris.conf --up /usr/libexec/openvpn-

It seems you max out your CPU via OpenVPN?

mludvig · April 2, 2023, 10:03pm

OpenVPN has nothing to do with this, I tried.

The latency doesn’t recover when I kill OpenVPN.
OpenVPN runs all the time even right after the reboot with quite a constant traffic. Yet it still takes a couple of days for the latency problems to show up.

It’s unlikely to be that.

tac2 · April 3, 2023, 6:26pm

Strange indeed.
I have 4ms over 2.4GHz wifi on an old accespoint to my MOX (6.3)

moeller0 · April 3, 2023, 6:37pm

What does dmesg reveal after your system slowed to a crawl?