Unprovoked OOM mass killing

Software SW help
1

Hi! I need some community help.

My Omnia (2Gb RAM, HBT) a couple of days ago suddenly started to reboot every few hours.
The reason appears to be the oom-killer, but I fail to see the reason. The logs say that a random process invokes oom-killer, then it shoots one process after another, until it kills something that triggers a reboot.
Now, I confess that I do run a lot on my omnia. The most notable is an lxc container with Home Assistant and a mariadb database.
But the memory consumption never comes close to the total RAM. Typically, there is over 1GB of RAM is still available. I even added 2GB of swap which it seems to completely ignore and never use.

What am I missing?

Here is an example:

[10357.325109] python3 invoked oom-killer: gfp_mask=0x400dc0(GFP_KERNEL_ACCOUNT|__GFP_ZERO), order=1, oom_score_adj=0
[10357.335544] CPU: 0 PID: 25914 Comm: python3 Tainted: G        W         5.15.80 #0
[10357.343135] Hardware name: Marvell Armada 380/385 (Device Tree)
[10357.349071] [<c010eab8>] (unwind_backtrace) from [<c010a814>] (show_stack+0x10/0x14)
[10357.356843] [<c010a814>] (show_stack) from [<c05edfc8>] (dump_stack_lvl+0x40/0x4c)
[10357.364435] [<c05edfc8>] (dump_stack_lvl) from [<c023bf1c>] (dump_header+0x58/0x1bc)
[10357.372203] [<c023bf1c>] (dump_header) from [<c023c92c>] (oom_kill_process+0x19c/0x1a0)
[10357.380229] [<c023c92c>] (oom_kill_process) from [<c023d158>] (out_of_memory+0x2b0/0x388)
[10357.388428] [<c023d158>] (out_of_memory) from [<c0289a4c>] (__alloc_pages+0xad4/0xfe0)
[10357.396370] [<c0289a4c>] (__alloc_pages) from [<c012a47c>] (copy_process+0x2b4/0x1660)
[10357.404313] [<c012a47c>] (copy_process) from [<c012b9bc>] (kernel_clone+0xbc/0x3b0)
[10357.411992] [<c012b9bc>] (kernel_clone) from [<c012be64>] (sys_fork+0x38/0x54)
[10357.419236] [<c012be64>] (sys_fork) from [<c0100060>] (ret_fast_syscall+0x0/0x48)
[10357.426740] Exception stack(0xc8d79fa8 to 0xc8d79ff0)
[10357.431804] 9fa0:                   00000000 b6fe4190 00000000 b6fca4ac beb8ff0c 00000008
[10357.440002] 9fc0: 00000000 b6fe4190 beb8ff0c 00000002 00000000 01eac6a0 01ebbec8 00000000
[10357.448198] 9fe0: b6babf74 beb8ff08 b6f9e09c b6f9e0a0
[10357.453746] Mem-Info:
[10357.456029] active_anon:159437 inactive_anon:46553 isolated_anon:0
[10357.456029]  active_file:46508 inactive_file:209709 isolated_file:0
[10357.456029]  unevictable:0 dirty:1001 writeback:0
[10357.456029]  slab_reclaimable:4216 slab_unreclaimable:9707
[10357.456029]  mapped:35188 shmem:51272 pagetables:1093 bounce:0
[10357.456029]  kernel_misc_reclaimable:0
[10357.456029]  free:23618 free_pcp:9 free_cma:0
[10357.492894] Node 0 active_anon:637748kB inactive_anon:186212kB active_file:186032kB inactive_file:838840kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:140752kB dirty:4004kB writeback:0kB shmem:205088kB writeback_tmp:0kB kernel_stack:2720kB pagetables:4372kB all_unreclaimable? yes
[10357.519500] Normal free:16352kB min:16384kB low:20480kB high:24576kB reserved_highatomic:0KB active_anon:25924kB inactive_anon:7712kB active_file:4944kB inactive_file:594060kB unevictable:0kB writepending:3696kB present:786432kB managed:753800kB mlocked:0kB bounce:0kB free_pcp:36kB local_pcp:36kB free_cma:0kB
[10357.547088] lowmem_reserve[]: 0 10240 10240
[10357.551377] HighMem free:78500kB min:512kB low:7632kB high:14752kB reserved_highatomic:0KB active_anon:611824kB inactive_anon:178500kB active_file:181004kB inactive_file:244748kB unevictable:0kB writepending:196kB present:1310720kB managed:1310720kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[10357.579243] lowmem_reserve[]: 0 0 0
[10357.582822] Normal: 1181*4kB (UME) 395*8kB (ME) 192*16kB (ME) 92*32kB (UME) 28*64kB (UME) 8*128kB (UE) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 16716kB
[10357.596891] HighMem: 6171*4kB (UM) 949*8kB (UM) 190*16kB (UM) 78*32kB (UM) 36*64kB (UM) 16*128kB (UM) 9*256kB (UM) 8*512kB (UM) 5*1024kB (UM) 4*2048kB (M) 4*4096kB (M) = 78260kB
[10357.613030] 307150 total pagecache pages
[10357.617049] 0 pages in swap cache
[10357.620373] Swap cache stats: add 0, delete 0, find 0/0
[10357.625720] Free swap  = 2097148kB
[10357.629246] Total swap = 2097148kB
[10357.632676] 524288 pages RAM
[10357.635563] 327680 pages HighMem/MovableOnly
[10357.639918] 8158 pages reserved
[10357.643090] Tasks state (memory values in pages):
[10357.647811] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[10357.656635] [    961]    81   961      267      157    10240        0             0 ubusd
[10357.664963] [    963]     0   963      183      105     8192        0             0 askfirst
[10357.673514] [   1022]     0  1022      409      291    10240        0             0 haveged
[10357.681936] [   2061]     0  2061      308       10     6144        0             0 watchdog
[10357.690429] [   2090]     0  2090     1393      365    12288        0             0 rpcd
[10357.698587] [   2573]     0  2573     1269      804    12288        0             0 hostapd
[10357.706995] [   2574]     0  2574     1249      478    10240        0             0 wpa_supplicant
[10357.715992] [   2634]     0  2634      421      265    10240        0             0 netifd
[10357.724392] [   2682]     0  2682     5269     4160    28672        0             0 python3
[10357.732836] [   2770]     0  2770      325      207    10240        0             0 odhcpd
[10357.741277] [   3899]     0  3899      308      137     6144        0             0 udhcpc
[10357.749639] [   3920]     0  3920      219      159     8192        0             0 odhcp6c
[10357.758109] [   4130]   453  4130      663      335    10240        0             0 dnsmasq
[10357.766504] [   4134]     0  4134      662      208    10240        0             0 dnsmasq
[10357.774918] [  21854]     0 21854      312      167     8192        0             0 miniupnpd
[10357.783490] [  21920]     0 21920      375      329     8192        0             0 sh
[10357.791435] [  22804]     0 22804      380      332    10240        0             0 sh
[10357.799395] [  23173]     0 23173      204      128     6144        0             0 atd
[10357.807446] [  23217]     0 23217      322      275    10240        0             0 sh
[10357.815400] [  23246]     0 23246      192      114     8192        0             0 hostapd_cli
[10357.824240] [  23250]     0 23250      192      145     8192        0             0 hostapd_cli
[10357.832998] [  23313]     0 23313      298      163    10240        0             0 ubus
[10357.841409] [  23380]     0 23380    74011      755    36864        0             0 php-fpm
[10357.849980] [  23381] 65534 23381    74695     8623    63488        0             0 php-fpm
[10357.858572] [  23382] 65534 23382    74686     8797    63488        0             0 php-fpm
[10357.867062] [  23461]     0 23461      876      730    12288        0         -1000 sshd
[10357.875220] [  23505]     0 23505     3281     2027    20480        0             0 syslog-ng
[10357.883975] [  23549]    52 23549     7086     6841    36864        0             0 tor
[10357.892061] [  23602]     0 23602      351      228     8192        0             0 dbus-daemon
[10357.900852] [  23781]     0 23781     1024      652    10240        0             0 bluetoothd
[10357.909529] [  23861]   200 23861      984      813    10240        0             0 mosquitto
[10357.918115] [  23914]     0 23914     4676     3660    28672        0             0 foris-ws
[10357.926802] [  23987]     0 23987     8639     7168    43008        0             0 foris-controlle
[10357.936064] [  24091]     0 24091     1351      223    14336        0             0 master
[10357.944448] [  24093]    25 24093     1384      480    14336        0             0 qmgr
[10357.952659] [  24117]     0 24117      867      439    10240        0             0 dovecot
[10357.961111] [  24148]    59 24148      809      181    10240        0             0 anvil
[10357.969340] [  24149]     0 24149      841      440    12288        0             0 log
[10357.977456] [  24151]     0 24151     1059      622    12288        0             0 config
[10357.985937] [  24159]     0 24159     1707     1344    14336        0             0 lighttpd
[10357.994439] [  24552]     0 24552      329      168    10240        0             0 umdns
[10358.002850] [  24585]     0 24585     5793     3950    30720        0             0 foris-controlle
[10358.011996] [  24637]     0 24637      418      218     6144        0             0 crond
[10358.020266] [  24676]     0 24676     6084     3999    30720        0             0 foris-controlle
[10358.029492] [  24710]     0 24710     5691     4745    28672        0             0 dev-detect-daem
[10358.038762] [  24832]     0 24832     1004      827    12288        0             0 openvpn
[10358.047250] [  25785]     0 25785     8239     6093    40960        0             0 foris-controlle
[10358.056400] [  25914]     0 25914     3651     3261    24576        0             0 python3
[10358.064893] [  25958]     0 25958     2806     2368    18432        0             0 python3
[10358.073363] [  26185]   376 26185    87510    23787   137216        0             0 mysqld
[10358.081687] [  26236]     0 26236     1880     1340    16384        0             0 sentinel-proxy
[10358.090702] [  26309]     0 26309    16566    13382    75776        0             0 Suricata-Main
[10358.099613] [  26355]     0 26355     8230     7341    43008        0             0 python3
[10358.108079] [  26441]     0 26441     1848      909    14336        0             0 sentinel-fwlogs
[10358.117198] [  26498]     0 26498     1859      909    16384        0             0 minipot [Master
[10358.126383] [  26511] 65534 26511     1852      312    14336        0             0 minipot [FTP]
[10358.135346] [  26512] 65534 26512     1899      329    14336        0             0 minipot [HTTP]
[10358.144398] [  26513] 65534 26513     1839       89    14336        0             0 minipot [Telnet
[10358.153495] [  26574]     0 26574      308      137     6144        0             0 ntpd
[10358.161802] [  26607]     0 26607     1098      405    12288        0             0 lxc-start
[10358.170460] [  26615]     0 26615      435      240    10240        0             0 init
[10358.178698] [  26666]     0 26666     1098      403    12288        0             0 lxc-start
[10358.187379] [  26669]     0 26669      435      240     8192        0             0 init
[10358.195542] [  27224]     0 27224      439       12     8192        0             0 syslogd
[10358.203953] [  27281]     0 27281      435      240     8192        0             0 crond
[10358.212266] [  27316]    25 27316     1374      515    12288        0             0 tlsmgr
[10358.220738] [  27353]     0 27353      439       12     8192        0             0 syslogd
[10358.229328] [  27421]     0 27421      436      244     8192        0             0 crond
[10358.237736] [  28076]     0 28076      435        8     8192        0             0 udhcpc
[10358.246120] [  28101]     0 28101      435        8     8192        0             0 udhcpc
[10358.254479] [  28147]  6449 28147      438        1    10240        0             0 hass_supervisor
[10358.263621] [  28149]  6449 28149      435      240    10240        0             0 logger
[10358.272142] [  28150]  6449 28150      438        1    10240        0             0 sh
[10358.280152] [  28151]  6449 28151    48180    36593   194560        0             0 hass
[10358.288425] [  28152]  6449 28152      437      258     8192        0             0 sed
[10358.296529] [  28153]  6449 28153      435      240     8192        0             0 logger
[10358.304868] [  28161]     0 28161      435      208    10240        0             0 getty
[10358.313098] [  28163]     0 28163      435      224    10240        0             0 getty
[10358.321328] [  28165]     0 28165      435      208     8192        0             0 getty
[10358.329540] [  28168]     0 28168      435      208    10240        0             0 getty
[10358.337851] [  28170]     0 28170      435      208    10240        0             0 getty
[10358.346139] [  28187]  6447 28187      439      273    10240        0             0 propagation.sh
[10358.355148] [  28244]  6447 28244      274      172    10240        0             0 inotifywait
[10358.363906] [  28245]  6447 28245      441      292     8192        0             0 propagation.sh
[10358.373084] [  28325]  6447 28325      438        1     8192        0             0 hass_supervisor
[10358.382462] [  28327]  6447 28327      435      240    10240        0             0 logger
[10358.390758] [  28328]  6447 28328      438        1    10240        0             0 sh
[10358.399001] [  28329]  6447 28329    60293    50803   251904        0             0 hass
[10358.407172] [  28331]  6447 28331      437      258    10240        0             0 sed
[10358.415228] [  28332]  6447 28332      435      240    10240        0             0 logger
[10358.423547] [  28337]     0 28337      435      208     8192        0             0 getty
[10358.432011] [  28338]     0 28338      435      224    10240        0             0 getty
[10358.440417] [  28339]     0 28339      435      224     8192        0             0 getty
[10358.448718] [  28340]     0 28340      435      208    10240        0             0 getty
[10358.456998] [  28341]     0 28341      435      224    10240        0             0 getty
[10358.465285] [  28548]     0 28548      880      751    12288        0             0 sshd
[10358.473542] [  28552]     0 28552     1093      485    12288        0             0 3
[10358.481491] [  28563]     0 28563      535      309    10240        0             0 screen
[10358.489823] [  28565]     0 28565      577      411    10240        0             0 screen
[10358.498150] [  28566]     0 28566      484      322    10240        0             0 ash
[10358.506328] [  28851]     0 28851      903      776    10240        0             0 sshd
[10358.514573] [  28856]     0 28856      310      263    10240        0             0 ash
[10358.522682] [  28905]    59 28905     1435      696    14336        0             0 stats
[10358.530910] [  28906]    59 28906     1539      745    12288        0             0 auth
[10358.539181] [  28908]     0 28908     1536      750    16384        0             0 auth
[10358.547444] [   1373]  3536  1373    10774     4843    36864        0             0 kresd
[10358.555692] [  26253]     0 26253     2970     2257    18432        0             0 updater-supervi
[10358.564796] [   9537]    25  9537     1368      311    14336        0             0 pickup
[10358.573155] [   5017]     0  5017      310      139     8192        0             0 sleep
[10358.581415] [   5216]     0  5216      310      123     8192        0             0 sleep
[10358.589685] [   9153]    59  9153     1535      900    14336        0             0 imap-login
[10358.598408] [   9160]  6447  9160     1213      786    16384        0             0 imap
[10358.606576] [   9347]    59  9347     1536      903    18432        0             0 imap-login
[10358.615315] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=hass,pid=28329,uid=6447
[10358.628042] Out of memory: Killed process 28329 (hass) total-vm:241172kB, anon-rss:179804kB, file-rss:23408kB, shmem-rss:0kB, UID:6447 pgtables:246kB oom_score_adj:0
[10358.744151] netifd invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
[10358.752580] CPU: 1 PID: 2634 Comm: netifd Tainted: G        W         5.15.80 #0
[10358.760005] Hardware name: Marvell Armada 380/385 (Device Tree)
[10358.765950] [<c010eab8>] (unwind_backtrace) from [<c010a814>] (show_stack+0x10/0x14)
[10358.773728] [<c010a814>] (show_stack) from [<c05edfc8>] (dump_stack_lvl+0x40/0x4c)
[10358.781321] [<c05edfc8>] (dump_stack_lvl) from [<c023bf1c>] (dump_header+0x58/0x1bc)
[10358.789089] [<c023bf1c>] (dump_header) from [<c023c92c>] (oom_kill_process+0x19c/0x1a0)
[10358.797115] [<c023c92c>] (oom_kill_process) from [<c023d158>] (out_of_memory+0x2b0/0x388)
[10358.805315] [<c023d158>] (out_of_memory) from [<c0289a4c>] (__alloc_pages+0xad4/0xfe0)
[10358.813258] [<c0289a4c>] (__alloc_pages) from [<c0289f6c>] (__get_free_pages+0x14/0x24)
[10358.821285] [<c0289f6c>] (__get_free_pages) from [<c03318f8>] (proc_pid_readlink+0x78/0x18c)
[10358.829751] [<c03318f8>] (proc_pid_readlink) from [<c02b9eac>] (vfs_readlink+0x118/0x12c)
[10358.837962] [<c02b9eac>] (vfs_readlink) from [<c02b3450>] (do_readlinkat+0xac/0x104)
[10358.845738] [<c02b3450>] (do_readlinkat) from [<c0100060>] (ret_fast_syscall+0x0/0x48)
[10358.853677] Exception stack(0xc333bfa8 to 0xc333bff0)
[10358.858741] bfa0:                   0000000e b6f32190 bea50d04 bea50ce8 0000000e 06c7eda9
[10358.866940] bfc0: 0000000e b6f32190 b6e1c364 00000055 bea50ce8 bea50cf8 0000000a bea50d4c
[10358.875136] bfe0: 0052ae40 bea50ce4 0050f558 b6f0ac04
[10358.880250] Mem-Info:
[10358.882546] active_anon:114673 inactive_anon:46406 isolated_anon:0
[10358.882546]  active_file:32707 inactive_file:209714 isolated_file:0
[10358.882546]  unevictable:0 dirty:993 writeback:63
[10358.882546]  slab_reclaimable:4216 slab_unreclaimable:9661
[10358.882546]  mapped:29224 shmem:51251 pagetables:967 bounce:0
[10358.882546]  kernel_misc_reclaimable:0
[10358.882546]  free:82140 free_pcp:605 free_cma:0
[10358.919248] Node 0 active_anon:458272kB inactive_anon:185624kB active_file:132340kB inactive_file:839780kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:116980kB dirty:4056kB writeback:756kB shmem:205004kB writeback_tmp:0kB kernel_stack:2608kB pagetables:3868kB all_unreclaimable? no
[10358.945696] Normal free:21660kB min:16384kB low:20480kB high:24576kB reserved_highatomic:0KB active_anon:19192kB inactive_anon:7712kB active_file:5012kB inactive_file:595200kB unevictable:0kB writepending:4064kB present:786432kB managed:753800kB mlocked:0kB bounce:0kB free_pcp:1512kB local_pcp:388kB free_cma:0kB
[10358.973461] lowmem_reserve[]: 0 10240 10240
[10358.977671] HighMem free:303904kB min:512kB low:7632kB high:14752kB reserved_highatomic:0KB active_anon:439068kB inactive_anon:177996kB active_file:127412kB inactive_file:244896kB unevictable:0kB writepending:208kB present:1310720kB managed:1310720kB mlocked:0kB bounce:0kB free_pcp:1604kB local_pcp:240kB free_cma:0kB
[10359.005895] lowmem_reserve[]: 0 0 0
[10359.009413] Normal: 1212*4kB (UME) 532*8kB (UME) 224*16kB (UME) 99*32kB (ME) 33*64kB (UME) 7*128kB (UME) 6*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 20400kB
[10359.023937] HighMem: 13360*4kB (UM) 5700*8kB (M) 2398*16kB (M) 943*32kB (UM) 380*64kB (UM) 133*128kB (UM) 69*256kB (M) 39*512kB (UM) 22*1024kB (UM) 9*2048kB (UM) 4*4096kB (M) = 303904kB
[10359.040567] 293857 total pagecache pages
[10359.044530] 0 pages in swap cache
[10359.047859] Swap cache stats: add 0, delete 0, find 0/0
[10359.053107] Free swap  = 2097148kB
[10359.056518] Total swap = 2097148kB
[10359.059925] 524288 pages RAM
[10359.062849] 327680 pages HighMem/MovableOnly
[10359.067148] 8158 pages reserved
[10359.070305] Tasks state (memory values in pages):
[10359.075045] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[10359.083722] [    961]    81   961      267      157    10240        0             0 ubusd
[10359.091967] [    963]     0   963      183      105     8192        0             0 askfirst
[10359.100453] [   1022]     0  1022      409      291    10240        0             0 haveged
[10359.108865] [   2061]     0  2061      308       10     6144        0             0 watchdog
[10359.117372] [   2090]     0  2090     1393      365    12288        0             0 rpcd
[10359.125520] [   2573]     0  2573     1269      804    12288        0             0 hostapd
[10359.133930] [   2574]     0  2574     1249      478    10240        0             0 wpa_supplicant
[10359.142944] [   2634]     0  2634      421      265    10240        0             0 netifd
[10359.151246] [   2682]     0  2682     5269     4160    28672        0             0 python3
[10359.159676] [   2770]     0  2770      325      207    10240        0             0 odhcpd
[10359.167986] [   3899]     0  3899      308      137     6144        0             0 udhcpc
[10359.176302] [   3920]     0  3920      219      159     8192        0             0 odhcp6c
[10359.184712] [   4130]   453  4130      663      335    10240        0             0 dnsmasq
[10359.193111] [   4134]     0  4134      662      208    10240        0             0 dnsmasq
[10359.201520] [  21854]     0 21854      312      167     8192        0             0 miniupnpd
[10359.210127] [  21920]     0 21920      375      329     8192        0             0 sh
[10359.218108] [  22804]     0 22804      380      332    10240        0             0 sh
[10359.226084] [  23173]     0 23173      204      128     6144        0             0 atd
[10359.234147] [  23217]     0 23217      322      275    10240        0             0 sh
[10359.242133] [  23246]     0 23246      192      114     8192        0             0 hostapd_cli
[10359.250869] [  23250]     0 23250      192      145     8192        0             0 hostapd_cli
[10359.259621] [  23313]     0 23313      298      163    10240        0             0 ubus
[10359.267764] [  23380]     0 23380    74011      755    36864        0             0 php-fpm
[10359.276189] [  23381] 65534 23381    74695     8623    63488        0             0 php-fpm
[10359.284604] [  23382] 65534 23382    74686     8797    63488        0             0 php-fpm
[10359.293010] [  23461]     0 23461      876      730    12288        0         -1000 sshd
[10359.301131] [  23505]     0 23505     3281     2027    20480        0             0 syslog-ng
[10359.309689] [  23549]    52 23549     7086     6841    36864        0             0 tor
[10359.317754] [  23602]     0 23602      351      228     8192        0             0 dbus-daemon
[10359.326489] [  23781]     0 23781     1024      652    10240        0             0 bluetoothd
[10359.335159] [  23861]   200 23861      984      813    10240        0             0 mosquitto
[10359.343723] [  23914]     0 23914     4676     3660    28672        0             0 foris-ws
[10359.352193] [  23987]     0 23987     8639     7168    43008        0             0 foris-controlle
[10359.361265] [  24091]     0 24091     1351      223    14336        0             0 master
[10359.369572] [  24093]    25 24093     1384      480    14336        0             0 qmgr
[10359.377702] [  24117]     0 24117      867      439    10240        0             0 dovecot
[10359.386094] [  24148]    59 24148      809      181    10240        0             0 anvil
[10359.394337] [  24149]     0 24149      841      440    12288        0             0 log
[10359.402403] [  24151]     0 24151     1059      622    12288        0             0 config
[10359.410703] [  24159]     0 24159     1707     1344    14336        0             0 lighttpd
[10359.419195] [  24552]     0 24552      329      168    10240        0             0 umdns
[10359.427432] [  24585]     0 24585     5793     3950    30720        0             0 foris-controlle
[10359.436535] [  24637]     0 24637      418      218     6144        0             0 crond
[10359.444755] [  24676]     0 24676     6084     3999    30720        0             0 foris-controlle
[10359.453842] [  24710]     0 24710     5691     4745    28672        0             0 dev-detect-daem
[10359.462926] [  24832]     0 24832     1004      827    12288        0             0 openvpn
[10359.471302] [  25785]     0 25785     8239     6093    40960        0             0 foris-controlle
[10359.480381] [  25914]     0 25914     3651     3261    24576        0             0 python3
[10359.488764] [  25958]     0 25958     2806     2368    18432        0             0 python3
[10359.497147] [  26185]   376 26185    87510    23787   137216        0             0 mysqld
[10359.505442] [  26236]     0 26236     1880     1340    16384        0             0 sentinel-proxy
[10359.514448] [  26309]     0 26309    16566    13382    75776        0             0 Suricata-Main
[10359.523356] [  26355]     0 26355     8230     7341    43008        0             0 python3
[10359.531740] [  26441]     0 26441     1848      909    14336        0             0 sentinel-fwlogs
[10359.540813] [  26498]     0 26498     1859      909    16384        0             0 minipot [Master
[10359.549892] [  26511] 65534 26511     1852      312    14336        0             0 minipot [FTP]
[10359.558797] [  26512] 65534 26512     1899      329    14336        0             0 minipot [HTTP]
[10359.567786] [  26513] 65534 26513     1839       89    14336        0             0 minipot [Telnet
[10359.576863] [  26574]     0 26574      308      137     6144        0             0 ntpd
[10359.584994] [  26607]     0 26607     1098      405    12288        0             0 lxc-start
[10359.593582] [  26615]     0 26615      435      240    10240        0             0 init
[10359.601702] [  26666]     0 26666     1098      403    12288        0             0 lxc-start
[10359.610265] [  26669]     0 26669      435      240     8192        0             0 init
[10359.618388] [  27224]     0 27224      439       12     8192        0             0 syslogd
[10359.626794] [  27281]     0 27281      435      240     8192        0             0 crond
[10359.635001] [  27316]    25 27316     1374      515    12288        0             0 tlsmgr
[10359.643293] [  27353]     0 27353      439       12     8192        0             0 syslogd
[10359.651671] [  27421]     0 27421      436      244     8192        0             0 crond
[10359.659881] [  28076]     0 28076      435        8     8192        0             0 udhcpc
[10359.668179] [  28101]     0 28101      435        8     8192        0             0 udhcpc
[10359.676474] [  28147]  6449 28147      438        1    10240        0             0 hass_supervisor
[10359.685550] [  28149]  6449 28149      435      240    10240        0             0 logger
[10359.693843] [  28150]  6449 28150      438        1    10240        0             0 sh
[10359.701785] [  28151]  6449 28151    48180    36593   194560        0             0 hass
[10359.709901] [  28152]  6449 28152      437      258     8192        0             0 sed
[10359.717932] [  28153]  6449 28153      435      240     8192        0             0 logger
[10359.726254] [  28161]     0 28161      435      208    10240        0             0 getty
[10359.734480] [  28163]     0 28163      435      224    10240        0             0 getty
[10359.742700] [  28165]     0 28165      435      208     8192        0             0 getty
[10359.750902] [  28168]     0 28168      435      208    10240        0             0 getty
[10359.759107] [  28170]     0 28170      435      208    10240        0             0 getty
[10359.767312] [  28187]  6447 28187      439      273    10240        0             0 propagation.sh
[10359.776300] [  28244]  6447 28244      274      172    10240        0             0 inotifywait
[10359.785030] [  28245]  6447 28245      441      292     8192        0             0 propagation.sh
[10359.794020] [  28337]     0 28337      435      208     8192        0             0 getty
[10359.802225] [  28338]     0 28338      435      224    10240        0             0 getty
[10359.810423] [  28339]     0 28339      435      224     8192        0             0 getty
[10359.818628] [  28340]     0 28340      435      208    10240        0             0 getty
[10359.826831] [  28341]     0 28341      435      224    10240        0             0 getty
[10359.835035] [  28548]     0 28548      880      751    12288        0             0 sshd
[10359.843155] [  28552]     0 28552     1093      485    12288        0             0 3
[10359.851007] [  28563]     0 28563      535      309    10240        0             0 screen
[10359.859298] [  28565]     0 28565      577      411    10240        0             0 screen
[10359.867591] [  28566]     0 28566      484      322    10240        0             0 ash
[10359.875622] [  28851]     0 28851      903      776    10240        0             0 sshd
[10359.883738] [  28856]     0 28856      310      263    10240        0             0 ash
[10359.891769] [  28905]    59 28905     1435      696    14336        0             0 stats
[10359.899970] [  28906]    59 28906     1539      745    12288        0             0 auth
[10359.908091] [  28908]     0 28908     1536      750    16384        0             0 auth
[10359.916212] [   1373]  3536  1373    10774     4843    36864        0             0 kresd
[10359.924419] [  26253]     0 26253     2970     2257    18432        0             0 updater-supervi
[10359.933521] [   9537]    25  9537     1368      311    14336        0             0 pickup
[10359.941815] [   5017]     0  5017      310      139     8192        0             0 sleep
[10359.950015] [   5216]     0  5216      310      123     8192        0             0 sleep
[10359.958217] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=hass,pid=28151,uid=6449
[10359.970928] Out of memory: Killed process 28151 (hass) total-vm:192720kB, anon-rss:123412kB, file-rss:22960kB, shmem-rss:0kB, UID:6449 pgtables:190kB oom_score_adj:0
2

sysctl:

vm.admin_reserve_kbytes = 8192
vm.compact_unevictable_allowed = 1
vm.compaction_proactiveness = 20
vm.dirty_background_bytes = 0
vm.dirty_background_ratio = 10
vm.dirty_bytes = 0
vm.dirty_expire_centisecs = 3000
vm.dirty_ratio = 20
vm.dirty_writeback_centisecs = 500
vm.dirtytime_expire_seconds = 43200
vm.extfrag_threshold = 500
vm.highmem_is_dirtyable = 0
vm.laptop_mode = 0
vm.legacy_va_layout = 0
vm.lowmem_reserve_ratio = 32    0       0
vm.max_map_count = 65530
vm.min_free_kbytes = 16384
vm.mmap_min_addr = 4096
vm.mmap_rnd_bits = 8
vm.oom_dump_tasks = 1
vm.oom_kill_allocating_task = 0
vm.overcommit_kbytes = 0
vm.overcommit_memory = 0
vm.overcommit_ratio = 50
vm.page-cluster = 3
vm.page_lock_unfairness = 5
vm.panic_on_oom = 0
vm.percpu_pagelist_high_fraction = 0
vm.stat_interval = 1
vm.swappiness = 60
vm.user_reserve_kbytes = 64380
vm.vfs_cache_pressure = 100
vm.watermark_boost_factor = 15000
vm.watermark_scale_factor = 10
3

hass processes eating more than 100M sound like some bug to me.

4

Oops, I thought it was haas but it’s really home assistant. Maybe it’s normal for it to consume so much; I don’t know.

5

Yes, it is normal and has been this way before

6

I start to believe it’s a kernel bug. The system has lots of memory for page cache (about 1GB), and 2GB of swap, but the kernel decides to kill a bunch of processes instead.
I will try this script to see if it makes any difference:

while sleep 60
do
  sync
  echo 3 > /proc/sys/vm/drop_caches
  echo 1 > /proc/sys/vm/compact_memory
done

Also, do we have a different kernel version in HBK (anything but 5.15.80)?

7

This workaround works.

8

Nope. It works partially. The OOM is less frequent but still once or twice per day

9

[quote=“Shulyaka, post:2, topic:18379, full:true”]
sysctl:

vm.oom_kill_allocating_task = 0

You could try setting overcommit_memory = 2 to disable overcommit, the offending process will see an allocation failure instead, or set oom_kill_allocating_task to one so you see which application might be responsible (but in reality that is complicated and the oom’d process might not be misbehaving at all).

10

Tried that. Didn’t help.
Setting vm.oom_kill_allocating_task = 1 didn’t make the OOM-killer less frequent, it just diverted it to a random process (I’ve seen the kernel were failed at allocating even 2 pages).
Setting vm.overcommit_memory = 2 Instantly made the system unresponsive, I couldn’t run any command because fork() was failing.

My problem is not that some process uses too much memory. It is that kernel does not do its job at managing the available memory. I suspect a kernel bug, memory leak or something.

I’ve now downgraded to TOS 6.0.3 to test the 5.15.78 kernel, so far so good. Will let you know after some more testing.

11

Not exactly, it will kill the process that tries to allocate something after memory is exhausted, but that is not necessarily the one most responsible for the exhaustion, hence the whole OOM selection heuristics, however with luck the offending process can be found that way, alas not in your case…

12

Yeah, that’s what I meant, not exactly random, but once the kernel thinks the memory is exhausted, the next allocating process will be killed, even if it needs a single page.

1 Like
13

5.15.78 seems stable

14

Could you try to use the latest kernel, which is in HBK? I think, this could be caused by MLGRU, which was enabled recently and makes me wonder, why it does not work for us or nobody reported it on OpenWrt master branch, yet.

1 Like
15

Sure, will test it, but probably not today.

16

5.15.82 in HBK does still have this problem. And I believe HBT now has the same kernel.
I’m back to 5.15.78.

You could be right about MLGRU.

17

Okay, thanks for letting me know. Yeah, HBK and HBT now has the same kernel. I’ve pushed to HBK commit, which disables MGLRU, so you will see if it helps you or not. The build should be within 2 hours.

If yes, we will release another RC.

1 Like
18

Is it this one?
kernel/core/5.15.82-3-a959515bba2f3252aa25c92c34d26059[5.15.82-2-7f2a96fa546df9a95c2e216c856a67e9]

19

Yes, this is the one.

20

Already testing! If all is good, I will let you know in a few hours. If not good, then earlier:)

1 Like