[HBT] Suspected kernel memory leak

Software SW help
1

Hi! I am running HBT branch on my Omnia. For the last few months I observe the following behavior:
After about a week or 10 days of uptime, I start receiving OOM killer messages (usually killing my Home Assistant process). According to Top, the system has more enough free memory.
I do not know exactly when this problem has started, but I can tell that it was definitely fine in June.

Here is an example:

[309159.872650] cat invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
[309159.882161] CPU: 1 PID: 18211 Comm: cat Tainted: G        W         5.15.127 #0
[309159.889589] Hardware name: Marvell Armada 380/385 (Device Tree)
[309159.895620] [<c010eac8>] (unwind_backtrace) from [<c010a814>] (show_stack+0x10/0x14)
[309159.903486] [<c010a814>] (show_stack) from [<c05e722c>] (dump_stack_lvl+0x40/0x4c)
[309159.911172] [<c05e722c>] (dump_stack_lvl) from [<c023d0e8>] (dump_header+0x58/0x1bc)
[309159.919050] [<c023d0e8>] (dump_header) from [<c023daf8>] (oom_kill_process+0x19c/0x1a0)
[309159.927173] [<c023daf8>] (oom_kill_process) from [<c023e324>] (out_of_memory+0x2b0/0x388)
[309159.935472] [<c023e324>] (out_of_memory) from [<c02815c0>] (__alloc_pages+0xad4/0xfe0)
[309159.943508] [<c02815c0>] (__alloc_pages) from [<c026b5b0>] (handle_mm_fault+0x488/0xe94)
[309159.951721] [<c026b5b0>] (handle_mm_fault) from [<c0112c78>] (do_page_fault+0x118/0x2bc)
[309159.959943] [<c0112c78>] (do_page_fault) from [<c0112f88>] (do_DataAbort+0x3c/0xbc)
[309159.967718] [<c0112f88>] (do_DataAbort) from [<c0100ae8>] (__dabt_svc+0x48/0x80)
[309159.975222] Exception stack(0xd536de38 to 0xd536de80)
[309159.980379] de20:                                                       0056e021 00000fdf
[309159.988677] de40: 00000000 00000055 00000051 c0d96500 004e0000 0056e021 0008e81c 0008e021
[309159.996967] de60: d868e480 c0e04f08 00000001 d536de88 c0304fb8 c05e3530 80000013 ffffffff
[309160.005257] [<c0100ae8>] (__dabt_svc) from [<c05e3530>] (__clear_user_std+0x1c/0x68)
[309160.013139] [<c05e3530>] (__clear_user_std) from [<c0304fb8>] (padzero+0x58/0x78)
[309160.020751] [<c0304fb8>] (padzero) from [<c0305898>] (load_elf_binary+0x8c0/0x148c)
[309160.028533] [<c0305898>] (load_elf_binary) from [<c02acffc>] (bprm_execve+0x2d4/0x594)
[309160.036583] [<c02acffc>] (bprm_execve) from [<c02ad4a0>] (do_execveat_common+0x1e4/0x204)
[309160.044884] [<c02ad4a0>] (do_execveat_common) from [<c02ae254>] (sys_execve+0x30/0x38)
[309160.052925] [<c02ae254>] (sys_execve) from [<c0100060>] (ret_fast_syscall+0x0/0x48)
[309160.060703] Exception stack(0xd536dfa8 to 0xd536dff0)
[309160.065864] dfa0:                   01239a3c 01239a10 01239a3c 01239a10 01239a1c 004eeb50
[309160.074158] dfc0: 01239a3c 01239a10 004cecc6 0000000b 004ee5dc 004ee708 004ee5d0 004ee5dc
[309160.082452] dfe0: 004edad0 befa93b4 004a1920 b6f37d88
[309160.087741] Mem-Info:
[309160.090118] active_anon:248767 inactive_anon:208683 isolated_anon:0
[309160.090118]  active_file:814 inactive_file:2897 isolated_file:260
[309160.090118]  unevictable:0 dirty:0 writeback:0
[309160.090118]  slab_reclaimable:6124 slab_unreclaimable:10064
[309160.090118]  mapped:7989 shmem:260204 pagetables:1540 bounce:0
[309160.090118]  kernel_misc_reclaimable:0
[309160.090118]  free:22198 free_pcp:477 free_cma:0
[309160.127255] Node 0 active_anon:995068kB inactive_anon:834732kB active_file:3340kB inactive_file:15704kB unevictable:0kB isolated(anon):0kB isolated(file):1040kB mapped:32460kB dirty:0kB writeback:0kB shmem:1040816kB writeback_tmp:0kB kernel_stack:3232kB pagetables:6160kB all_unreclaimable? no
[309160.153345] Normal free:84112kB min:53248kB low:57344kB high:61440kB reserved_highatomic:4096KB active_anon:311472kB inactive_anon:231060kB active_file:3180kB inactive_file:9572kB unevictable:0kB writepending:0kB present:786432kB managed:753796kB mlocked:0kB bounce:0kB free_pcp:1236kB local_pcp:696kB free_cma:0kB
[309160.181261] lowmem_reserve[]: 0 10240 10240
[309160.185553] HighMem free:916kB min:22640kB low:29760kB high:36880kB reserved_highatomic:0KB active_anon:683596kB inactive_anon:603080kB active_file:552kB inactive_file:5372kB unevictable:0kB writepending:0kB present:1310720kB managed:1310720kB mlocked:0kB bounce:0kB free_pcp:740kB local_pcp:76kB free_cma:0kB
[309160.213013] lowmem_reserve[]: 0 0 0
[309160.216619] Normal: 4996*4kB (UMEH) 2734*8kB (UMEH) 1476*16kB (UMEH) 359*32kB (UMEH) 143*64kB (UEH) 1*128kB (H) 1*256kB (H) 1*512kB (H) 1*1024kB (H) 1*2048kB (H) 0*4096kB = 90080kB
[309160.232881] HighMem: 166*4kB (UM) 14*8kB (UM) 21*16kB (UM) 1*32kB (M) 3*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1336kB
[309160.245750] 263006 total pagecache pages
[309160.249780] 0 pages in swap cache
[309160.253190] Swap cache stats: add 0, delete 0, find 0/0
[309160.258529] Free swap  = 0kB
[309160.261510] Total swap = 0kB
[309160.264490] 524288 pages RAM
[309160.267478] 327680 pages HighMem/MovableOnly
[309160.271850] 8159 pages reserved
[309160.275086] Tasks state (memory values in pages):
[309160.279903] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[309160.288656] [   1034]    81  1034      272       26    10240        0             0 ubusd
[309160.296965] [   1039]     0  1039      183        6     8192        0             0 askfirst
[309160.305522] [   1099]     0  1099      409      160     8192        0             0 haveged
[309160.314005] [   1975]     0  1975      308       10     6144        0             0 watchdog
[309160.322579] [   2004]     0  2004     1538      137    16384        0             0 rpcd
[309160.330799] [   2501]     0  2501     1356      151    12288        0             0 hostapd
[309160.339284] [   2502]     0  2502     1331       50    14336        0             0 wpa_supplicant
[309160.348372] [   2562]     0  2562      431       55    10240        0             0 netifd
[309160.356760] [   2621]     0  2621     5366     2141    32768        0             0 python3
[309160.365228] [   2702]     0  2702      330       35     8192        0             0 odhcpd
[309160.373620] [   3845]     0  3845      308       10     6144        0             0 udhcpc
[309160.382014] [   3866]     0  3866      219       14     8192        0             0 odhcp6c
[309160.390496] [   4075]   453  4075      669       37    12288        0             0 dnsmasq
[309160.398979] [   4079]     0  4079      662       30    10240        0             0 dnsmasq
[309160.407465] [  21254]     0 21254      387       25     8192        0             0 miniupnpd
[309160.416111] [  21320]     0 21320      381       82    10240        0             0 sh
[309160.424157] [  22225]     0 22225      380       85     8192        0             0 sh
[309160.432205] [  23053]     0 23053      204       11     6144        0             0 atd
[309160.440341] [  23102]     0 23102      322       26     8192        0             0 sh
[309160.448387] [  23160]     0 23160      192       11     6144        0             0 hostapd_cli
[309160.457218] [  23167]     0 23167      192       11     8192        0             0 hostapd_cli
[309160.466038] [  23273]     0 23273      298       10    10240        0             0 ubus
[309160.474256] [  23390]     0 23390     1866      571    16384        0             0 lighttpd
[309160.482824] [  23431]     0 23431    74639      800    38912        0             0 php8-fpm
[309160.491384] [  23432] 65534 23432    75790     7314    65536        0             0 php8-fpm
[309160.499942] [  23433] 65534 23433    75789     7295    65536        0             0 php8-fpm
[309160.508505] [  23503]     0 23503      949       76    12288        0         -1000 sshd
[309160.516718] [  23612]    52 23612     9017     7113    40960        0             0 tor
[309160.524836] [  23647]     0 23647      349       19     8192        0             0 dbus-daemon
[309160.533659] [  23862]     0 23862     1017       11    10240        0             0 bluetoothd
[309160.542391] [  23938]   200 23938     1114      172    12288        0             0 mosquitto
[309160.551036] [  23991]     0 23991     4759     2320    28672        0             0 foris-ws
[309160.559597] [  24064]     0 24064     8927     5794    45056        0             0 foris-controlle
[309160.568768] [  24169]     0 24169     1433       64    16384        0             0 master
[309160.577154] [  24172]    25 24172     1466       67    14336        0             0 qmgr
[309160.585365] [  24199]     0 24199      867       48    12288        0             0 dovecot
[309160.593832] [  24229]    59 24229      809       26    12288        0             0 anvil
[309160.602122] [  24231]     0 24231      841       59    10240        0             0 log
[309160.610244] [  24233]     0 24233     1059      243    10240        0             0 config
[309160.618632] [  24569]     0 24569      346       37     8192        0             0 umdns
[309160.626932] [  24663]     0 24663      418       81     8192        0             0 crond
[309160.635223] [  24753]     0 24753     5878     3116    32768        0             0 foris-controlle
[309160.644393] [  24818]     0 24818     6171     3338    32768        0             0 foris-controlle
[309160.653563] [  25696]     0 25696     8512     5473    43008        0             0 foris-controlle
[309160.662727] [  25834]     0 25834     3758     2214    22528        0             0 python3
[309160.671193] [  25878]     0 25878     2806     1554    22528        0             0 python3
[309160.679671] [  26105]   376 26105    98108    25327   182272        0             0 mysqld
[309160.688065] [  26220]     0 26220    19868    13995    88064        0             0 Suricata-Main
[309160.697071] [  26338]     0 26338     1963      144    18432        0             0 sentinel-fwlogs
[309160.706246] [  26399]     0 26399     1964      121    16384        0             0 minipot [Master
[309160.715414] [  26409] 65534 26409     1934      101    14336        0             0 minipot [FTP]
[309160.724424] [  26410] 65534 26410     1984      128    14336        0             0 minipot [HTTP]
[309160.733518] [  26411] 65534 26411     1921       89    14336        0             0 minipot [Telnet
[309160.742701] [  26453]     0 26453      319       22     8192        0             0 sh
[309160.750751] [  26571]     0 26571      309       11     6144        0             0 ntpd
[309160.758971] [  26611]     0 26611     1187       78    12288        0             0 lxc-start
[309160.767626] [  26621]     0 26621      447        8    10240        0             0 init
[309160.775832] [  26677]     0 26677     1187       77    12288        0             0 lxc-start
[309160.784488] [  26682]     0 26682      439        8    10240        0             0 init
[309160.792706] [  27283]     0 27283      451       13     8192        0             0 syslogd
[309160.801195] [  27338]     0 27338      447        9    10240        0             0 crond
[309160.809505] [  27379]     0 27379      443       13    10240        0             0 syslogd
[309160.817982] [  27470]     0 27470      440        9    10240        0             0 crond
[309160.826283] [  28247]     0 28247      447        8     8192        0             0 udhcpc
[309160.834661] [  28265]     0 28265      439        8     8192        0             0 udhcpc
[309160.843062] [  28335]  6449 28335      450       11    10240        0             0 hass_supervisor
[309160.852243] [  28337]  6449 28337      447        9    10240        0             0 logger
[309160.860632] [  28338]  6449 28338      450        1     8192        0             0 sh
[309160.868675] [  28339]  6449 28339    70622    38798   284672        0             0 hass
[309160.876896] [  28342]  6449 28342      448       10     8192        0             0 sed
[309160.885019] [  28343]  6449 28343      447        9    10240        0             0 logger
[309160.893417] [  28359]     0 28359      447        9    10240        0             0 getty
[309160.901725] [  28360]  6447 28360      443        9     8192        0             0 propagation.sh
[309160.910819] [  28361]     0 28361      447        9    10240        0             0 getty
[309160.919128] [  28362]     0 28362      447        9    10240        0             0 getty
[309160.927436] [  28365]     0 28365      447        9    10240        0             0 getty
[309160.935734] [  28367]     0 28367      447        9     8192        0             0 getty
[309160.944044] [  28450]  6447 28450      252       19    10240        0             0 inotifywait
[309160.952874] [  28451]  6447 28451      445       14     8192        0             0 propagation.sh
[309160.961971] [  28579]  6447 28579      442        1     8192        0             0 hass_supervisor
[309160.971152] [  28580]  6447 28580      439        9    10240        0             0 logger
[309160.979549] [  28581]  6447 28581      442        1    10240        0             0 sh
[309160.987596] [  28582]  6447 28582      442       12     8192        0             0 sed
[309160.995719] [  28583]  6447 28583      439        9    10240        0             0 logger
[309161.004114] [  28584]  6447 28584    97003    55880   380928        0             0 hass
[309161.012339] [  28595]     0 28595      439        9     8192        0             0 getty
[309161.020654] [  28596]     0 28596      439        9    10240        0             0 getty
[309161.028972] [  28597]     0 28597      439        9     8192        0             0 getty
[309161.037283] [  28598]     0 28598      439        9     8192        0             0 getty
[309161.045584] [  28601]     0 28601      439        9    10240        0             0 getty
[309161.053897] [  28635]    25 28635     1456       67    14336        0             0 tlsmgr
[309161.062295] [  29884]    59 29884     1618      141    14336        0             0 imap-login
[309161.071041] [  29886]    59 29886     1517       76    14336        0             0 stats
[309161.079341] [  30018]  6447 30018     1335      199    16384        0             0 imap
[309161.087556] [  31396]     0 31396     2921      424    22528        0             0 sentinel-proxy
[309161.096648] [  26205] 65534 26205    75790     7260    65536        0             0 php8-fpm
[309161.105219] [  18459]     0 18459     3475      615    20480        0             0 syslog-ng
[309161.113876] [    863]     0   863     1826      206    14336        0             0 openconnect
[309161.122703] [  28263]     0 28263     1114      170    14336        0             0 openvpn
[309161.131181] [  24939]  3536 24939    11322     4720    40960        0             0 kresd
[309161.139490] [  12656]    25 12656     1450       63    14336        0             0 pickup
[309161.147895] [   9232]     0  9232     2971     1471    20480        0             0 updater-supervi
[309161.157076] [  14927]     0 14927      195        9     6144        0             0 sleep
[309161.165374] [  16890]     0 16890     1275      224    12288        0             0 wget
[309161.173597] [  17191]    25 17191     1580       94    14336        0             0 smtpd
[309161.181902] [  17194]    25 17194     1446       64    12288        0             0 proxymap
[309161.190467] [  17195]    25 17195     1449       60    14336        0             0 anvil
[309161.198770] [  18036]     0 18036      319        6    10240        0             0 sh
[309161.206813] [  18038]     0 18038     1949     1220    16384        0             0 python3
[309161.215286] [  18046]     0 18046      319        6    10240        0             0 sh
[309161.223332] [  18048]     0 18048     1949     1187    14336        0             0 python3
[309161.231802] [  18050]     0 18050      319        6    10240        0             0 sh
[309161.239845] [  18052]     0 18052      319        6    10240        0             0 sh
[309161.247894] [  18054]     0 18054      319        6     8192        0             0 sh
[309161.255930] [  18056]     0 18056     1795      877    14336        0             0 python3
[309161.264416] [  18057]     0 18057     1949     1165    18432        0             0 python3
[309161.272899] [  18058]     0 18058     1796     1211    18432        0             0 python3
[309161.281381] [  18059]     0 18059      319        6    10240        0             0 sh
[309161.289428] [  18061]     0 18061      319        6    10240        0             0 sh
[309161.297473] [  18063]     0 18063     1815      842    16384        0             0 python3
[309161.305943] [  18064]     0 18064     1695     1138    16384        0             0 python3
[309161.314424] [  18065]     0 18065      319        6    10240        0             0 sh
[309161.322469] [  18067]     0 18067     1692     1309    16384        0             0 python3
[309161.330948] [  18070]     0 18070      319        6    10240        0             0 sh
[309161.338992] [  18073]     0 18073     1692     1216    16384        0             0 python3
[309161.347471] [  18074]     0 18074      319        6    10240        0             0 sh
[309161.355507] [  18078]     0 18078     1695     1071    12288        0             0 python3
[309161.363992] [  18083]     0 18083      319        6    10240        0             0 sh
[309161.372045] [  18087]     0 18087     1695     1106    18432        0             0 python3
[309161.380528] [  18091]     0 18091      319        6     8192        0             0 sh
[309161.388574] [  18093]     0 18093     1695     1072    16384        0             0 python3
[309161.397056] [  18096]     0 18096      319        6    10240        0             0 sh
[309161.405092] [  18098]     0 18098     1544      970    16384        0             0 python3
[309161.413578] [  18102]     0 18102      319        6     8192        0             0 sh
[309161.421624] [  18105]     0 18105     1544      938    14336        0             0 python3
[309161.430110] [  18108]     0 18108      319        6     8192        0             0 sh
[309161.438161] [  18110]     0 18110     1606      809    14336        0             0 python3
[309161.446641] [  18113]     0 18113      319        6     8192        0             0 sh
[309161.454677] [  18114]     0 18114     1549      718    14336        0             0 python3
[309161.463163] [  18124]     0 18124      319        6     8192        0             0 sh
[309161.471200] [  18129]     0 18129     1454      627    14336        0             0 python3
[309161.479673] [  18131]     0 18131      319        6    10240        0             0 sh
[309161.487721] [  18133]     0 18133     1444      863    12288        0             0 python3
[309161.496195] [  18138]     0 18138      319        6    10240        0             0 sh
[309161.504227] [  18140]     0 18140     1444      791    18432        0             0 python3
[309161.512701] [  18146]     0 18146      319        6    10240        0             0 sh
[309161.520741] [  18150]     0 18150     1444      718    14336        0             0 python3
[309161.529215] [  18154]     0 18154      319        6     8192        0             0 sh
[309161.537254] [  18157]     0 18157     1312      558    14336        0             0 python3
[309161.545721] [  18166]     0 18166      319        6    10240        0             0 sh
[309161.553759] [  18169]     0 18169     1223      445    12288        0             0 python3
[309161.562228] [  18176]     0 18176      319        6    10240        0             0 sh
[309161.570257] [  18179]     0 18179     1175      354    12288        0             0 python3
[309161.578720] [  18185]     0 18185      319       19     8192        0             0 sh
[309161.586749] [  18189]     0 18189     1103      264    10240        0             0 python3
[309161.595209] [  18196]     0 18196      319        6    10240        0             0 sh
[309161.603236] [  18198]     0 18198     1109      298    14336        0             0 python3
[309161.611708] [  18204]     0 18204      523       11     6144        0             0 ip
[309161.619743] [  18205]     0 18205      312       15    10240        0             0 sh
[309161.627774] [  18207]     0 18207      319       19     8192        0             0 sh
[309161.635799] [  18210]     0 18210      319       22     8192        0             0 sh
[309161.643834] [  18211]     0 18211      161        1     2048        0             0 cat
[309161.651962] [  18212]     0 18212      180       14     6144        0             0 ubus
[309161.660177] [  18213]     0 18213      823        7     6144        0             0 python3
[309161.668654] [  18214]     0 18214     1949      896    16384        0             0 python3
[309161.677127] [  18215]     0 18215     1949      896    16384        0             0 python3
[309161.685591] [  18216]     0 18216     1949      896    14336        0             0 python3
[309161.694059] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=hass,pid=28584,uid=6447
[309161.706864] Out of memory: Killed process 28584 (hass) total-vm:388012kB, anon-rss:223520kB, file-rss:0kB, shmem-rss:0kB, UID:6447 pgtables:372kB oom_score_adj:0

(hass is not to be confused with haas)

2

Looks more like your system ran out of memory, probably because of hass. I assume this doesn’t happen if hass is not active?

3

I used this tool to parse:
https://www.carstengrohmann.de/oom/OOMAnalyser.html

To summarize:
43% of memory is in use, which means 57% is supposed to be free.
hass uses 10%.
cat wanted to allocate just 4 kBytes of memory which triggered OOM Killer

1 Like
4

Edit: Oh no, I misread, I thought cat wanted 4 MB instead of 4 kB. In that case, I don’t have an answer. However, I’ll leave it here, it might still be useful.

I think this is the problem. cat wants to allocate a pretty large chunk of memory all at once (4 kB). But as you can see, there are no more 4 kB chunks left in the memory.

So this is not a true out of memory error in the sense that the RAM is full. It is a memory fragmentation problem - some app fragments the memory so much that it is in a state where no consecutive 4 kB are free. I guess hass could be the culprit.

To defragment the currently occupied RAM, it is needed to restart some RAM-hungry apps. If you don’t mind restarting hass once a day, I think this could really help. Just set up a cron job to do that.

5

Another log from today:

[652124.436628] ntpd invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
[652124.446156] CPU: 1 PID: 26557 Comm: ntpd Tainted: G        W         5.15.127 #0
[652124.453660] Hardware name: Marvell Armada 380/385 (Device Tree)
[652124.459682] [<c010eac8>] (unwind_backtrace) from [<c010a814>] (show_stack+0x10/0x14)
[652124.467543] [<c010a814>] (show_stack) from [<c05e722c>] (dump_stack_lvl+0x40/0x4c)
[652124.475221] [<c05e722c>] (dump_stack_lvl) from [<c023d0e8>] (dump_header+0x58/0x1bc)
[652124.483076] [<c023d0e8>] (dump_header) from [<c023daf8>] (oom_kill_process+0x19c/0x1a0)
[652124.491188] [<c023daf8>] (oom_kill_process) from [<c023e324>] (out_of_memory+0x2b0/0x388)
[652124.499475] [<c023e324>] (out_of_memory) from [<c02815c0>] (__alloc_pages+0xad4/0xfe0)
[652124.507501] [<c02815c0>] (__alloc_pages) from [<c0238f14>] (pagecache_get_page+0x148/0x460)
[652124.515961] [<c0238f14>] (pagecache_get_page) from [<c0239868>] (filemap_fault+0x63c/0x84c)
[652124.524420] [<c0239868>] (filemap_fault) from [<c02668a4>] (__do_fault+0x38/0x14c)
[652124.532103] [<c02668a4>] (__do_fault) from [<c026bc5c>] (handle_mm_fault+0xb34/0xe94)
[652124.540044] [<c026bc5c>] (handle_mm_fault) from [<c0112c78>] (do_page_fault+0x118/0x2bc)
[652124.548250] [<c0112c78>] (do_page_fault) from [<c0112f88>] (do_DataAbort+0x3c/0xbc)
[652124.556015] [<c0112f88>] (do_DataAbort) from [<c0100e5c>] (__dabt_usr+0x3c/0x40)
[652124.563519] Exception stack(0xc7845fb0 to 0xc7845ff8)
[652124.568669] 5fa0:                                     00000000 005615d3 0000002c 00000000
[652124.576954] 5fc0: beebdb00 005615d3 beebdbda beebda00 00000000 00000000 beebd9fc beebda28
[652124.585243] 5fe0: 00000018 beebd6e0 b6f23640 b6ef323c 80000010 ffffffff
[652124.592004] Mem-Info:
[652124.594380] active_anon:201912 inactive_anon:262111 isolated_anon:0
[652124.594380]  active_file:29 inactive_file:1608 isolated_file:160
[652124.594380]  unevictable:0 dirty:0 writeback:0
[652124.594380]  slab_reclaimable:5182 slab_unreclaimable:10024
[652124.594380]  mapped:15962 shmem:218681 pagetables:1449 bounce:0
[652124.594380]  kernel_misc_reclaimable:0
[652124.594380]  free:15254 free_pcp:174 free_cma:0
[652124.631498] Node 0 active_anon:807648kB inactive_anon:1048444kB active_file:388kB inactive_file:5992kB unevictable:0kB isolated(anon):0kB isolated(file):640kB mapped:63848kB dirty:0kB writeback:0kB shmem:874724kB writeback_tmp:0kB kernel_stack:2856kB pagetables:5796kB all_unreclaimable? no
[652124.657337] Normal free:57192kB min:16384kB low:20480kB high:24576kB reserved_highatomic:0KB active_anon:160516kB inactive_anon:411412kB active_file:48kB inactive_file:1192kB unevictable:0kB writepending:0kB present:786432kB managed:753796kB mlocked:0kB bounce:0kB free_pcp:212kB local_pcp:20kB free_cma:0kB
[652124.684634] lowmem_reserve[]: 0 10240 10240
[652124.688927] HighMem free:3824kB min:22640kB low:29760kB high:36880kB reserved_highatomic:0KB active_anon:647132kB inactive_anon:637032kB active_file:348kB inactive_file:4752kB unevictable:0kB writepending:0kB present:1310720kB managed:1310720kB mlocked:0kB bounce:0kB free_pcp:680kB local_pcp:360kB free_cma:0kB
[652124.716560] lowmem_reserve[]: 0 0 0
[652124.720157] Normal: 2063*4kB (UME) 1411*8kB (UME) 967*16kB (UME) 133*32kB (UME) 139*64kB (UME) 56*128kB (UM) 6*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 56868kB
[652124.735195] HighMem: 21*4kB (U) 30*8kB (U) 11*16kB (U) 2*32kB (U) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 564kB
[652124.747193] 221082 total pagecache pages
[652124.751226] 0 pages in swap cache
[652124.754647] Swap cache stats: add 0, delete 0, find 0/0
[652124.759973] Free swap  = 0kB
[652124.762955] Total swap = 0kB
[652124.765930] 524288 pages RAM
[652124.768902] 327680 pages HighMem/MovableOnly
[652124.773279] 8159 pages reserved
[652124.776519] Tasks state (memory values in pages):
[652124.781320] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[652124.790066] [   1034]    81  1034      270       26     8192        0             0 ubusd
[652124.798360] [   1039]     0  1039      183        6     8192        0             0 askfirst
[652124.806916] [   1099]     0  1099      409      160     8192        0             0 haveged
[652124.815384] [   1975]     0  1975      308       10     6144        0             0 watchdog
[652124.823935] [   2004]     0  2004     1538      127    18432        0             0 rpcd
[652124.832133] [   2501]     0  2501     1355       76    16384        0             0 hostapd
[652124.840597] [   2502]     0  2502     1331       50    14336        0             0 wpa_supplicant
[652124.849670] [   2562]     0  2562      427       61     8192        0             0 netifd
[652124.858046] [   2620]     0  2620     5323     2163    30720        0             0 python3
[652124.866559] [   2701]     0  2701      330       35     8192        0             0 odhcpd
[652124.874940] [   3871]     0  3871      308       11     8192        0             0 udhcpc
[652124.883317] [   3892]     0  3892      219       14     6144        0             0 odhcp6c
[652124.891776] [   4359]   453  4359      671       38    12288        0             0 dnsmasq
[652124.900243] [   4362]     0  4362      662       30    10240        0             0 dnsmasq
[652124.908708] [  21177]     0 21177      390       25     8192        0             0 miniupnpd
[652124.917348] [  21243]     0 21243      381       82     8192        0             0 sh
[652124.925375] [  22168]     0 22168      375       79    10240        0             0 sh
[652124.933406] [  23000]     0 23000      204       11     6144        0             0 atd
[652124.941518] [  23044]     0 23044      322       25     8192        0             0 sh
[652124.949550] [  23152]     0 23152      298       10    10240        0             0 ubus
[652124.957756] [  23333]     0 23333     1849      426    16384        0             0 lighttpd
[652124.966308] [  23374]     0 23374    74639      800    38912        0             0 php8-fpm
[652124.974859] [  23375] 65534 23375    75800    10638    79872        0             0 php8-fpm
[652124.983418] [  23376] 65534 23376    75802    10728    79872        0             0 php8-fpm
[652124.991966] [  23455]     0 23455      949       76    14336        0         -1000 sshd
[652125.000170] [  23560]    52 23560    14717    11839    67584        0             0 tor
[652125.008289] [  23588]     0 23588      349       20     8192        0             0 dbus-daemon
[652125.017108] [  23805]     0 23805     1017       32    10240        0             0 bluetoothd
[652125.025834] [  23881]   200 23881     1179      227    12288        0             0 mosquitto
[652125.034473] [  23934]     0 23934     4758     2320    28672        0             0 foris-ws
[652125.043024] [  24007]     0 24007     8921     5788    43008        0             0 foris-controlle
[652125.052184] [  24110]     0 24110     1433       64    14336        0             0 master
[652125.060557] [  24113]    25 24113     1467       68    12288        0             0 qmgr
[652125.068758] [  24136]     0 24136      867       49    12288        0             0 dovecot
[652125.077224] [  24178]    59 24178      809       26    10240        0             0 anvil
[652125.085514] [  24179]     0 24179      841       60    10240        0             0 log
[652125.093632] [  24182]     0 24182     1059      243    14336        0             0 config
[652125.102007] [  24522]     0 24522      346       37    10240        0             0 umdns
[652125.110299] [  24602]     0 24602      418       80     8192        0             0 crond
[652125.118599] [  24655]     0 24655     5894     3255    32768        0             0 dev-detect-daem
[652125.127768] [  24802]     0 24802     1171      191    12288        0             0 openvpn
[652125.136242] [  24821]     0 24821     5876     3129    30720        0             0 foris-controlle
[652125.145406] [  24943]     0 24943     6171     3338    32768        0             0 foris-controlle
[652125.154569] [  25735]     0 25735     8510     5471    43008        0             0 foris-controlle
[652125.163735] [  25808]     0 25808     3761     2146    22528        0             0 python3
[652125.172205] [  25839]     0 25839     2806     1520    18432        0             0 python3
[652125.180666] [  26079]   376 26079    98500    38354   219136        0             0 mysqld
[652125.189055] [  26205]     0 26205    21002    13708    94208        0             0 Suricata-Main
[652125.198091] [  26248]     0 26248     8346     5622    45056        0             0 python3
[652125.206560] [  26335]     0 26335     1930      107    16384        0             0 sentinel-fwlogs
[652125.215722] [  26399]     0 26399     1964      119    18432        0             0 minipot [Master
[652125.224888] [  26420] 65534 26420     1934      101    16384        0             0 minipot [FTP]
[652125.233876] [  26421] 65534 26421     1984      128    16384        0             0 minipot [HTTP]
[652125.242950] [  26422] 65534 26422     1921       89    16384        0             0 minipot [Telnet
[652125.252106] [  26458]     0 26458      319       22    10240        0             0 sh
[652125.260134] [  26557]     0 26557      308       11     6144        0             0 ntpd
[652125.268337] [  26667]     0 26667     1187       94    14336        0             0 lxc-start
[652125.276975] [  26671]     0 26671      439        8    10240        0             0 init
[652125.285181] [  27381]     0 27381      443       13     8192        0             0 syslogd
[652125.293648] [  27493]     0 27493      443       11    10240        0             0 crond
[652125.301935] [  28217]     0 28217     2310      359    18432        0             0 sentinel-proxy
[652125.311012] [  28281]     0 28281      439        8     8192        0             0 udhcpc
[652125.319411] [  28363]  6447 28363      443        9     8192        0             0 propagation.sh
[652125.328499] [  28415]  6447 28415      252       19    10240        0             0 inotifywait
[652125.337315] [  28416]  6447 28416      445       14     8192        0             0 propagation.sh
[652125.346399] [  28523]    25 28523     1458       68    12288        0             0 tlsmgr
[652125.354788] [  28580]     0 28580      439        9    10240        0             0 getty
[652125.363095] [  28581]     0 28581      439        9    10240        0             0 getty
[652125.371390] [  28582]     0 28582      439        9    10240        0             0 getty
[652125.379706] [  28583]     0 28583      439        9     8192        0             0 getty
[652125.388009] [  28584]     0 28584      439        9     8192        0             0 getty
[652125.396305] [  29652]    59 29652     1517       76    14336        0             0 stats
[652125.404611] [   9300] 65534  9300    75801    10600    79872        0             0 php8-fpm
[652125.413173] [  30247]     0 30247      483       86    10240        0             0 screen
[652125.421547] [  30248]     0 30248      484       57    10240        0             0 ash
[652125.429669] [   7333]  6447  7333      482       55    10240        0             0 ash
[652125.437796] [   8932]  6447  8932      442        1    10240        0             0 hass_supervisor
[652125.446967] [   8933]  6447  8933      439        9     8192        0             0 logger
[652125.455353] [   8934]  6447  8934      442        1     8192        0             0 sh
[652125.463400] [   8935]  6447  8935   137319    99146   555008        0             0 hass
[652125.471610] [   8938]  6447  8938      442       12    10240        0             0 sed
[652125.479744] [   8939]  6447  8939      439        9    10240        0             0 logger
[652125.488125] [   9699]    59  9699     1618      141    14336        0             0 imap-login
[652125.496862] [   9816]  6447  9816     1353      171    14336        0             0 imap
[652125.505079] [  28723]     0 28723     3513      641    22528        0             0 syslog-ng
[652125.513730] [  20089]     0 20089      952       81    10240        0             0 sshd
[652125.521937] [  20091]     0 20091      310        6    10240        0             0 ash
[652125.530069] [  20205]     0 20205      952       83    10240        0             0 sshd
[652125.538285] [  20210]     0 20210      310       14     8192        0             0 ash
[652125.546418] [  20422]     0 20422      952       84    12288        0             0 sshd
[652125.554630] [  20427]     0 20427      310       14    10240        0             0 ash
[652125.562754] [  20694]     0 20694      308        6     8192        0             0 ash
[652125.570870] [  20695]     0 20695      308       11     6144        0             0 grep
[652125.579075] [  20696]     0 20696      308        6     6144        0             0 sh
[652125.587118] [  20697]     0 20697      309       12    10240        0             0 grep
[652125.595330] [  20698]     0 20698      309       12    10240        0             0 sed
[652125.603455] [  20699]     0 20699      311       13     8192        0             0 tail
[652125.611659] [  13372]     0 13372     1187       78    12288        0             0 lxc-start
[652125.620305] [  13379]     0 13379      447        9    10240        0             0 init
[652125.628519] [  13611]     0 13611      451       13     8192        0             0 syslogd
[652125.636995] [  13638]     0 13638      448       10    10240        0             0 crond
[652125.645301] [  13788]     0 13788      447        8     8192        0             0 udhcpc
[652125.653686] [  13828]  6449 13828      450       11    10240        0             0 hass_supervisor
[652125.662851] [  13829]  6449 13829      447        9    10240        0             0 logger
[652125.671225] [  13830]  6449 13830      450        1    10240        0             0 sh
[652125.679258] [  13832]  6449 13832    72897    40191   292864        0             0 hass
[652125.687476] [  13838]  6449 13838      450       12    10240        0             0 sed
[652125.695612] [  13839]  6449 13839      447        9     8192        0             0 logger
[652125.704018] [  13845]     0 13845      447        9    10240        0             0 getty
[652125.712315] [  13846]     0 13846      447        9     8192        0             0 getty
[652125.720604] [  13847]     0 13847      447        9    10240        0             0 getty
[652125.728909] [  13848]     0 13848      447        9    10240        0             0 getty
[652125.737201] [  13849]     0 13849      447        9     8192        0             0 getty
[652125.745499] [  15296]     0 15296     1826      164    16384        0             0 openconnect
[652125.754316] [  17268]  3536 17268    11140     4875    43008        0             0 kresd
[652125.762622] [  12547]    25 12547     1450       63    12288        0             0 pickup
[652125.771004] [  16884]     0 16884     2971     1471    18432        0             0 updater-supervi
[652125.780178] [  15347]     0 15347      195        9     8192        0             0 sleep
[652125.788479] [  19963]     0 19963      461        9     6144        0             0 ip
[652125.796518] [  19967]     0 19967      319       22     8192        0             0 sh
[652125.804553] [  19969]     0 19969      281        1     6144        0             0 cat
[652125.812678] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=hass,pid=8935,uid=6447
[652125.825437] Out of memory: Killed process 8935 (hass) total-vm:549276kB, anon-rss:396584kB, file-rss:0kB, shmem-rss:0kB, UID:6447 pgtables:542kB oom_score_adj:0
root@shulyaka:~# uptime
 11:09:18 up 7 days, 13:27,  load average: 0.35, 0.60, 1.50
root@shulyaka:~# free
              total        used        free      shared  buff/cache   available
Mem:        2064516      720568      201736      875920     1142212      349140
Swap:             0           0           0
6

Is it normal that the OS has accumulated over 850M of cache that it is unable to clear?

root@shulyaka:~# free
              total        used        free      shared  buff/cache   available
Mem:        2064516      374288      465176      835652     1225052      734348
Swap:             0           0           0
root@shulyaka:~# echo 3 > /proc/sys/vm/drop_caches
root@shulyaka:~# free
              total        used        free      shared  buff/cache   available
Mem:        2064516      367044      818536      836072      878936      759680
Swap:             0           0           0
7

This is still happening as of Turris OS 7.0

8

“buff/cache” that’s not “available” is typically from tmpfs contents, which on Turris is mainly /tmp and /var.

1 Like