Turris OS 3.11.12 is out!

Dear Turris users,

We released Turris OS 3.11.12 from RC to everyone. This is a security release for git as there were found many CVEs. If you are interested in security vulnerabilities, which were found, you can take a look at openwall.com

Changelog:

Enjoy the release and Merry Christmas!

2 Likes

4 posts were split to a new topic: Turris OS 3.11.12 not solved issues

Does it fixes the various DNS issues coming from 3.11.11 or not ?

No, we have no idea what they are so far. (Not reproduced for us and almost no information from the two people who reported it.)

1 Like

Update from 3.11.10 went smoothly on Turris Omnia even though my memory has been nearly eaten up after 15 days of runtime (left 80MiB out of 1GiB). I will reply here again in a couple of days if memory is eaten up again on 3.11.12.

So it is me again: memory is eaten up again after only 3 days of uptime:

Quite some share of this is used by foris:
{foris-ws} /usr/bin/python3.6 /usr/bin/foris-ws -a ubus --host 127.0.0.1 --port 9080 ubus --path /var/run/ubus.sock 2%
{foris-controlle} /usr/bin/python3.6 /usr/bin/foris-controller -b openwrt -C /var/run/foris-controller-client.sock ubus --path /var/run/ubus.sock 2%
{foris} /usr/bin/python3.6 /usr/bin/foris -s flup -a config -b ubus --bus-socket /var/run/ubus.sock 2%
{foris-controlle} /usr/bin/python3.6 /usr/bin/foris-controller -b openwrt -C /var/run/foris-controller-client.sock ubus --path /var/run/ubus.sock 2%
{foris-controlle} /usr/bin/python3.6 /usr/bin/foris-controller -b openwrt -C /var/run/foris-controller-client.sock ubus --path /var/run/ubus.sock 2%
{foris-controlle} /usr/bin/python3.6 /usr/bin/foris-controller -b openwrt -C /var/run/foris-controller-client.sock ubus --path /var/run/ubus.sock 2%
{foris-controlle} /usr/bin/python3.6 /usr/bin/foris-controller -b openwrt -C /var/run/foris-controller-client.sock ubus --path /var/run/ubus.sock 2%
{foris-controlle} /usr/bin/python3.6 /usr/bin/foris-controller -b openwrt -C /var/run/foris-controller-client.sock ubus --path /var/run/ubus.sock 2%
{foris-controlle} /usr/bin/python3.6 /usr/bin/foris-controller -b openwrt -C /var/run/foris-controller-client.sock ubus --path /var/run/ubus.sock 2%
{foris-controlle} /usr/bin/python3.6 /usr/bin/foris-controller -b openwrt -C /var/run/foris-controller-client.sock ubus --path /var/run/ubus.sock 2%
{foris-controlle} /usr/bin/python3.6 /usr/bin/foris-controller -b openwrt -C /var/run/foris-controller-client.sock ubus --path /var/run/ubus.sock 2%
{foris-controlle} /usr/bin/python3.6 /usr/bin/foris-controller -b openwrt -C /var/run/foris-controller-client.sock ubus --path /var/run/ubus.sock 2%
{foris-controlle} /usr/bin/python3.6 /usr/bin/foris-controller -b openwrt -C /var/run/foris-controller-client.sock ubus --path /var/run/ubus.sock 2%
{foris-controlle} /usr/bin/python3.6 /usr/bin/foris-controller -b openwrt -C /var/run/foris-controller-client.sock ubus --path /var/run/ubus.sock 2%
{foris-controlle} /usr/bin/python3.6 /usr/bin/foris-controller -b openwrt -C /var/run/foris-controller-client.sock ubus --path /var/run/ubus.sock 2%
{foris-controlle} /usr/bin/python3.6 /usr/bin/foris-controller -b openwrt -C /var/run/foris-controller-client.sock ubus --path /var/run/ubus.sock 2%
{foris-controlle} /usr/bin/python3.6 /usr/bin/foris-controller -b openwrt -C /var/run/foris-controller-client.sock ubus --path /var/run/ubus.sock 2%
{foris-controlle} /usr/bin/python3.6 /usr/bin/foris-controller -b openwrt -C /var/run/foris-controller-client.sock ubus --path /var/run/ubus.sock 2%
{foris-controlle} /usr/bin/python3.6 /usr/bin/foris-controller -b openwrt -C /var/run/foris-controller-client.sock ubus --path /var/run/ubus.sock 2%
{foris-controlle} /usr/bin/python3.6 /usr/bin/foris-controller -b openwrt -C /var/run/foris-controller-client.sock ubus --path /var/run/ubus.sock 2%
{foris-controlle} /usr/bin/python3.6 /usr/bin/foris-controller -b openwrt -C /var/run/foris-controller-client.sock ubus --path /var/run/ubus.sock 2%

So foris, which I am not really using, has eaten up 42% of the memory. WHY?!
Another 11% is eaten up by “normal” router services, even if I really don’t understand why kresd is using 7% - dnsmasq on vanilla OpenWrt is using 0%…
But there are still 43% of memory (equals 430 MiB!) used by temporary files. On vanilla OpenWrt it is half of this amount - have a look at the following figure


This WRT32x has the same packages installed, but processes running on it are using 0 (in words: zero!) percent of its memory, so I assume the temporary files are using ca. 230 MiB.

Can someone please shed light on this?

If I remember correctly:

  • these charts in luci are useless, because they lump disk cache together with really used memory;
  • these foris-controller stuff are threads in the same process, i.e. those 2% are the same and don’t “add up”.

If you want to measure memory usage, I’d e.g. recommend using htop or atop (over ssh). Even the free command through busybox seems really crippled by missing the most useful column (“available”).

1 Like

It seems, that memory is used for disk cache - https://forum.turris.cz/t/memory-leak-in-turris-system/1047/4.
You can check from command line - connect using SSH and start command:

free

Ok, understood.
But why is that much disk cache used? And why is foris-controlle (which is obviously missing a “r”) stuff using that much threads?

  • why? We’re getting a bit off topic, so let me be short. One way to look at it is that completely unused memory is of no use, so it’s used for disk cache (which is dropped instantly in case more memory is needed).
  • htop shows the full name for me.
2 Likes

Unfortunately, this does not explain why it does not appear on a Linksys where the RAM is available. Besides the updater gives me the message by mail that the RAM is full and it can’t work anymore. Network then also stops.
Cache I would therefore exclude.

Sounds you really are out of RAM. On Turris that typically happens through filled up /tmp, I think. Example report from today: Pakon/Surikata traffic whitelisting
(There might be more common ways to run out of RAM; I just don’t really know, you’ll most likely have to investigate a bit more.)

I have activated Pakon/Suricata via the updater,
but the fact that it fills up all the RAM right after startup is odd.
Should I test it to deactivate Pakon and then reboot?
I’m actually on 4.0.3 but the release seems to affect both, so I’ll answer here

I have found the cause (probably)
it’s Pakon, uninstalled it, after several attempts I managed to do it without the RAM getting full… Had to disable and stop it in LuCi.
After that the RAM jumped directly back to 200MB, it’s still a bit weird because I almost didn’t activate any services, but everything works again.
The rest will probably be system etc.

Where the fault lies here…no idea

But I hope this workaround will help for now.

Best regards

There is simple explanation. I am not saying that Foris has small footprint, it has not. On the other hand it is not hot as it seems. Child processes share a lot of memory pages so although new child seemingly consume same amount of memory it in reality does not. Example with foris-controller on my Mox (Turris OS 4.0.3):
There are four processed for foris-controller. Reported memory consumption of each in percents is 3,7%, 3,3%, 2.2% and 2.2%. That is in total 11.4%. Total consumption of memory of system with foris-controller is 176M and without it 108M. That gives me 68M of consumptions. My Mox is 1G version so I have about 997M of RAM. 11.4% of that memory is 113.66M. You can see that just sum of percents of memory consumption is is twice as big. (This is just to show that summing percents results gives incorrect result)

Another reason is that Turris compared to plain OpenWrt does more file operations and so IO cache is used more. Specially we have syslog-ng instead of busybox’s syslogd. That is difference between memory buffer and file in memory. Luci sums those two memory usages together. On my 1G Mox running for few hours it is around 350M memory usage. With longer run cache usage rises and Luci can report less memory. The important part is to check real memory usage.

If memory is really exhausted then it is common a case of some problem triggering a lot of logs and filling memory with that faster than logrotate can handle it. Together with this you should check if cron is running as otherwise logrotate is not called and your router slowly logs itself to abyss.

1 Like
1 Like