Suricata uses all disk space in /tmp

Hi,
few weeks ago I started having an issue that after few days since restart DNS stops resolving and reboot always fixes that. Until now I haven’t had enough time to check this but now I found out that issue is probably because of no free space available at /tmp mount.

The biggest folder in this mount point is /tmp/log/suricata/ with eve.json that grows really quickly. After quick search here I noticed that Suricata gets installed when Device detection package is enabled.

It seems that this file is not truncated at all and grows until a restart clears everything within /tmp. Is this common behavior or I am missing something, eg. logrotate action for this file?

Vl.

If you fill /tmp/, knot-resolver will stop working with very high probability. I think I’ve seen other similar stories to yours with the experimental “device detection” checkbox, here around the forum.

Yes, yes, thank you for confirmation. I just came home and exactly same issue as yesterday, full /tmp and DNS resolution was not working.

Also it seems that device detection itself is also not working as I just connected new phone to the network and no notification was sent.

OK, so unfortunately the best way to resolve this for now seems to be by disabling device detection and forget about this feature.

I am little bit curious if anybody else who is using device detection is running into this issue?

I have been having the same problem with the Turris Omnia – the problem of DNS resolving fine for a few days, and then at some point within a few days or a week simply not resolving. This has been happening quite a bit since I got the T.O. 2 or 3 months ago.

In my case, as a temporary fix when this happens, I sometimes (a) plug my cable modem directly into my desktop computer, or I (b) plug the modem (and computer) into my old router, or © I just unplug the Turris Omnia for 20 seconds or so, and then plug it back in (i.e., reboot).

Doing any one of those things fixes my connectivity, but they’re all temporary fixes and I’m getting tired of performing these gyrations.

I do not have Device detection activated (foris -> Updater -> Package lists -> Device detection), and I’ve never had it activated.

Therefore I do not have a /tmp/log/suricata file. What I do have in my /tmp folder that may be causing a similar problem is a large “luci-indexcache” file:

-rw-------    1 root     root        106105 Nov 28 02:08 luci-indexcache

This file is by far the largest in my /tmp directory. A cursory web search shows a bunch of advice saying to remove it to solve various problems.

But before I go removing it, I’d like to ask here what this file is for, should I remove it, is it OK to remove it, and what might the consequences be, if any, of removing it.

And could this be what is filling up too much space in the /tmp dir causing DNS not to resolve?

Why does the router even stop resolving DNS when a /tmp file gets too big in the first place?

Also, what permanent solution should I apply to this problem, as opposed to having to regularly SSH into the router and manually remove the file (if that is even what I need to do as a temporary solution)? My DNS at the moment (even with this large file) is working OK. And maybe – as far as i know – the luci-indexcache file is not the problem.

This is a bit technical thing. The caching of DNS in knot-resolver is done via LMDB. That mmaps the whole cache, 20 MiB by default on Omnia IIRC, and Linux over-commits the map as usual, only lazily allocating pages when they get written to. So when space runs out, during regular memory access to the mmapped file, and kernel finds it can’t fulfill the promise, it just kills the process (via SIGBUS).

Hi,
have you edited the configuration file of Suricata somehow?
This used to be an issue with the first few versions of Suricata, but that was fixed (changed in configuration file) 4 months ago, so that should be released already. But if you changed the configuration file, following updates won’t overwrite the file to distribution version. So that might be the cause.

Anyway, eve.json log can be easily disabled (and is disabled by default), in configuration file /etc/suricata/output_conf.d/common.yaml

- eve-log:
    enabled: no #!this is important!
    filetype: regular
    filename: eve.json

See distribution file: https://gitlab.labs.nic.cz/turris/turris-os-packages/blob/test/net/suricata/files/output_common.yaml#L10

Hi,
when I posted this issue I’ve disabled the Device detection in Updater for a few days and then enabled again hoped that clean installation will be ok, but today I ran again into having full /tmp mount.

I’ve checked mentioned configuration file /etc/suricata/output_conf.d/common.yaml but it looked as expected eve-log option is disabled here. So I checked other config files in suricata folder and I found out that in file /etc/suricata/suricata.yaml is that section outputs again, but this time with eve-log enabled. But on https://gitlab.labs.nic.cz/turris/turris-os-packages/blob/test/net/suricata/files/suricata.yaml it is not.

Last change date of suricata.yaml file is Mar 13 2017, but I am not aware of any modification from my side, I found out about suricata’s existence just after this issue with full /tmp mount.

Could it be part of the initial default installation which was then changed?

That’s weird. It seems you have some very old version of suricata.yaml (the output section which contains eve-log was separated to config files in /etc/suricata/output_conf.d/ a long time ago).

The problem is that /etc/suricata/suricata.yaml is marked as configuration file. When you edit the file (or updater for any reason thinks so), it doesn’t update the file nor removes it when uninstalling the package. But it puts the distribution file there too, with -opkg suffix.

So please have a look if there is /etc/suricata/suricata.yaml-opkg file. If so, you can do:

mv /etc/suricata/suricata.yaml-opkg /etc/suricata/suricata.yaml; /etc/init.d/suricata restart

Alternatively, if you want to be sure, you can disable dev detection (that will remove suricata package), then do rm -rf /etc/suricata and enable device detection again. That way, you should end up with the distribution version of configuration files.

There might be a bug in updater, that doesn’t update the file even if the file was not modified by user. I already have some hint that this could be happening, I will discuss that with the developer of updater.

Hi,
I had finally more time to look again into this again and as you mentioned I really had suricata.yaml-opkg config file along with to actually used suricata.yaml one. To be sure I followed your advice to uninstall, remove config folder and install again and no eve.json file anymore, thank you!

But I am still not able to start receiving notifications about newly detected devices on the network again.

I noticed that in file /usr/share/pakon-dev-detect/known_macs some of the MAC addresses are saved, but for example I connected to the network new laptop and no notification was sent nor I see it’s MAC address in that known_macs file. In suricata configuration I see definition of socket file for this and socket itself /tmp/suricata/dev_detect.sock exists.

How is the actual python script for device detection started please?

Vl.