Trying new data collecting system Sentinel

As you might have heard, we are working on redesign of old uCollect system that was collecting firewall logs and various data about people trying to attack your router. In Turris OS 3.11 we will make first parts of our new software – called Sentinel – available. So we would like to ask you to test it out. :slight_smile:

How to do that? First of all you have to disable the old system. You can do that simply in updater tab of Foris web interface where you uncheck Data Collection list.

Installation of the new one is a little trickier and you need LuCI/CLI to do that. First update list of packages and afterwards install sentinel-dynfw-client and sentinel-minipot packages with these commands:

   opkg update
   opkg install sentinel-dynfw-client sentinel-minipot

We don’t have fancy list in updater yet as it is highly experimental at the moment. There is also no web UI that you can check to see whether it works. The only way to do so is to play an attacker and try connecting via telnet to your internet facing IP. If everything works, you should get prompt asking for credentials. Those credentials will be logged, be aware!

Also you can check that you are getting the list of attackers using command ipset -L turris-sn-wan-input-block. But don’t search there for your IP. Attacking one router is not evil enough to get you on the list.

Our current plan is to get more testers, put the system under some stress and fine tune the new detection algorithms. Over the time, we will phase out the old system and replace it completely with the new one.

If you are interested in more details about how we are redesigning our data collection system and you happen to speak Czech and be in Prague next week, we are going to have a talk about it on IT18 conference.

6 Likes

Is this correct?

First attempt

Souhrn
root@turris:~# opkg install sentinel-dynfw-client sentinel-minipot
Installing sentinel-dynfw-client (1.0-3) to root...
Downloading https://repo.turris.cz/turris-rc/packages//turrispackages/sentinel-dynfw-client_1.0-3_mpc85xx.ipk
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  4730  100  4730    0     0  66619      0 --:--:-- --:--:-- --:--:-- 67571
Installing libzmq-nc (4.2.2-1) to root...
Downloading https://repo.turris.cz/turris-rc/packages//packages/libzmq-nc_4.2.2-1_mpc85xx.ipk
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  235k  100  235k    0     0  1461k      0 --:--:-- --:--:-- --:--:-- 1470k
Installing python3-zmq (16.0.2-2) to root...
Downloading https://repo.turris.cz/turris-rc/packages//turrispackages/python3-zmq_16.0.2-2_mpc85xx.ipk
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  322k  100  322k    0     0  1680k      0 --:--:-- --:--:-- --:--:-- 1688k
Installing python3-msgpack (0.5.6-2) to root...
Downloading https://repo.turris.cz/turris-rc/packages//turrispackages/python3-msgpack_0.5.6-2_mpc85xx.ipk
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 13100  100 13100    0     0   180k      0 --:--:-- --:--:-- --:--:--  182k
Configuring libzmq-nc.
Configuring python3-zmq.
Configuring python3-msgpack.
Configuring sentinel-dynfw-client.
Installing sentinel-minipot (1-6) to root...
Downloading https://repo.turris.cz/turris-rc/packages//turrispackages/sentinel-minipot_1-6_mpc85xx.ipk
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 10068  100 10068    0     0   132k      0 --:--:-- --:--:-- --:--:--  134k
Installing czmq (20171102-1) to root...
Downloading https://repo.turris.cz/turris-rc/packages//turrispackages/czmq_20171102-1_mpc85xx.ipk
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  168k  100  168k    0     0  1267k      0 --:--:-- --:--:-- --:--:-- 1276k
Installing msgpack-c (2.1.5-2) to root...
Downloading https://repo.turris.cz/turris-rc/packages//turrispackages/msgpack-c_2.1.5-2_mpc85xx.ipk
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 15384  100 15384    0     0   165k      0 --:--:-- --:--:-- --:--:--  165k
Installing libpaho-mqtt-c (1.2.0-1) to root...
Downloading https://repo.turris.cz/turris-rc/packages//turrispackages/libpaho-mqtt-c_1.2.0-1_mpc85xx.ipk
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  208k  100  208k    0     0  1365k      0 --:--:-- --:--:-- --:--:-- 1383k
Installing python3-six (1.9.0-2) to root...
Downloading https://repo.turris.cz/turris-rc/packages//turrispackages/python3-six_1.9.0-2_mpc85xx.ipk
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  9605  100  9605    0     0   142k      0 --:--:-- --:--:-- --:--:--  144k
Installing python3-ply (3.8-3) to root...
Downloading https://repo.turris.cz/turris-rc/packages//turrispackages/python3-ply_3.8-3_mpc85xx.ipk
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 48331  100 48331    0     0   476k      0 --:--:-- --:--:-- --:--:--  481k
Installing python3-pycparser (2.14-3) to root...
Downloading https://repo.turris.cz/turris-rc/packages//turrispackages/python3-pycparser_2.14-3_mpc85xx.ipk
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 50396  100 50396    0     0   518k      0 --:--:-- --:--:-- --:--:--  523k
Installing python3-cffi (1.5.2-2) to root...
Downloading https://repo.turris.cz/turris-rc/packages//turrispackages/python3-cffi_1.5.2-2_mpc85xx.ipk
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  137k  100  137k    0     0  1112k      0 --:--:-- --:--:-- --:--:-- 1121k
Installing python3-idna (2.1-1) to root...
Downloading https://repo.turris.cz/turris-rc/packages//turrispackages/python3-idna_2.1-1_mpc85xx.ipk
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 49563  100 49563    0     0   537k      0 --:--:-- --:--:-- --:--:--  543k
Installing python3-pyasn1 (0.1.9-1) to root...
Downloading https://repo.turris.cz/turris-rc/packages//turrispackages/python3-pyasn1_0.1.9-1_mpc85xx.ipk
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 29463  100 29463    0     0   346k      0 --:--:-- --:--:-- --:--:--  350k
Installing python3-cryptography (1.7.2-1) to root...
Downloading https://repo.turris.cz/turris-rc/packages//turrispackages/python3-cryptography_1.7.2-1_mpc85xx.ipk
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  239k  100  239k    0     0  1443k      0 --:--:-- --:--:-- --:--:-- 1452k
Installing sentinel-certgen (1-1) to root...
Downloading https://repo.turris.cz/turris-rc/packages//turrispackages/sentinel-certgen_1-1_mpc85xx.ipk
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  7947  100  7947    0     0   104k      0 --:--:-- --:--:-- --:--:--  106k
Installing sentinel-proxy (1-6) to root...
Downloading https://repo.turris.cz/turris-rc/packages//turrispackages/sentinel-proxy_1-6_mpc85xx.ipk
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  5900  100  5900    0     0  84285      0 --:--:-- --:--:-- --:--:-- 85507
Configuring python3-ply.
Configuring python3-pycparser.
Configuring msgpack-c.
Configuring python3-idna.
Configuring czmq.
Configuring libpaho-mqtt-c.
Configuring python3-six.
Configuring python3-cffi.
Configuring python3-pyasn1.
Configuring python3-cryptography.
Configuring sentinel-certgen.
Configuring sentinel-proxy.
Command failed: Not found
Configuring python3-ply.
Configuring python3-pycparser.
Configuring msgpack-c.
Configuring python3-idna.
Configuring czmq.
Configuring libpaho-mqtt-c.
Configuring python3-six.
Configuring python3-cffi.
Configuring python3-pyasn1.
Configuring python3-cryptography.
Configuring sentinel-certgen.
Configuring sentinel-proxy.
Command failed: Not found
Configuring sentinel-minipot.
Command failed: Not found
root@turris:~# Configuring python3-ply.
-ash: Configuring: not found
root@turris:~# Configuring python3-pycparser.
-ash: Configuring: not found
root@turris:~# Configuring msgpack-c.
-ash: Configuring: not found
root@turris:~# Configuring python3-idna.
-ash: Configuring: not found
root@turris:~# Configuring czmq.
-ash: Configuring: not found
root@turris:~# Configuring libpaho-mqtt-c.
-ash: Configuring: not found
root@turris:~# Configuring python3-six.
-ash: Configuring: not found
root@turris:~# Configuring python3-cffi.
-ash: Configuring: not found
root@turris:~# Configuring python3-pyasn1.
-ash: Configuring: not found
root@turris:~# Configuring python3-cryptography.
-ash: Configuring: not found
root@turris:~# Configuring sentinel-certgen.
-ash: Configuring: not found
root@turris:~# Configuring sentinel-proxy.
-ash: Configuring: not found
root@turris:~# Command failed: Not found
-ash: Command: not found
root@turris:~#

Second attempt

root@turris:~# opkg install sentinel-dynfw-client sentinel-minipot
Package sentinel-dynfw-client (1.0-3) installed in root is up to date.
Package sentinel-minipot (1-6) installed in root is up to date.

Test:

root@turris:~# ipset -L turris-sn-wan-input-block
Name: turris-sn-wan-input-block
Type: hash:ip
Revision: 4
Header: family inet hashsize 1024 maxelem 65536
Size in memory: 16264
References: 1
Members:
88.174.x.xx

root@turris:~# ps | grep sentinel*
 5074 root     23176 S    /usr/bin/python3 /usr/bin/sentinel-dynfw-client
 5696 root     15864 S    /usr/bin/sentinel-proxy
 9526 root     11276 S    /usr/bin/sentinel-minipot -T 2333
 9528 nobody    3004 S    /usr/bin/sentinel-minipot -T 2333
 9542 root      1352 S    grep sentinel*

Turris 1.0 RC 3.11 2018-10-31 17:15

Looks ok. During the first attempt it looks like you accidentally copypasted output of install command but in the end everything got installed. Only suspicious thing only one IP in turris-sn-wan-input-block, but that should fix itself automatically after some time.

IP addresses it out a lot, I did not want to copy it to the forum.

Ok, then it’s fine :slight_smile:

Installed, tested, list of attackers has about 300 items …
Question: is Sentinel connected with Data collection tab in Foris? Do I have to disable data collection on Data Collection tab?

My installation fails on missing python3-cryptography package. This package isn’t available.

Package sentinel-dynfw-client (1.0-3) installed in root is up to date.
Installing sentinel-minipot (1-2) to root...
Downloading https://repo.turris.cz/omnia/packages//turrispackages/sentinel-minipot_1-2_mvebu.ipk
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  9166  100  9166    0     0  15175      0 --:--:-- --:--:-- --:--:-- 15175
Collected errors:
 * satisfy_dependencies_for: Cannot satisfy the following dependencies for sentinel-minipot:
 * 	python3-cryptography *
 * opkg_install_cmd: Cannot install package sentinel-minipot.

Hello,

This is available in Turris OS 3.11, which is currently in RC. You’ll need to opt-in to RC, which you can do using the following command:

switch-branch rc

From the output, which you included, I see you’re using deploy branch, which has version 3.10.8 and indeed, there is no python3-cryptography.

1 Like

Hello everyone,

I am trying the Sentinel and it looks like everything is working.

Unfortunately, I am getting notification e-mails about outages in data collection (Firewall, uCollect).

Is this something I should simply ignore or you can easily address?

Thanks!

The same for me.
Probably the new system (Sentinel) collect data to other database then the old one (uCollect).

I have no problem. Turris 1.0 RC 3.11 2018-11-19 14:33

Should ucollect be working after deselecting Data Collection list and checkbox in Data Collection tab?
In process list I have:

* From new system:
/usr/bin/sentinel-minipot
/usr/bin/sentinel-proxy
/usr/bin/python3 /usr/bin/sentinel-dynfw-client

* From old system or used by new?
/usr/bin/ucollect /tmp/ucollect
socat STDIO OPENSSL:api.turris.cz:5679,cafile=/etc/ssl/ucollect-server.pem...
/bin/sh /usr/share/ucollect/scripts/ucollect-add-firewall

Ok, it looks like there are weird dependencies.
foris-ssbackups-plugin depends on foris-data_collect-plugin
That depends on foris-controller-data_collect-module, and that one on ucollect-prog
So if I want to have cloud backups - I need to have ucollect installed…
I assume, that this mess will be cleaned when sentinel is production-ready and published as updater’s list.

1 Like

Being curious about sentinel but not wanting to mess with the router I decided to give it a shot in a container instead, providing an isolated environment. unprivileged container would be icing on the cake…

eth0 -> vlan eth0.2 -> br-sen -> guest fw zone -> privileged container with ip subnet different from guest network

Stripped TOS down to bare essentials

`

Summary

Uninstall(“hd-idle”, “knot-resolver”, “dnsmasq”, “resolver”, “collectd”, “vpnc”, “wol”, “tinyproxy”, “samba36-server”, “samba36-client”, “openssh-sftp-client”, “openssh-sftp-client”, “openssh-sftp-server”, “openvpn-openssl”, “mjpg-streamer”, “miniupnpd”, “minidlna”, “transmission”, “ahcpd”, “ddns-scripts”, “luci-app-wol”, “luci-app-transmission”, “etherwake”, “odhcpd”, “kmod-rxrpc”)

`
Installed the sentinel stuff

Install("sentinel-minipot")

but skipped sentinel-dynfw-client since it makes only sense to run on the host.

From a remote node ran telnet against the router’s ip and the telnet login at the router is popping up and thus confirmed working. :+1:

Questions (remaining after discovery):

  1. sentinel-nikola can be configured in the container to access the nf entries in the host’s kernel log? If so what are the requirements or how to?

  2. node data collected/submitted by sentinel-nikola to the TO project can be viewed per node (similar like haas)where?


Notable issues :slightly_frowning_face:

  1. sentinel-dynfw-client fails to start at boot
  2. sentinel-nikola not automatically enabling nf/ipt logging in the wan zone
  3. firewall restart fails to load the wan_input_rule for ipset turris-sn-wan-input-block
  4. sentinel-dynfw-client fails to detect wan ip rollover
  5. nf/ipt records reportedly not being parsed - see below

Having sentinel-nicola installed on the host

info nikola: recognized WAN interfaces: lo, pppoe-wan
info nikola: Establishing connection took 0.000012 seconds
info nikola: Records parsed: 0
info nikola: Records after filtering: 0
info nikola: Records filtering took 0.000825 seconds
info nikola: Sending records took 0.008599 seconds

Even after having manually enabled nf/ipt logging for the wan zone it appears that nf/ipt records are not parsed, perhaps missing dependants?


err nikola: turris firewall rules might not be active

Llooks like a leftover from the legacy code since being handled now by dynfw-client via ipset and a governing ipt rule.

Some transparency would be welcome on the source(s) the turris-sn-wan-input-block is generated/derived from, e.g. self-harvested and/or including other block list sources.

As pointed out by @Cabal, cloud backups have problems when testing sentinel. Is there currently a way to have both?

//EDIT: Ah, it was just turris servers were down, now it seems to work.

What does this mean and how to fix it?

2019-05-02 16:35:59 info sentinel[]: INFO [certgen.process_init:309] Certificate file does not exist or is to be renewed. Re-certifying.
2019-05-02 16:35:59 err sentinel[]: ERROR [certgen.process_get:357] Get Error. Sleeping for 300 seconds before restart.
2019-05-02 16:35:59 emerg foris-controller[3898]: [2019-05-02 16:35:59] ERROR [certgen.process_get:357] Get Error. Sleeping for 300 seconds before restart.

We have released new certgen version 4.2 which fixes this issue. It would be part of Turris OS 4.0 beta1 and 3.11.5

Ok, thanks for update.