Trying new data collecting system Sentinel

sentinel

#1

As you might have heard, we are working on redesign of old uCollect system that was collecting firewall logs and various data about people trying to attack your router. In Turris OS 3.11 we will make first parts of our new software – called Sentinel – available. So we would like to ask you to test it out. :slight_smile:

How to do that? First of all you have to disable the old system. You can do that simply in updater tab of Foris web interface where you uncheck Data Collection list.

Installation of the new one is a little trickier and you need LuCI/CLI to do that. First update list of packages and afterwards install sentinel-dynfw-client and sentinel-minipot packages with these commands:

   opkg update
   opkg install sentinel-dynfw-client sentinel-minipot

We don’t have fancy list in updater yet as it is highly experimental at the moment. There is also no web UI that you can check to see whether it works. The only way to do so is to play an attacker and try connecting via telnet to your internet facing IP. If everything works, you should get prompt asking for credentials. Those credentials will be logged, be aware!

Also you can check that you are getting the list of attackers using command ipset -L turris-sn-wan-input-block. But don’t search there for your IP. Attacking one router is not evil enough to get you on the list.

Our current plan is to get more testers, put the system under some stress and fine tune the new detection algorithms. Over the time, we will phase out the old system and replace it completely with the new one.

If you are interested in more details about how we are redesigning our data collection system and you happen to speak Czech and be in Prague next week, we are going to have a talk about it on IT18 conference.


Turris OS 3.11 is out!
#2

Is this correct?

First attempt

Souhrn
root@turris:~# opkg install sentinel-dynfw-client sentinel-minipot
Installing sentinel-dynfw-client (1.0-3) to root...
Downloading https://repo.turris.cz/turris-rc/packages//turrispackages/sentinel-dynfw-client_1.0-3_mpc85xx.ipk
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  4730  100  4730    0     0  66619      0 --:--:-- --:--:-- --:--:-- 67571
Installing libzmq-nc (4.2.2-1) to root...
Downloading https://repo.turris.cz/turris-rc/packages//packages/libzmq-nc_4.2.2-1_mpc85xx.ipk
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  235k  100  235k    0     0  1461k      0 --:--:-- --:--:-- --:--:-- 1470k
Installing python3-zmq (16.0.2-2) to root...
Downloading https://repo.turris.cz/turris-rc/packages//turrispackages/python3-zmq_16.0.2-2_mpc85xx.ipk
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  322k  100  322k    0     0  1680k      0 --:--:-- --:--:-- --:--:-- 1688k
Installing python3-msgpack (0.5.6-2) to root...
Downloading https://repo.turris.cz/turris-rc/packages//turrispackages/python3-msgpack_0.5.6-2_mpc85xx.ipk
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 13100  100 13100    0     0   180k      0 --:--:-- --:--:-- --:--:--  182k
Configuring libzmq-nc.
Configuring python3-zmq.
Configuring python3-msgpack.
Configuring sentinel-dynfw-client.
Installing sentinel-minipot (1-6) to root...
Downloading https://repo.turris.cz/turris-rc/packages//turrispackages/sentinel-minipot_1-6_mpc85xx.ipk
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 10068  100 10068    0     0   132k      0 --:--:-- --:--:-- --:--:--  134k
Installing czmq (20171102-1) to root...
Downloading https://repo.turris.cz/turris-rc/packages//turrispackages/czmq_20171102-1_mpc85xx.ipk
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  168k  100  168k    0     0  1267k      0 --:--:-- --:--:-- --:--:-- 1276k
Installing msgpack-c (2.1.5-2) to root...
Downloading https://repo.turris.cz/turris-rc/packages//turrispackages/msgpack-c_2.1.5-2_mpc85xx.ipk
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 15384  100 15384    0     0   165k      0 --:--:-- --:--:-- --:--:--  165k
Installing libpaho-mqtt-c (1.2.0-1) to root...
Downloading https://repo.turris.cz/turris-rc/packages//turrispackages/libpaho-mqtt-c_1.2.0-1_mpc85xx.ipk
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  208k  100  208k    0     0  1365k      0 --:--:-- --:--:-- --:--:-- 1383k
Installing python3-six (1.9.0-2) to root...
Downloading https://repo.turris.cz/turris-rc/packages//turrispackages/python3-six_1.9.0-2_mpc85xx.ipk
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  9605  100  9605    0     0   142k      0 --:--:-- --:--:-- --:--:--  144k
Installing python3-ply (3.8-3) to root...
Downloading https://repo.turris.cz/turris-rc/packages//turrispackages/python3-ply_3.8-3_mpc85xx.ipk
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 48331  100 48331    0     0   476k      0 --:--:-- --:--:-- --:--:--  481k
Installing python3-pycparser (2.14-3) to root...
Downloading https://repo.turris.cz/turris-rc/packages//turrispackages/python3-pycparser_2.14-3_mpc85xx.ipk
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 50396  100 50396    0     0   518k      0 --:--:-- --:--:-- --:--:--  523k
Installing python3-cffi (1.5.2-2) to root...
Downloading https://repo.turris.cz/turris-rc/packages//turrispackages/python3-cffi_1.5.2-2_mpc85xx.ipk
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  137k  100  137k    0     0  1112k      0 --:--:-- --:--:-- --:--:-- 1121k
Installing python3-idna (2.1-1) to root...
Downloading https://repo.turris.cz/turris-rc/packages//turrispackages/python3-idna_2.1-1_mpc85xx.ipk
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 49563  100 49563    0     0   537k      0 --:--:-- --:--:-- --:--:--  543k
Installing python3-pyasn1 (0.1.9-1) to root...
Downloading https://repo.turris.cz/turris-rc/packages//turrispackages/python3-pyasn1_0.1.9-1_mpc85xx.ipk
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 29463  100 29463    0     0   346k      0 --:--:-- --:--:-- --:--:--  350k
Installing python3-cryptography (1.7.2-1) to root...
Downloading https://repo.turris.cz/turris-rc/packages//turrispackages/python3-cryptography_1.7.2-1_mpc85xx.ipk
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  239k  100  239k    0     0  1443k      0 --:--:-- --:--:-- --:--:-- 1452k
Installing sentinel-certgen (1-1) to root...
Downloading https://repo.turris.cz/turris-rc/packages//turrispackages/sentinel-certgen_1-1_mpc85xx.ipk
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  7947  100  7947    0     0   104k      0 --:--:-- --:--:-- --:--:--  106k
Installing sentinel-proxy (1-6) to root...
Downloading https://repo.turris.cz/turris-rc/packages//turrispackages/sentinel-proxy_1-6_mpc85xx.ipk
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  5900  100  5900    0     0  84285      0 --:--:-- --:--:-- --:--:-- 85507
Configuring python3-ply.
Configuring python3-pycparser.
Configuring msgpack-c.
Configuring python3-idna.
Configuring czmq.
Configuring libpaho-mqtt-c.
Configuring python3-six.
Configuring python3-cffi.
Configuring python3-pyasn1.
Configuring python3-cryptography.
Configuring sentinel-certgen.
Configuring sentinel-proxy.
Command failed: Not found
Configuring python3-ply.
Configuring python3-pycparser.
Configuring msgpack-c.
Configuring python3-idna.
Configuring czmq.
Configuring libpaho-mqtt-c.
Configuring python3-six.
Configuring python3-cffi.
Configuring python3-pyasn1.
Configuring python3-cryptography.
Configuring sentinel-certgen.
Configuring sentinel-proxy.
Command failed: Not found
Configuring sentinel-minipot.
Command failed: Not found
root@turris:~# Configuring python3-ply.
-ash: Configuring: not found
root@turris:~# Configuring python3-pycparser.
-ash: Configuring: not found
root@turris:~# Configuring msgpack-c.
-ash: Configuring: not found
root@turris:~# Configuring python3-idna.
-ash: Configuring: not found
root@turris:~# Configuring czmq.
-ash: Configuring: not found
root@turris:~# Configuring libpaho-mqtt-c.
-ash: Configuring: not found
root@turris:~# Configuring python3-six.
-ash: Configuring: not found
root@turris:~# Configuring python3-cffi.
-ash: Configuring: not found
root@turris:~# Configuring python3-pyasn1.
-ash: Configuring: not found
root@turris:~# Configuring python3-cryptography.
-ash: Configuring: not found
root@turris:~# Configuring sentinel-certgen.
-ash: Configuring: not found
root@turris:~# Configuring sentinel-proxy.
-ash: Configuring: not found
root@turris:~# Command failed: Not found
-ash: Command: not found
root@turris:~#

Second attempt

root@turris:~# opkg install sentinel-dynfw-client sentinel-minipot
Package sentinel-dynfw-client (1.0-3) installed in root is up to date.
Package sentinel-minipot (1-6) installed in root is up to date.

Test:

root@turris:~# ipset -L turris-sn-wan-input-block
Name: turris-sn-wan-input-block
Type: hash:ip
Revision: 4
Header: family inet hashsize 1024 maxelem 65536
Size in memory: 16264
References: 1
Members:
88.174.x.xx

root@turris:~# ps | grep sentinel*
 5074 root     23176 S    /usr/bin/python3 /usr/bin/sentinel-dynfw-client
 5696 root     15864 S    /usr/bin/sentinel-proxy
 9526 root     11276 S    /usr/bin/sentinel-minipot -T 2333
 9528 nobody    3004 S    /usr/bin/sentinel-minipot -T 2333
 9542 root      1352 S    grep sentinel*

Turris 1.0 RC 3.11 2018-10-31 17:15


#3

Looks ok. During the first attempt it looks like you accidentally copypasted output of install command but in the end everything got installed. Only suspicious thing only one IP in turris-sn-wan-input-block, but that should fix itself automatically after some time.


#4

IP addresses it out a lot, I did not want to copy it to the forum.


#5

Ok, then it’s fine :slight_smile:


#6

Installed, tested, list of attackers has about 300 items …
Question: is Sentinel connected with Data collection tab in Foris? Do I have to disable data collection on Data Collection tab?


#7

My installation fails on missing python3-cryptography package. This package isn’t available.

Package sentinel-dynfw-client (1.0-3) installed in root is up to date.
Installing sentinel-minipot (1-2) to root...
Downloading https://repo.turris.cz/omnia/packages//turrispackages/sentinel-minipot_1-2_mvebu.ipk
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  9166  100  9166    0     0  15175      0 --:--:-- --:--:-- --:--:-- 15175
Collected errors:
 * satisfy_dependencies_for: Cannot satisfy the following dependencies for sentinel-minipot:
 * 	python3-cryptography *
 * opkg_install_cmd: Cannot install package sentinel-minipot.

#8

Hello,

This is available in Turris OS 3.11, which is currently in RC. You’ll need to opt-in to RC, which you can do using the following command:

switch-branch rc

From the output, which you included, I see you’re using deploy branch, which has version 3.10.8 and indeed, there is no python3-cryptography.


#9

Hello everyone,

I am trying the Sentinel and it looks like everything is working.

Unfortunately, I am getting notification e-mails about outages in data collection (Firewall, uCollect).

Is this something I should simply ignore or you can easily address?

Thanks!


#10

The same for me.
Probably the new system (Sentinel) collect data to other database then the old one (uCollect).


#11

I have no problem. Turris 1.0 RC 3.11 2018-11-19 14:33


#12

Should ucollect be working after deselecting Data Collection list and checkbox in Data Collection tab?
In process list I have:

* From new system:
/usr/bin/sentinel-minipot
/usr/bin/sentinel-proxy
/usr/bin/python3 /usr/bin/sentinel-dynfw-client

* From old system or used by new?
/usr/bin/ucollect /tmp/ucollect
socat STDIO OPENSSL:api.turris.cz:5679,cafile=/etc/ssl/ucollect-server.pem...
/bin/sh /usr/share/ucollect/scripts/ucollect-add-firewall