Performance degradation when portscanning from IP blocked via nikola(sentinel)

fantomas · September 19, 2020, 5:34pm

Hello,

our of curiosity, I tried to portscan my omnia from the net (I am behind static IP and have machine on the net)

in a short time, my pppoe connection failed:

Summary

Sep 19 15:46:38 gw pppd[24116]: No response to 5 echo-requests
Sep 19 15:46:38 gw pppd[24116]: Serial link appears to be disconnected.
Sep 19 15:46:38 gw pppd[24116]: Connect time 1306.6 minutes.
Sep 19 15:46:38 gw pppd[24116]: Sent 265689695 bytes, received 692739615 bytes.

and in a while again:

Summary

Sep 19 15:47:18 gw pppd[30684]: No response to 5 echo-requests
Sep 19 15:47:18 gw pppd[30684]: Serial link appears to be disconnected.
Sep 19 15:47:18 gw pppd[30684]: Connect time 0.5 minutes.
Sep 19 15:47:18 gw pppd[30684]: Sent 99232 bytes, received 161944 bytes.

While the portscan was running, when logged to the router (I moved to machine connected directly via ethernet), I noticed laggy connection and

I have stopped all sentinel processes, openvpn, and haas-proxy.
I have flushed turris-sn-dynfw-block ipset, then it got better.

Then I found out that packets from blocked hosts are repeatedly processed:

Chain zone_wan_input (1 references)
 pkts bytes target     prot opt in     out     source               destination         
 1172 64266 zone_wan_src_DROP  all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set turris-sn-dynfw-block src mark match ! 0x10/0x10 ctstate NEW /* !sentinel: dynamic firewall block */

Chain zone_wan_src_DROP (2 references)
 pkts bytes target     prot opt in     out     source               destination         
 1172 64266 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            limit: avg 500/sec burst 5 /* !sentinel: Nikola */ LOG flags 0 level 4 prefix "DROP wan in: "
 1190 65286 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            limit: avg 500/sec burst 5 /* !sentinel: Nikola */ LOG flags 0 level 4 prefix "DROP wan in: "
 1459 81106 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            limit: avg 500/sec burst 5 /* !sentinel: Nikola */ LOG flags 0 level 4 prefix "DROP wan in: "
 1543 85926 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            limit: avg 500/sec burst 5 /* !sentinel: Nikola */ LOG flags 0 level 4 prefix "DROP wan in: "
 1650 92246 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            limit: avg 500/sec burst 5 /* !sentinel: Nikola */ LOG flags 0 level 4 prefix "DROP wan in: "
 1686 94346 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            limit: avg 500/sec burst 5 /* !sentinel: Nikola */ LOG flags 0 level 4 prefix "DROP wan in: "
 1686 94346 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            limit: avg 500/sec burst 5 /* !sentinel: Nikola */ LOG flags 0 level 4 prefix "DROP wan in: "
 1745 97790 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            limit: avg 500/sec burst 5 /* !sentinel: Nikola */ LOG flags 0 level 4 prefix "DROP wan in: "
 1918  108K LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            limit: avg 500/sec burst 5 /* !sentinel: Nikola */ LOG flags 0 level 4 prefix "DROP wan in: "
 1926  109K LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            limit: avg 500/sec burst 5 /* !sentinel: Nikola */ LOG flags 0 level 4 prefix "DROP wan in: "
 2067  117K LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            limit: avg 500/sec burst 5 /* !sentinel: Nikola */ LOG flags 0 level 4 prefix "DROP wan in: "
 2080  119K LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            limit: avg 500/sec burst 5 /* !sentinel: Nikola */ LOG flags 0 level 4 prefix "DROP wan in: "
 2142  123K LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            limit: avg 500/sec burst 5 /* !sentinel: Nikola */ LOG flags 0 level 4 prefix "DROP wan in: "
 2150  123K LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            limit: avg 500/sec burst 5 /* !sentinel: Nikola */ LOG flags 0 level 4 prefix "DROP wan in: "
 2200  126K LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            limit: avg 500/sec burst 5 /* !sentinel: Nikola */ LOG flags 0 level 4 prefix "DROP wan in: "
 2209  127K LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            limit: avg 500/sec burst 5 /* !sentinel: Nikola */ LOG flags 0 level 4 prefix "DROP wan in: "
 2756  159K LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            limit: avg 500/sec burst 5 /* !sentinel: Nikola */ LOG flags 0 level 4 prefix "DROP wan in: "
 2849  165K LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            limit: avg 500/sec burst 5 /* !sentinel: Nikola */ LOG flags 0 level 4 prefix "DROP wan in: "
 2883  167K LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            limit: avg 500/sec burst 5 /* !sentinel: Nikola */ LOG flags 0 level 4 prefix "DROP wan in: "
 2883  167K LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            limit: avg 500/sec burst 5 /* !sentinel: Nikola */ LOG flags 0 level 4 prefix "DROP wan in: "
 2936  170K LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            limit: avg 500/sec burst 5 /* !sentinel: Nikola */ LOG flags 0 level 4 prefix "DROP wan in: "
 2974  172K LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            limit: avg 500/sec burst 5 /* !sentinel: Nikola */ LOG flags 0 level 4 prefix "DROP wan in: "
 3110  180K LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            limit: avg 500/sec burst 5 /* !sentinel: Nikola */ LOG flags 0 level 4 prefix "DROP wan in: "
 7239  383K LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            limit: avg 500/sec burst 5 /* !sentinel: Nikola */ LOG flags 0 level 4 prefix "DROP wan in: "
 7239  383K LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            limit: avg 500/sec burst 5 /* !sentinel: Nikola */ LOG flags 0 level 4 prefix "DROP wan in: "
11444  565K LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            limit: avg 500/sec burst 5 /* !sentinel: Nikola */ LOG flags 0 level 4 prefix "DROP wan in: "
11447  565K LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            limit: avg 500/sec burst 5 /* !sentinel: Nikola */ LOG flags 0 level 4 prefix "DROP wan in: "
11447  565K LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            limit: avg 500/sec burst 5 /* !sentinel: Nikola */ LOG flags 0 level 4 prefix "DROP wan in: "
11447  565K LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            limit: avg 500/sec burst 5 /* !sentinel: Nikola */ LOG flags 0 level 4 prefix "DROP wan in: "
11447  565K LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            limit: avg 500/sec burst 5 /* !sentinel: Nikola */ LOG flags 0 level 4 prefix "DROP wan in: "
11447  565K DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* !sentinel */

Can someone confirm if they have similar issue?

fantomas · September 21, 2020, 6:12pm

I rebooted and after 3 days:

Chain zone_wan_src_DROP (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            limit: avg 500/sec burst 5 /* !sentinel: Nikola */ LOG flags 0 level 4 prefix "DROP wan in: "
    0     0 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            limit: avg 500/sec burst 5 /* !sentinel: Nikola */ LOG flags 0 level 4 prefix "DROP wan in: "
    0     0 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            limit: avg 500/sec burst 5 /* !sentinel: Nikola */ LOG flags 0 level 4 prefix "DROP wan in: "
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* !sentinel */

does something add the LOG line to zone_wan_src_DROP daily?

cynerd · September 22, 2020, 7:41am

We have fix for that already prepared. It is just stuck in our internal review process. For time being restart of firewall or device should fix the issue. I am sorry for inconvenience, we missed that it can affect performance of device this way. (https://gitlab.nic.cz/turris/turris-os-packages/-/merge_requests/429)

fantomas · September 22, 2020, 3:13pm

I have searched gitlab if I find this in issued or merge requests.
Apparently not thoroughly enough.

thank you.

fantomas · September 27, 2020, 4:32pm

…unfortunately this still happens in 5.1.1, the commit wasn’t apparently included.
I’m patching /usr/libexec/sentinel/firewall.sh manually,

i guess Makefile and uci-defaults aren’t needed here.