Sentinel: ipset turris-sn-dynfw-block_v4 is full

General discussion
1

Hi
Since about one hour ago I have started to see this kind of errors on my omnia.

Aug 26 18:47:07 turris kernel: [4763830.125048] Set turris-sn-dynfw-block_v4 is full, maxelem 65536 reached
Aug 26 18:47:59 turris kernel: [4763881.951678] Set turris-sn-dynfw-block_v4 is full, maxelem 65536 reached
Aug 26 18:48:18 turris kernel: [4763901.464858] Set turris-sn-dynfw-block_v4 is full, maxelem 65536 reached
Aug 26 18:49:30 turris kernel: [4763973.455229] Set turris-sn-dynfw-block_v4 is full, maxelem 65536 reached
Aug 26 18:49:45 turris kernel: [4763988.285029] Set turris-sn-dynfw-block_v4 is full, maxelem 65536 reached
Aug 26 18:50:30 turris kernel: [4764032.703117] Set turris-sn-dynfw-block_v4 is full, maxelem 65536 reached
Aug 26 18:50:47 turris kernel: [4764049.541701] Set turris-sn-dynfw-block_v4 is full, maxelem 65536 reached
Aug 26 18:50:49 turris kernel: [4764052.014215] Set turris-sn-dynfw-block_v4 is full, maxelem 65536 reached

ipset list shows the following:

Name: turris-sn-dynfw-block_v4
Type: hash:ip
Revision: 4
Header: family inet hashsize 32768 maxelem 65536
Size in memory: 1305188
References: 0
Number of entries: 65598
Members:
46.101.157.195
87.236.176.226
46.101.133.50

ipset list turris-sn-dynfw-block_v4 |wc -l

65608

Maybe the sentinel list has older entries that can be purged?

Regards

2

I restarted the sentinel-dynfw-client and the ipset in question is no longer full :slight_smile:
I guess it become hung somehow…

3

sorry for the spam.

The ipset is full again, this is after reboot…

Aug 26 20:00:49 turris kernel: [ 336.182460] Set turris-sn-dynfw-block_v4 is full, maxelem 65536 reached
Aug 26 20:01:11 turris kernel: [ 357.739941] net_ratelimit: 99 callbacks suppressed
Aug 26 20:01:11 turris kernel: [ 357.739952] Set turris-sn-dynfw-block_v4 is full, maxelem 65536 reached
Aug 26 20:01:27 turris kernel: [ 373.578685] Set turris-sn-dynfw-block_v4 is full, maxelem 65536 reached
Aug 26 20:01:43 turris kernel: [ 389.666235] Set turris-sn-dynfw-block_v4 is full, maxelem 65536 reached
Aug 26 20:03:43 turris kernel: [ 510.133884] Set turris-sn-dynfw-block_v4 is full, maxelem 65536 reached

1 Like
4

Same here, but only on the MOX, not on the TO.

5

Interesting, here Dynamic Firewall | Sentinel View you can see that the current list contains 67969 entries, normally that is between 9 and 12K.

What does worry me is that no one is reacting from .cz? Since this is your FW.

6

The ipset gets filled very slowly, so it might look okay for a time, but will be full again later.

Until an official solution is available, here is a crude, quick and dirty workaround.

/usr/libexec/sentinel/firewall.d/50-dynfw.sh:

#!/bin/sh
set -e
. "${0%/*}/common.sh"
. /lib/functions.sh

IPSET="turris-sn-dynfw-block"
MAXELEM=131072 # Double the default size of 65,536

# Always create IP set to prevent iptables error about missing ipset.
##ipset create "$IPSET"_v4 hash:ip family inet -exist
ipset create "$IPSET"_v4 hash:ip maxelem $MAXELEM family inet -exist
##ipset create "$IPSET"_v4 hash:ip family inet -exist
ipset create "$IPSET"_v6 hash:net maxelem $MAXELEM family inet6 -exist

...
3 Likes
7

Same shit here ! Mox at 7.0.2

Dynfw sentinel stats

Updated	Thu, 29 Aug 2024 07:17:35 GMT
Current list version / Serial	1724915855 / 61392

~75827 entries
1 Like
8

I’m seeing the same on a Turris Omnia running 7.0.2. I tried the maxelem $MAXELEM trick/hack - but then I got errors when IP addresses was to be added … so the ipset list was just empty. I had to revert to the default of 64K entries.

I just hope the migration to nftables will happen soon and that ipset can be moved over to an nft set as well. iptables/ipset does have quite some limitations.

9

Number of records in Dynamic Firewall | Sentinel View is increasing over and over:
Updated Sat, 31 Aug 2024 07:01:06 GMT
Current list: 82890

10

This is happening to me at my TO. I am from Czech republic.

11

Problem seems to be solved now.
Last entry in log is from 12:12.
Current size of list is 14112:

root@turris:~# ipset list | less
Name: turris-sn-dynfw-block_v4
Type: hash:ip
Revision: 5
Header: family inet hashsize 32768 maxelem 65536 bucketsize 12 initval 0x544b89b
f
Size in memory: 378340
References: 2
Number of entries: 14112

12

Yes, the problem has been solved. Negative delta messages (removals of addresses) were not generated correctly and some already-unblocked addresses remained in ipsets. This problem has been fixed and it should work now. We apologize for the problem. Furthermore, we are discussing to set a higher limit for the ipset (because it is possible to have larger sets even if everything works correctly).

4 Likes
13

Okay, thanks.

If you considering higher limits, here is some inspiration …

https://gist.github.com/alainwolf/c754e9db501ef5353e1833e26b467908

I count the records and calculate the new size for the ipset accordingly:

    # Count the number of entries in the blacklist file
    echo -n "Counting records ... "
    # shellcheck disable=SC2126
    ENTRY_COUNT=$(grep -v '^\s*$\|^\s*#' "$LOCAL_FILE" | wc -l)
    printf "%s records found\n" "$ENTRY_COUNT"

    # Calculate hashsize and maxelem
    echo -n "Calculating size ... "
    HASH_SIZE=$((ENTRY_COUNT / 64))
    [ $HASH_SIZE -lt 1024 ] && HASH_SIZE=1024
    MAX_ELEM=$((ENTRY_COUNT + 1000))
    printf "a hash size of %s and an max lenght of %s is recommended.\n" "$HASH_SIZE" "$MAX_ELEM"

Unfortunately, you can’t change the size of an already existing ipset. You need to destroy it and create a new one.

But then, you can’t destroy it either, since the kernel has a lock on it,

So I create a new temporary ipset with the calculated size:

    # Create a temporary ipset
    TEMP_IPSET_NAME="${IPSET_NAME}_temp"
    echo -n "Creating a temporary ipset ... "
    ipset create "$TEMP_IPSET_NAME" hash:ip family inet \
        hashsize $HASH_SIZE maxelem $MAX_ELEM timeout $EXPIRE_SECONDS

Fill the temporary ipset using the restore command from file, since the conventional way using add one-by-one, takes hours to complete on a big set.

Which by the way. was the reason, why some user thought the problem was solved after a reboot, but then later reappeared.

    # Create a temporary file for ipset restore
    IPSET_TEMP_FILE=$(mktemp "/tmp/ipset-restore.XXXXXX")

    # Write ipset commands to the temporary file
    {
        while IFS= read -r ip; do
            # Skip comments and empty lines
            case "$ip" in
            '' | \#*) continue ;;
            *) echo "add $TEMP_IPSET_NAME $ip" ;;
            esac
        done <"$LOCAL_FILE"
    } >"$IPSET_TEMP_FILE"

    # Restore the temporary ipset from the temporary file
    ipset restore <"$IPSET_TEMP_FILE"
    echo "Done."

Then you can swap, since that somehow works around the kernel lock:


    # Swap the temporary ipset with the original ipset
    echo -n "Swapping new temporary ipset with old one ... "
    ipset swap "$TEMP_IPSET_NAME" "$IPSET_NAME"
    echo "ipsets swapped."

Hope this helps. I did this accidentally just a few weeks ago, with some of the ideas contributed by Githuib Copilot. Sometimes he can really be helpful and creative, on other times, he just drags you down into a dark rabbit hole.

4 Likes
14

It seems the dynamic firewall list has been reduced on the service side, now it’s just below 15k.

As a quick-fix, run:

root@turris:~#  service sentinel-dynfw-client status

or do a reboot. That should fix it for now … until the list gets way too large again.

15

This topic was automatically closed after 60 days. New replies are no longer allowed.