Knot resolver and RPZ

jklaas · December 4, 2016, 1:43pm

I read the articles on doing an RPZ and I think it would be best to have this as something that can be done within the Knot resolver, it has that capability. However, the documentation is vague, especially with respect to how the Omnia runs.

I added

echo "policy.add(policy.rpz(policy.DENY, 'blacklist.rpz'))" >>$CONFIGFILE

to /etc/init.d/kresd. Any file like “blacklist.rpz” has to be put in $DEFAULT_RUNDIR which is “/tmp/kresd”. Otherwise a full path is needed. The Knot documentation states

rpz - implementes a subset of the RPZ format. Currently it can be used with a zonefile, a binary database support is on the way. Binary database can be updated by an external process on the fly.

I generate my “conf” file from http://pgl.yoyo.org/adservers/ as a “bind 8 config” format which is like:

zone "yieldmanager.net" { type master; notify no; file "null.zone.file"; };

and created a null.zone.file, also in /etc/init.d/kresd that is basically a wildcard pointing to 127.0.0.1

This doesn’t appear to work, but I think I have close to all the pieces. I expect my “bind 8 config” file wont really work and I need a different format.

tgalarneau · December 4, 2016, 11:21pm

I have been looking at this too. The policy.DENY seems to only allow NXDOMAIN as action.

“Respond with NXDOMAIN to all queries matching the rule.”

So it looks like ‘blacklist.rpz’ should only contain name i.e.

‘example.com’

However documentation is not very clear. Where exactly are you putting in /etc/init.d/kresd file.

jklaas · December 5, 2016, 1:23pm

Since I’m also doing local resolution with dnsmasq, I’ve added it as the last line in the “init_headers” stanza, just before the closing bracket.

On your suggestion, I tried just a list of hosts, but that didn’t seem to work either. Since knot was created by cz.nic, I was hoping someone there might chime in.

tgalarneau · December 5, 2016, 5:10pm

Yes it would be nice to hear from developers of Knot.

I found this sample configuration from Knot for loading file.

– Load Useful modules
modules = {
‘policy’, – Block queries to local zones/bad sites
’hints’, – Load /etc/hosts and allow custom root hints
’stats’, – Track internal statistics
’predict’, – Prefetch expiring/frequent records
}

tgalarneau · December 5, 2016, 8:55pm

Seems like your setup should work. But no luck.

Even tried wire wiremat :

“Note: The module (and kres) expects domain names in wire format, not textual representation. So each label in
name is prefixed with its length, e.g. “example.com” equals to “\7example\3com”. You can use convenience
function todname(’example.com’) for automatic conversion.”

Tried both rpz file which than stops all addresses with dns conf error. And direct policy.add(policy.suffix(policy.DENY, {’\6badboy\2cz’})) type with no success.

jklaas · December 6, 2016, 6:57pm

OK, I figured it out. After perusing some of the links in the knot-resolver source, I played around with the files a little and came up with this blacklist.rpz file:

$TTL 60
@               IN      SOA  localhost. root.localhost.  (
                        2   ; serial 
                        3H  ; refresh 
                        1H  ; retry 
                        1W  ; expiry 
                        1H) ; minimum 
                IN      NS    localhost.

101com.com      CNAME   .
101order.com    CNAME   .
123found.com    CNAME   .
180hits.de      CNAME   .
180searchassistant.com  CNAME   .
1x1rank.com     CNAME   .
207.net CNAME   .
247media.com    CNAME   .
24log.com       CNAME   .
...
zanox.com       CNAME   .
zantracker.com  CNAME   .
zedo.com        CNAME   .
zencudo.co.uk   CNAME   .
zenkreka.com    CNAME   .
zenzuu.com      CNAME   .
zeus.developershed.com  CNAME   .
zeusclicks.com  CNAME   .
zintext.com     CNAME   .
zmedia.com      CNAME   .
zv1.november-lax.com    CNAME   .

Before I would get:

$ dig zantracker.com

; <<>> DiG 9.9.5-9+deb8u8-Debian <<>> zantracker.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 21564
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;zantracker.com.                        IN      A

;; ANSWER SECTION:
zantracker.com.         60      IN      A       45.33.35.195

;; Query time: 516 msec
;; SERVER: 192.168.4.1#53(192.168.4.1)
;; WHEN: Tue Dec 06 13:38:13 EST 2016
;; MSG SIZE  rcvd: 59

Now I get:

$ dig zantracker.com

; <<>> DiG 9.9.5-9+deb8u8-Debian <<>> zantracker.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 62428
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;zantracker.com.                        IN      A

;; AUTHORITY SECTION:
blocked.                900     IN      SOA     blocked. . 0 3600 900 604800 900

;; Query time: 0 msec
;; SERVER: 192.168.4.1#53(192.168.4.1)
;; WHEN: Tue Dec 06 13:53:06 EST 2016
;; MSG SIZE  rcvd: 85

So satisfying.

This should be trivial to automate with some quick scripting.
[edit] I also moved the “policy.add(policy.rpz(policy.DENY, ‘/etc/kresd/blacklist.rpz’))” line to just after the “modules” stanza and before the other “policy.add” lines.

jklaas · December 6, 2016, 7:52pm

OK, I’ve knocked together a script to generate the rpz. I thought I should summarize all the steps to do this here though:

Add

echo "policy.add(policy.rpz(policy.DENY, 'blacklist.rpz'))" >>$CONFIGFILE

to /etc/init.d/kresd, just after the “modules” stanza. So, that means after the

modules = {
...
}

block.

Create a script file called “generate_blacklist.sh” somewhere. There really should be a “/usr/local…” structure for this, but I just put it in “/root/rpz” for now.

#!/bin/bash

blacklistfile="/tmp/kresd/blacklist.rpz"

cat > $blacklistfile <<EOF
$TTL 60
@ IN SOA localhost. root.localhost. (
2 ; serial
3H ; refresh
1H ; retry
1W ; expiry
1H) ; minimum
IN NS localhost.

EOF

for host in wget --quiet -O - 'http://pgl.yoyo.org/adservers/serverlist.php?hostformat=nohtml&showintro=1&startdate%5Bday%5D=&startdate%5Bmonth%5D=&startdate%5Byear%5D=&mimetype=plaintext'; do
echo “$host CNAME .” >> $blacklistfile
done

/etc/init.d/kresd reload

The “blacklistfile” must match the location listed in part 1. If there’s just a “blacklist.rpz”, then “blacklistfile” must be in “/tmp/kresd”. Otherwise specify the appropriate path. Also “/etc/init.d/kresd reload” is required since a file is being changed and kresd needs to reload the file. Don’t forget to make “generate_blacklist.sh” executable.

You want to make sure your blacklist file is up to date. Create a cron job for this.
crontab -e
and add a line like the following:
6 1 * * 3 /root/rpz/generate_blacklist.sh
Please make sure you change the time you get the file. We don’t want to be bombarding the site with requests at the same time. A cron line looks like:
- ```
*     *   *    *        command to be executed
```
| | | | |
| | | | ±---- day of week (0 - 6) (Sunday=0)
| | | ±------ month (1 - 12)
| | ±-------- day of month (1 - 31)
| ±---------- hour (0 - 23)
±------------ min (0 - 59)
(from http://www.adminschoice.com/crontab-quick-reference).

I’m happy to explain the script to any who are interested. After all, you shouldn’t just be running a script you don’t understand on your router.

tgalarneau · December 6, 2016, 9:29pm

Great, no works on my system. Just need to add more sources. Too much still slips thru with yoyo.

Thanks

Marek_Ruzicka · December 7, 2016, 8:29pm

@jklaas thanks for the quick/simple guide…works just fine, but still does not filter enough stuff out.

Do you have any particular source in mind?
The most important thing for me is to block youtube adds (on my tv) and this yoyo list is not really doing that.

einar · December 7, 2016, 8:42pm

For the record, the new update that’s supposed to come soon will add support for a custom configuration file for Knot which wlll be read along with the generated one. (I don’t work for the team, but I backported some of their patches locally to get local host resolution working).

tgalarneau · December 7, 2016, 9:05pm

I don’t use Youtube much but there are a lot of ad blocking lists available you could try. You should look at the list supplied with https://github.com/notracking/hosts-blocklists to get ideas for sources.

My problem has been when have created a larger rpz file and used it, then blocking fails altogether. I don’t know if this is related to cache size or something else in my file, but this variable is set through a confusing automated process in init.d.

I keep messing with this and hope the update improves the process. It would be nice if the documentation was clearer too.

jklaas · December 8, 2016, 7:54pm

I also realized if the router reboots it wont find “/tmp/kresd/blacklist.rpz”. It might be better to specify a permanent location for the blacklist.rpz file, someplace not on the local storage that can stand to be written to a lot (depending on how much you udpate).

jklaas · December 8, 2016, 8:42pm

One more update. Sorry for spamming my own entry.

I just got through with fixing all the “updates”. One nice thing is a new entry in the
config resolver 'kresd'
section, there’s a new entry
option include_config '/tmp/kresd.custom.conf'
I wanted it to be persistent so I changed it to
option include_config '/etc/kresd/custom.conf'
and put
policy.add(policy.rpz(policy.DENY, 'blacklist.rpz'))
in there along with my other policy.add lines and they show up in my /tmp/kresd.config file. Still no way to edit via luci as far as I can tell, but it’s a step forward.

Leonardo · December 10, 2016, 5:00am

This source merged multiple hosts files.

github.com

StevenBlack/hosts/blob/master/hosts

# This hosts file is a merged collection of hosts from reputable sources,
# with a dash of crowd sourcing via Github
#
# Date: January 10 2018
# Number of unique domains: 43,088
#
# Fetch the latest version of this file: https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
# Project home page: https://github.com/StevenBlack/hosts
#
# ===============================================================

127.0.0.1 localhost
127.0.0.1 localhost.localdomain
127.0.0.1 local
255.255.255.255 broadcasthost
::1 localhost
fe80::1%lo0 localhost
0.0.0.0 0.0.0.0

# Custom host records are listed here.

This file has been truncated. show original

It needs to be parsed to be in the right format.

Or it can be used as /etc/hosts file directly.

/etc/config/resolver

has a new option too:

option hostname_config ‘/etc/hosts’

jklaas · December 10, 2016, 2:45pm

I ended up getting three sources:

http://malwaredomains.lehigh.edu/files/immortal_domains.txt
http://malwaredomains.lehigh.edu/files/BOOT

and the aforementioned one from yoyo. It ends up with about 17700 entries after duplicates are removed. I am interested in removing most ads but I’m more interested in removing links to malware.

I can share my updated script if anyone is interested.

Jason_A · December 10, 2016, 5:45pm

@jklaas please do - so are you using the hosts file option or the rpz file option, it looks like either would work, right?

jklaas · December 10, 2016, 8:41pm

I’m using the rpz file option. Using dig returns a “blocked” reply which is essentially an NXDOMAIN. If you know your way around setting up the hosts file it shouldn’t be difficult to modify the script.

It looks like I can’t upload a non-image file. Perhaps that can be adjusted. Meanwhile:

#!/bin/bash

blacklistfile="/etc/kresd/blacklist.rpz"
cnamesection="/tmp/cnamesection"

cat > $blacklistfile <<EOF
\$TTL 60
@		IN	SOA  localhost. root.localhost.  (
			2   ; serial 
			3H  ; refresh 
			1H  ; retry 
			1W  ; expiry 
			1H) ; minimum 
		IN	NS    localhost.

EOF

hostlist1=`wget -q 'http://malwaredomains.lehigh.edu/files/immortal_domains.txt' -O - | grep -v \#`
hostlist2=`wget -q 'http://pgl.yoyo.org/adservers/serverlist.php?hostformat=nohtml&showintro=1&startdate%5Bday%5D=&startdate%5Bmonth%5D=&startdate%5Byear%5D=&mimetype=plaintext' -O - `
hostlist3=`wget -q 'http://malwaredomains.lehigh.edu/files/BOOT' -O - | grep PRIMARY | cut -d " " -f 2 `

for host in `echo $hostlist1$hostlist2$hostlist3`; do
	echo "$host	CNAME	." >>  $cnamesection
done

sort -u $cnamesection >> $blacklistfile

rm $cnamesection

/etc/init.d/kresd reload

Maybe someone better at shell scripting can do without the use of the external temporary file. I tried multiple greps but

Jason_A · December 10, 2016, 11:46pm

have you noticed any performance hit with that large a rpz file? i’m seeing some timeouts on the first query but it does immediately resolve on the second try.

jklaas · December 11, 2016, 1:51pm

I’m not sure. Sometimes a browser on a phone or other client is a little slow getting a site, but using dig on the command line works every time.

I also have turned forwarding off since my ISP likes to mess with my DNS “for security reason” (so they can try to sell things to me). Still too many ads getting through. I’m debating adding a “*” to the beginning of all entries in the list so it gets all the subdomains as well, but that will be detrimental without further editing.

Cabal · January 29, 2017, 12:30pm

Here you have version without temporary file:

#!/bin/bash

blacklistfile="/etc/kresd/blacklist.rpz"

cat > $blacklistfile <<EOF
\$TTL 60
@               IN      SOA  localhost. root.localhost.  (
            2   ; serial
            3H  ; refresh
            1H  ; retry
            1W  ; expiry
            1H) ; minimum
        IN      NS    localhost.

EOF

{
    wget -q 'http://malwaredomains.lehigh.edu/files/immortal_domains.txt' -O - | grep -v \#; \
    wget -q 'http://pgl.yoyo.org/adservers/serverlist.php?hostformat=nohtml&showintro=1&startdate%5Bday%5D=&startdate%5Bmonth%5D=&startdate%5Byear%5D=&mimetype=plaintext' -O - ; \
    wget -q 'http://malwaredomains.lehigh.edu/files/BOOT' -O - | grep PRIMARY | cut -d " " -f 2; \
} | sort -u | sed -e 's/.*/\0\tCNAME\t.\n*.\0\tCNAME\t./' >> $blacklistfile

/etc/init.d/kresd reload