Knot resolver and RPZ

Giblet · January 6, 2017, 6:27pm

Hey All, I would love to get some adblocking working for my omnia turris.
Would it be possible for someone to either point me to a guide, or maybe even write one on here?

Simon_Walker · January 6, 2017, 7:14pm

I would also really appreciate it, if they remembered that not everyone is a Unux whizz and the 3(?) (FCI,LUCI and Putty connect) different ways of accessing and changing things on the Omnia, aren’t a help, rather a hinderence for us basic users.
I read these suggestions and my brain starts to fizz!

I must admit to finding it really painful that there is an adblock app(plug-in whatever you want to call it), which doesn’t work with the basic system, yet is one that the update process seems dedicated to install.

I understand that people are maxed out, and this is the Christmas/new year holiday period. But this software has, I understand, been in development for ages now, yet it still seems to be full of contradictions and requirements for things to be done in 2 or 3 different places.

I thought the device was supposed to get getting regular updates and foolishly, I had hoped these updates would make things easier for me to use the device, but so far I’ve not seen it.

I also understand we’re at early days here, but at the moment it feels we are heading towards something you need a degree in Unix and its config to set up, as opposed to an end user friendly device which works neatly from a single interface

vcunat · January 6, 2017, 7:30pm

To me personally it doesn’t seem like ad-blocking is a basic router function that needs to work just by clicking a checkbox in Foris. AFAIK most (cheap) routers won’t let you do any advanced stuff at all; here you can do whatever you want with the system.

Leonardo · January 7, 2017, 3:58am

I would suggest to install ‘nano’, it is a very simple file editor.
Otherwise you can look at ‘vi’.

This is the script I use, very similar to @Cabal script.

#!/bin/bash

blacklistfile="/etc/kresd/blacklist.rpz"

cat > $blacklistfile <<EOF
\$TTL 60
@               IN      SOA  localhost. root.localhost.  (
                        2   ; serial
                        3H  ; refresh
                        1H  ; retry
                        1W  ; expiry
                        1H) ; minimum
                IN      NS    localhost.

EOF


{
    curl 'https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts' | gr                                           ep "^0" | cut -d " " -f 2; \
} | sort -u | sed -e 's/$/\tCNAME\t./' >> $blacklistfile


/etc/init.d/kresd reload

Be sure to add:
policy.add(policy.rpz(policy.DENY, '/etc/kresd/blacklist.rpz'))
to
/etc/kresd/custom.conf

Giblet · January 7, 2017, 1:05pm

So for everyone willing to try this.
I took the guide jklaas posted and combined it with other info in the thread to make a beginner guide.

1). First open /etc/config/resolver with a text editor such as vi:

vi /etc/config/resolver

Look for the following:

config resolver 'kresd'
       option rundir '/tmp/kresd'
       option log_stderr '1'
       option log_stdout '1'
       option forks '1'
       #option include_config '/tmp/kresd.custom.conf'
       #option hostname_config '/etc/hosts'

And change it to:

config resolver 'kresd'
       option rundir '/tmp/kresd'
       option log_stderr '1'
       option log_stdout '1'
       option forks '1'
       option include_config '/etc/kresd/custom.conf'
       #option hostname_config '/etc/hosts'

Also look for the following:

config resolver 'common'
        list interface '0.0.0.0'
        list interface '::0'
        option port '53'
        option keyfile '/etc/root.keys'
        option verbose '0'
        option msg_buffer_size '4096'
        option msg_cache_size '20M'
        option net_ipv6 '1'
        option net_ipv4 '1'
        option forward_upstream '1'
        option prefered_resolver 'kresd'
        option ignore_root_key '0'
        option prefetch 'yes'

And change it to the following:

config resolver 'common'
        list interface '0.0.0.0'
        list interface '::0'
        option port '53'
        option keyfile '/etc/root.keys'
        option verbose '0'
        option msg_buffer_size '4096'
        option msg_cache_size '20M'
        option net_ipv6 '1'
        option net_ipv4 '1'
        option forward_upstream '0'
        option prefered_resolver 'kresd'
        option ignore_root_key '0'
        option prefetch 'yes'

2). Open /etc/kresd/custom.conf with a text editor:

vi /etc/kresd/custom.conf

Add the following lines in the specified to the file:

policy.add(policy.rpz(policy.DENY, '/etc/kresd/blacklist.rpz'))
policy.add(policy.all(policy.FORWARD('8.8.8.8')))
policy.add(policy.all(policy.FORWARD('8.8.4.4')))

3). Create a script file called “generate_blacklist.sh” in the following location /usr/local witht he following command:

vi /etc/kresd/generate_blacklist.sh

Add the following text/script:

#!/bin/bash

blacklistfile="/etc/kresd/blacklist.rpz"

cat > $blacklistfile <<EOF
\$TTL 60
@		IN	SOA  localhost. root.localhost.  (
			2   ; serial 
			3H  ; refresh 
			1H  ; retry 
			1W  ; expiry 
			1H) ; minimum 
		IN	NS    localhost.

EOF

{
    wget -q 'http://malwaredomains.lehigh.edu/files/immortal_domains.txt' -O - | grep -v \#; \
    wget -q 'http://pgl.yoyo.org/adservers/serverlist.php?hostformat=nohtml&showintro=1&startdate%5Bday%5D=&startdate%5Bmonth%5D=&startdate%5Byear%5D=&mimetype=plaintext' -O - ; \
    wget -q 'http://malwaredomains.lehigh.edu/files/BOOT' -O - | grep PRIMARY | cut -d " " -f 2; \
} | sort -u | sed -e 's/.*/\0\tCNAME\t.\n*.\0\tCNAME\t./' >> $blacklistfile

/etc/init.d/kresd reload

4). Make the above script exacuteable with the following command:

chmod +x /etc/kresd/generate_blacklist.sh

5). Create a cronjob to exactue the above script, make a cronjob with the following command:

crontab -e

And add the following line:

6 1 * * 3 /etc/kresd/generate_blacklist.sh

That should be all…
Please give love/thanks to jklaas, cabal, Leonardo and all others that helped in this thread

P.S. If I made any mistakes please let me know and I’ll correct them asap

Giblet · January 7, 2017, 2:27pm

Hmmzz,
So I used the above to setup my adblocking.
But it doesn;t block anything.
I see the blacklists gets created and contains thousands of entries…
But I suspect requests aren’t checked against it.
Is there anything I’m forgetting?

Pepe · January 7, 2017, 2:52pm

Why it shouldnt be there? In Foris in updater we can find for example tor, web cam, tvheadend, majordomo and etc.
I think it’s good idea to have it there!

In other case on some cheap router you can replace stock firmware with OpenWRT/DD-WRT/LEDE and then you can have ad-blocking

Cabal · January 7, 2017, 3:41pm

I think that policy order is important.
In my case I have forwarding disabled (via foris or by setting 0 in /etc/config/resolver somewhere around forward.
And I’ve added forwarders to /etc/kresd/custom.conf (after that line with DENY):

policy.add(policy.rpz(policy.DENY, '/etc/kresd/blacklist.rpz'))
policy:add(policy.all(policy.FORWARD('8.8.8.8')))
policy:add(policy.all(policy.FORWARD('8.8.4.4')))

You can get those forwarders by typing (before disabling forwarding):

cat /tmp/kresd.conf

Giblet · January 7, 2017, 4:43pm

Why does your first line begin with: policy.add
and the subsequent lines start with: policy:add
?

Cabal · January 7, 2017, 5:05pm

It’s quite funny, but looks like both versions are ok.

Docs (http://knot-resolver.readthedocs.io/en/latest/modules.html#query-policies) says to use dot, but /etc/init.d/kresd uses colon to generate forwarders.

I changed it to dot version as docs says.

white · January 7, 2017, 6:27pm

policy:add(action) is equal with policy.add(policy, action). They are Knot Resolver 1.0.0 API calls and policy.add(action) is 1.1.0 API call. The support for policy:add(action) will be removed in 1.2.0 API.

Giblet · January 7, 2017, 6:35pm

Working now!

Updated my “guide”

Leonardo · January 7, 2017, 7:01pm

sort -u
It removes duplicates.

lampra · January 28, 2017, 9:05pm

Thank you all for this.

Now, while this works I do have a problem.
In order to include some own domains I use a second file and I added a policy
policy.add(policy.rpz(policy.DENY, '/etc/kresd/ownbl.rpz'))
While for almost all domains this works, there is at least one domain that this is not working.
For stoiximan.gr this is working but not for www.stoiximan.gr

#dig stoiximan.gr
; <<>> DiG 9.9.8-P4 <<>> stoiximan.gr
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 35433
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;stoiximan.gr.			IN	A

;; AUTHORITY SECTION:
blocked.		900	IN	SOA	blocked. . 0 3600 900 604800 900

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Sat Jan 28 23:07:21 EET 2017
;; MSG SIZE  rcvd: 83

BUT

#dig www.stoiximan.gr
    ; <<>> DiG 9.9.8-P4 <<>> www.stoiximan.gr
    ;; global options: +cmd
    ;; Got answer:
    ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 58848
    ;; flags: qr rd ra ad; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 1

    ;; OPT PSEUDOSECTION:
    ; EDNS: version: 0, flags:; udp: 4096
    ;; QUESTION SECTION:
    ;www.stoiximan.gr.		IN	A

    ;; ANSWER SECTION:
    www.stoiximan.gr.	242	IN	CNAME	stoiximan.gr.
    stoiximan.gr.		242	IN	A	104.16.102.199
    stoiximan.gr.		242	IN	A	104.16.99.199
    stoiximan.gr.		242	IN	A	104.16.98.199
    stoiximan.gr.		242	IN	A	104.16.101.199
    stoiximan.gr.		242	IN	A	104.16.100.199

    ;; Query time: 105 msec
    ;; SERVER: 127.0.0.1#53(127.0.0.1)
    ;; WHEN: Sat Jan 28 23:02:12 EET 2017
    ;; MSG SIZE  rcvd: 139

vcunat · January 28, 2017, 9:22pm

The policies are only applied to incoming queries, so if that RPZ only matches stoiximan.gr and not www.stoiximan.gr, this would be OK.

lampra · January 29, 2017, 11:51am

Thank you for the answer. Indeed it’s ok

Cabal · January 29, 2017, 12:33pm

Check my script.
I’ve updated sed parameters and now it creates entries for domain and wildcard for subdomains:

domain.com CNAME .
*.domain.com CNAME .

@Giblet - update Your guide

Giblet · January 31, 2017, 6:27am

Done, and thanks for the headsup!

okam · February 5, 2017, 5:02pm

Hi, can you please summarize the above described procedure step by step for a man who does not control the Linux commands. Thank you

Giblet · February 5, 2017, 6:52pm

My post/guide contains the EXACT commands needed to get this to work.
I am not sure what I could do to make it any more clear