Adblock package release for turris omnia


#1

Hi,

in the OpenWrt stable & snapshot package repo you’ll find the first turris omnia compatible adblock package (plus LuCI companion/configuration package) with native kresd support.

current stable version in LEDE 17.01.: adblock 3.4.3 plus luci companion package

latest snapshot version: adblock 3.5.2 plus luci companion package

Link to the latest adblock documentation

Comprehensive adblock related Turris wiki page with more detailed installation instructions.


Installation:

  1. copy both packages (e.g. via winscp) to your router and install them …
    opkg install <package-name>

  2. enable the adblock service …
    /etc/init.d/adblock enable

  3. all other options should be configured via LuCI-GUI,
    see screenshots in the second post, also check the online documentation


Limitations:

  • adblock blocks domain access by dns/kresd. Whenever you run into problems with your dns resolver or some pages are not loading, please first try to suspend the adblock service before you contact the turris support. Furthermore you can use the adblock query function (available in LuCI advanced section or via ssh commandline) to check if a certain domain will be blocked by adblock.

  • the current chaos calmer based omnia release level does not support network interface trigger and does not support uclient-fetch as download utility, therefore please only use the “timed” startup trigger and “wget” as download utility


Changelog


release 3.5.2

  • add generic blocklist archive support
  • add support for blacklist archive from Toulouse 1 University Capitole
  • add support for urlhaus RPZ domains by abuse.ch
  • archive sub-categories (shalla & ut_capitole) are now configurable via LuCI CBI template
  • small bugfixes & enhancements

    release 3.5.1
  • maintenance update, just small bugfixes

    release 3.5.0
  • major performance boost: add a flexible ‘Download Queue’ to handle downloads & list processing in parallel, default queue size is ‘4’, you can raise this e.g. to ‘8’ or ‘16’ to get it really fast
  • replace former ‘whitelist mode’: the new ‘Jail’ option builds an additional ‘adb_list.jail’ list in parallel to block access to all domains except those listed in the whitelist file, which can be used manually for guest wifi or kidsafe configurations
  • regex parser & query function now fully support IDN domains with non-ASCII characters
  • add error handling in tld compression, to handle OOM conditions better
  • adblock.notify sends now html emails, to get a better look & feel, even on mobile devices
  • add czech regional blocklist maintained by turris omnia users
  • LuCI: Support new ‘Download Queue’ & ‘Jail’ options
  • LuCI: fix field width in “Runtime Information” section

    release 3.4.3
  • add pidfile writing / check to prevent further race conditions
  • ease the download utility selection: uclient-fetch (default), wget, curl, aria2c, wget-nossl, busybox-wget are fully pre-configured available
  • add debug download logging in case of an error, e.g. wrong url
  • change ‘malware’ blocklist source url
  • add logfile information to email template
  • LuCI: add ‘Download Utility’ select box
  • LuCI: add new “running” status

    release 3.4.1
  • enable code to support Turris Omnia forthcoming upstream change
    (new kresd ‘keep_cache’ option) to preserve kresd DNS cache
  • fix a ‘status’ race condition while the adblock process is running in parallel
  • various small speed improvements
  • rework debug output
  • refine blacklist handling
  • enable the (empty) blacklist source in the default config
  • email notification supports mstmp, even without sendmail symlink
  • email notification writes minimal status to log (one-liner)
  • LuCI: refine logfile search term
  • LuCI: Textarea ‘autoscroll down’ in logfile view
  • LuCI: Left-align blocklist source table plus a more compact design

    release 3.4.0
  • preserve DNS cache after adblock processing,
    • ‘unbound’ and ‘named’ support this (please check readme)
    • ‘dnsmasq’ now uses the ‘servers-file’ directive to minimize the reload disruption,
      even though the dns cache will be cleared after SIGHUP
    • ‘kresd’ dns cache is persistent by upstream default, anyway Turris Omnia devices
      need a small software change which is not implemented yet, see
      Proposal: keep / preserve kresd dns cache after restart (via config option)
  • email notification in case of an error or domain count < n (default 0, check readme)
  • removed securemecca from default config (service has been closed, read http://securemecca.blogspot.de/)
  • new separate functions for hash compare and list/overall count
  • add missing package dependencies
  • various clean-ups
  • update documentation

    release 3.1.1
  • new function to set/delete options in external uci config files
    • kresd: automated ‘rpz_file’ handling in /etc/config/resolver
    • firewall: automated ‘force_dns’ handling if you enable or disable adblock
  • support sha256sum (default) and md5sum for blocklist comparison & conditional dns restarts
  • cosmetics

    release 3.1.0
  • add ‘whitelist mode’, block access to all domains except those explicitly listed in the whitelist file
  • rework awk regex for all blocklist sources
  • include ‘third-party’ domains for all regional lists
  • change adguard url and refine filter ruleset
  • use POSIX character classes
  • fix regex for whitelist preparation
  • fix corner case parsing issues
  • fix enable/disable behavior
  • various other small fixes
  • documentation update
  • caution: config file update required!

    release 3.0.3
  • add new list source to default config to block browser-based crypto mining

    release 3.0.2
  • better system information
  • several kresd related documentation fixes

    release 3.0.1
  • fix startup issues with backends like dnscrypt-proxy or kresd which does not come up without an existing block list
  • fix a small ‘chown’ issue

    release 3.0.0
  • add kresd & turris omnia support
  • add dnscrypt-proxy support
  • change start priority to 30, to fix possible trigger issues on slow booting hardware
  • simplify suspend/resume handling (no longer use a hideout directory)
  • default config change (please update your config!), adblock is now disabled by default
  • enhanced LuCI frontend
  • many small changes & improvements
  • documentation update

Have fun!
Dirk


How to install adblock
New Omnia owner: Looking for pointers on projects and goals
Webinterface (LUCI/Foris) cannot be reached anymore
Adblock not creating/updating adb_list.overall (Omnia with adblock1.3.3-1)
Some questions from potential buyer
Webinterface (LUCI/Foris) cannot be reached anymore
Time limit for editing old messages/threads?
Webinterface (LUCI/Foris) cannot be reached anymore
Adblock not creating/updating adb_list.overall (Omnia with adblock1.3.3-1)
#2

Overview page:



Logfile-Viewer:

Advanced Options:

Domain Query:


#3

Sorry @dibdot,

I couldn’t get it working. I did the following:

wget https://downloads.lede-project.org/snapshots/packages/x86_64/packages/adblock_3.0.1-1_all.ipk
wget https://downloads.lede-project.org/snapshots/packages/x86_64/luci/luci-app-adblock_git-17.255.51369-6d4370d-1_all.ipk
opkg install adblock_3.0.1-1_all.ipk
opkg install luci-app-adblock_git-17.255.51369-6d4370d-1_all.ipk

Then:

root@sr-router:~# uci show adblock.global
adblock.global=adblock

adblock.global.adb_manmode='0’
adblock.global.adb_enabled=‘1’

adblock.global.adb_dns='kresd’
adblock.global.adb_debug=‘1’

But I get (this is exactly what I get, i.e. no output):

root@sr-router:~# /etc/init.d/adblock start
root@sr-router:~# /etc/init.d/adblock status
root@sr-router:~# cat /var/log/messages | grep adb

Also, cat /var/log/messages | tail provides no useful output.

Any ideas?


#4

Yes, please enter …
/etc/init.d/adblock enable
… and try it again - that happend in LEDE automatically. Sorry, I’ll add that additional step in the first post.


#5

Easier setup for kresd, without having to mess with custom configuration. If you have TurrisOS > 3.6, add

list rpz_file "/etc/kresd/adb_list.overall"

to /etc/config/resolver and restart the resolver itself.

Proof of working from a client machine. Before:

[lb@leon ~]$ host ziheyuan.com
ziheyuan.com is an alias for mymb.jumi.com.
mymb.jumi.com has address 47.89.60.137

After:

[lb@leon ~]$ host ziheyuan.com
Host ziheyuan.com not found: 3(NXDOMAIN)

This post is where the config option is mentioned.


Network-level ad blocking
How to install adblock
#6

As an addition, I need to start adblock manually via adblock.sh or it won’t work:

root@seldon:~# /etc/init.d/adblock enable
root@seldon:~# /etc/init.d/adblock start
root@seldon:~# /etc/init.d/adblock status

After:

root@seldon:~# /usr/bin/adblock.sh
root@seldon:~# /etc/init.d/adblock status
::: adblock runtime information
  + adblock_status  : enabled
  + adblock_version : 3.0.1
  + blocked_domains : 5333
  + fetch_utility   : wget (built-in)
  + dns_backend     : kresd (/etc/kresd)
  + last_rundate    : 14.09.2017 07:09:47
  + system_release  : OpenWrt omnia 15.05

#7

Thanks, I will add this alternative approach to the online documentation.


#8

Please post the global section of your adblock config.


#9

Thanks, here it is:

config adblock 'global'
        option adb_forcesrt '0'
        option adb_forcedns '0'
        option adb_whitelist '/etc/adblock/adblock.whitelist'
        option adb_whitelist_rset '\$1 ~/^([A-Za-z0-9_-]+\.){1,}[A-Za-z]+/{print tolower(\"^\"\$1\"\\\|[.]\"\$1)}'
        option adb_backup '0'
        option adb_backupdir '/mnt'
        option adb_rtfile '/tmp/adb_runtime.json'
        option adb_enabled '1'
        option adb_dns 'kresd'
        option adb_trigger 'timed'
        option adb_manmode '0'
        option adb_debug '0'

#10

config looks OK, please make sure that you’ve a S30adblock-entry in /etc/rc.d (if not start /etc/init.d/adblock enable once) and reboot your router … and look afterwards for log entries with …
cat /var/log/messages | grep "adblock"


#11

I tried to debug a bit myself but I don’t understand LEDE/openWRT’s boot sequence.
Debugging doesn’t give me that much information:

root@seldon:/etc/rc.d# /etc/init.d/adblock start
{ "name": "adblock", "script": "\/etc\/init.d\/adblock", "instances": { "adblock": { "command": [ "\/usr\/bin\/adblock.sh" ], "stdout": true, "stderr": true } }, "triggers": [ [ "config.change", [ "if", [ "eq", "package", "adblock" ], [ "run_script", "\/etc\/init.d\/adblock", "reload" ] ] ] ] }

(I can’t reboot the router at the moment, but the symlink is there)


#12

That looks all OK, just a rough guess: during your first reboot test you’ve the adb_trigger ‘wan’ in your config and that’s currently not supported on turris omnia devices. ‘timed’ should work in any case.


#13

Briefly looking at the docs, having multiple policy.add(policy.all lines does not do what you intend. The first matching rule wins; if you want kresd to choose automatically from a set of (up to four) IPs, you have to pass a list into a single rule, e.g. see examples in kresd docs.


#14

Thanks for your input, so I assume that this is a correct example, isn’t it?

policy.add(policy.all(policy.FORWARD({‘8.8.8.8’, ‘8.8.4.4’})))


#15

Correct.
Post must be at least 20 characters.


#16

Well, I feel pretty stupid for missing that one out!


#17

After installing the packages mentioned in first post, i get an error in luci:

/usr/lib/lua/luci/dispatcher.lua:460: Failed to execute cbi dispatcher target for entry '/admin/services/adblock'.
The called action terminated with an exception:
/usr/lib/lua/luci/cbi.lua:53: Model 'adblock' not found!
stack traceback:
[C]: in function 'assert'
/usr/lib/lua/luci/dispatcher.lua:460: in function 'dispatch'
/usr/lib/lua/luci/dispatcher.lua:141: in function </usr/lib/lua/luci/dispatcher.lua:140>

Anyone idea?


#18

Probably a LuCI caching issue. Please submit both commands:
> rm -rf /tmp/luci-*
> /etc/init.d/lighttpd restart

After that it should work … fingers crossed! :wink:


#19

Working like a charm, had this before, but didn’t remember the solution! =)


#20

Cant wait to Adblock in official turris repository (possible with sha256sum) :slight_smile: What do you think @miska? :slight_smile: