Starting with the update to 3.11, DNS keeps breaking, so I perfomed a factory reset (3 LED). I changed the SSID and passwords, but all other settings are default. I updated to 3.11.1.
I also factory reset my cable modem (connected to Comcast). I turned off DHCP and WiFi, then enabled Rg passthrough (“bridge mode”), specifying the Turris Omnia WAN MAC address.
After a day or so, most devices cannot load websites because the DNS does not work. The devices that specify a DNS server (e.g. 8.8.8.8 in MacOS system preferences) on their own work fine. Using foris, I have tried the following:
“Use forwarding” unchecked
“Use forwarding” checked, DNS forwarder: Cloudflare
“Use forwarding” checked, DNS forwarder: Use provider’s DNS resolver, “Disable DNSSEC” checked.
After about a day, when DNS stops working, if I switch between any of the above it instantly starts working without a reboot (even if I switch back to the one that was just broken). I tested the second two configurations on 3.11, and the first configuration (no forwarding) on 3.11.1. Now I’m testing the second (use Cloudflare) on 3.11.1. Tomorrow, assuming it breaks, I’ll test the third (use Comcast with no DNSSEC).
Tomorrow I’ll try echo 'verbose(true)' | socat - /tmp/kresd/tty/"$(pgrep kresd)"
and post the logs. I did notice this error when I log into foris:
My DNS broke similar way after updating to 3.11.1 (didn’t notice any issues under 3.11).
I didn’t have time to investigate it yet, but from my logs it looks like /tmp/kresd/hints.tmp is corrupted on the first run. After restarting DNS subsystem via Foris web interface, the file seems to fix itself.
Initial run:
2018-12-22T04:43:24+01:00 info dnsmasq[2931]: started, version 2.78 DNS disabled
2018-12-22T04:43:24+01:00 info dnsmasq[2931]: compile time options: IPv6 GNU-getopt no-DBus no-i18n no-IDN DHCP no-DHCPv6 no-Lua TFTP no-conntrack no-ipset no-auth no-DNSSEC loop-detect inotify
2018-12-22T04:43:24+01:00 info dnsmasq-dhcp[2931]: DHCP, IP range 192.168.4.10 -- 192.168.4.159, lease time 12h
2018-12-22T04:43:24+01:00 info dnsmasq-dhcp[2931]: DHCP, IP range 192.168.3.10 -- 192.168.3.159, lease time 12h
2018-12-22T04:43:24+01:00 info dnsmasq-dhcp[2931]: read /etc/ethers - 0 addresses
2018-12-22T03:43:24+01:00 info dhcp_host_domain_ng.py[]: DHCPv4 new lease
2018-12-22T03:43:24+01:00 info dhcp_host_domain_ng.py[]: DHCP update hostname [old,whale,192.168.3.201]
2018-12-22T03:43:24+01:00 err dhcp_host_domain_ng.py[]: Kresd socket failed:<class 'socket.error'>,[Errno 104] Connection reset by peer
2018-12-22T03:43:24+01:00 err dhcp_host_domain_ng.py[]: Wrong host format '/tmp/kresd/hints.tmp' in host file 192.168.3.201 whale.lan
2018-12-22T03:43:24+01:00 err dhcp_host_domain_ng.py[]: Kresd socket failed:<class 'socket.error'>,[Errno 104] Connection reset by peer
2018-12-22T03:43:24+01:00 err dhcp_host_domain_ng.py[]: Wrong host format '/tmp/kresd/hints.tmp' in host file 192.168.3.202 minime.lan
After kicked by Foris web interface:
2018-12-22T03:44:34+01:00 info dhcp_host_domain_ng.py[]: Refresh kresd leases
2018-12-22T04:44:34+01:00 info dnsmasq[2931]: exiting on receipt of SIGTERM
2018-12-22T04:44:34+01:00 info dnsmasq[4138]: started, version 2.78 DNS disabled
2018-12-22T04:44:34+01:00 info dnsmasq[4138]: compile time options: IPv6 GNU-getopt no-DBus no-i18n no-IDN DHCP no-DHCPv6 no-Lua TFTP no-conntrack no-ipset no-auth no-DNSSEC loop-detect inotify
2018-12-22T04:44:34+01:00 info dnsmasq-dhcp[4138]: DHCP, IP range 192.168.5.100 -- 192.168.5.249, lease time 12h
2018-12-22T04:44:34+01:00 info dnsmasq-dhcp[4138]: DHCP, IP range 192.168.4.10 -- 192.168.4.159, lease time 12h
2018-12-22T04:44:34+01:00 info dnsmasq-dhcp[4138]: DHCP, IP range 192.168.3.10 -- 192.168.3.159, lease time 12h
2018-12-22T04:44:34+01:00 info dnsmasq-dhcp[4138]: read /etc/ethers - 0 addresses
2018-12-22T03:44:34+01:00 info dhcp_host_domain_ng.py[]: DHCPv4 new lease
2018-12-22T03:44:34+01:00 warning dhcp_host_domain_ng.py[]: Add_lease, hostname check failed
2018-12-22T03:44:34+01:00 info dhcp_host_domain_ng.py[]: DHCP update hostname [old,whale,192.168.3.201]
2018-12-22T03:44:35+01:00 info kresd[3700]: > hints.del('turris')
2018-12-22T03:44:35+01:00 info kresd[3700]: [result] => true
2018-12-22T03:44:35+01:00 info kresd[3700]:
2018-12-22T03:44:35+01:00 info kresd[3700]: > hints.del('minime')
2018-12-22T03:44:35+01:00 info kresd[3700]: [result] => true
2018-12-22T03:44:35+01:00 info kresd[3700]:
2018-12-22T03:44:35+01:00 info kresd[3700]: > hints.del('whale')
2018-12-22T03:44:35+01:00 info kresd[3700]: [result] => true
2018-12-22T03:44:35+01:00 info kresd[3700]:
I just upgraded to 3.11.1, and DNS on my TO stopped working as well. Just after 10 minutes after a cold reboot. What I noticed is that the kresd process started out quickly after reboot as a 77 Mbyte process, and in 10 minutes (with a low load on the network) it grew to 108 Mbyte.
I saw similar behavior (process growing in size and stopping all work soon) of kresd in 3.11 with DNS over TLS enabled. But DNS over TLS is now off!
With DNS failing constantly, complaints from the family are countless. I kindly ask to look into fixing these DNS issues with priority. Thank you.
So far we haven’t experienced anything like that. I’ve been even using TLS forwarding on Omnia for me and some relatives, for weeks without issues noticeable by laymen.
I’m afraid the problem will be hard to find without us getting more information somehow, e.g. I’d start with verbose logs from a failing moment, ideally around the moment it starts to fail but not necessarily. Also note that everyone is on vacation now
@vcunat I have the same issue. What I want to add is that simple restart doesn’t help me every time. Sometimes I have to comment out TLS_FORWARD section in /etc/kresd/custom.conf:
I want to add my name to the same problem, I too saw a similar behavior with kresd (process growing in size and stopping all work) in 3.11 and 3.11.1 with DNS over TLS enabled from foris (quad9999 and cloudflare). I did a factory reset to the latest medkit and I did not restore any settings, I used the default config files.
I turned off forwarding and using DNS over TLS in foris, I deleted the foris TLS line in the resolver config file that is listed under “common” and add to the “kresd” section `option include_config ‘/etc/kresd/custom.conf’. I made sure that Forward_upstream was off.
This has been working for the past 5 days without a drop in dns resolving. I agree with the others, there is a problem somewhere between kresd and the resolver. Just my 2 cents.
Ok, I started with your command echo 'verbose(true)' | socat - /tmp/kresd/tty/"$(pgrep kresd)" and got the > true as answer.
Right after that I changed DNS-settings via FORIS to
and ran the connection test - the status is already shown in the picture. DNS stopped working instantly - webpages can be accessed directly via IP (e.g. 1.1.1.1), but not by TLD. When running /etc/init.d/resolver restart DNS works for a couple of seconds before it stops again. Syslog isn’t showing any additional entry.
After that I reverted DoT, went to Turris Documentation, followed the instructions, started debug-logging and activated DoT again. Ironically DNS is now working, despite connection test telling me it isn’t. I will post cleaned logs tomorrow.
Just note that the configuration commands to kresd through socat are only applied once, i.e. they only make effect until the process is restarted (such as when reconfigured from Foris). The buttons from the tutorial use the same inside, so it also applies to them as well.
I had a similar problem in 3.11.1
The problem was easy to fix: The DNS Serverport was set to “0”; after entering the value “53” everything worked fine. ( … Network … DHCP & DNS … Advanced Settings )
Sorry, but that cannot be the solution. Those settings go to dnsmasq, which only serves for dhcp entries on standard TO setup. DNS is done by kresd whose settings cannot be changed via luci.
The setting you mention is located in /etc/config/dhcp
config dnsmasq
option port '0'
and has been there unchanged at least since 3.10 - that’s my oldest snapshot which I just mounted to very that.
“Unfortunately” DNS wasn’t broken this night.
I saved the 1,4MB debug-log anyway. I will restart my TO and start debugging again. Hopefully I will have the error printed, when I return in 2 days.
edit: When starting the debug, I get right away three errors:
2018/12/24 12:35:22 socat[5733] E connect(5, AF=1 "/tmp/kresd/tty/5266", 21): No such file or directory
2018/12/24 12:35:23 socat[5847] E connect(5, AF=1 "/tmp/kresd/tty/", 17): Connection refused
2018/12/24 12:35:23 socat[5859] E connect(5, AF=1 "/tmp/kresd/tty/", 17): Connection refused
Ok, I’ve got some time to play with this today. Please scratch my previous message about /tmp/kresd/hints.tmp being corrupted on the first run. I found the real cause. It was my mistake.
Many months ago I tweaked my /etc/config/resolver by following https://doc.turris.cz/doc/en/public/dns_knot_misc specifically what is now labeled as “Setup in Turris OS 3.9.6 through 3.10.8”. I added option include_config '/etc/kresd/custom.conf' with this content:
This caused the consistent problem on first run for some reason. It probably conflicts with new DNS custom forwarding system added in 3.11 via option forward_custom.
But I identified another confusing issue when playing with Foris web interface under DNS page. It looks like when disabling “Use forwarding” the script is not clearing option forward_custom - it just flips option forward_upstream in /etc/config/resolver.
I enabled verbose logging in /etc/config/resolver via option verbose '1' and observed the logs.
after enabling “Use forwarding”, and “DNS Forwarder” to “Cloudflare (TLS)”, in /etc/config/resolver I can see option forward_upstream '1' and option forward_custom '99_cloudflare' (newly added), kresd verbose logs are properly mentioning CloudFlare’s 1.1.1.1
after disabling “Use forwarding”, I can see option forward_upstream '0' but option forward_custom '99_cloudflare' stayed intact, in logs I can still see CloudFlare’s 1.1.1.1 being used - I would expect my ISP’s server instead
after enabling “Use forwarding” and setting “DNS Forwarder” to “Use provider’s DNS resolver”, I can see “option forward_custom ‘99_cloudflare’” was removed and option forward_upstream '1'. This is as expected and logs mention my ISP servers.
fter disabling “Use forwarding” while having “DNS Forwarder” set to “Use provider’s DNS resolver”, I can see “option forward_custom ‘99_cloudflare’” is still removed and option forward_upstream '0'. This is as expected and logs mention my ISP servers.
TL;DR; after disabling “Use forwarding” option forward_custom ... should be removed. If this is correct behaviour then the Foris web UI is confusing, because it hides “DNS Forwarder” option.
Seems you are correct and that is the root cause. At least I have rebooted three times and DNS is ok every time while previously it was never working.
Pretty sad that an OS update can cause such serious incompatibility even with the settings slightly altered according to the official documentation. I expected better testing of the updates before making them public since the default configuration updates automatically and if I wasn’t near the router at the moment of that update, I’d lose an access to it completely.
What is strange as well is that all the facts were clearly described in that topic and you, not a member of Turris or Knot team described the solution.
extraTrees = policy.todnames({‘faketldtest’, ‘sld.example’, ‘internal.example.com’})
– Beware: the rule order is important, as STUB is not a chain action.
policy.add(policy.suffix(policy.FLAGS({‘NO_CACHE’}), extraTrees))
policy.add(policy.suffix(policy.STUB({‘2001:db8::1’}), extraTrees))
So today it worked out - I just got a full debug log containing the time when the DNS gets broken.
As this contains plenty of personal data - is there something specific I can search for as I don’t want to post my data and also don’t want to spend hours anonymizing the logs?
edit: snippet of time frame in which DNS got broken:
