Internal network including dhcp working fine this morning. WAN wasn’t operational. Firewall machine on the wan pipe had good connectivity. Since I access the firewall box via ssh over the Omnia, that indicated to me that routing thru the wan was functioning, even though other traffic from the network (.i.e email, Omnia trying to send statistics, web) was failing. I restarted resolver, odhcpd and firewall processes on the Omnia - wan is working once again for all traffic.
Some info from last night’s log that I’d appreciate folks’ assessments of:
-
Resolver is being restarted frequently - this is from the last 1000 lines of messages log:
2017-01-12T11:45:07-05:00 warning watchdog[]: Restarted resolver
2017-01-12T11:55:07-05:00 warning watchdog[]: Restarted resolver
2017-01-12T12:05:07-05:00 warning watchdog[]: Restarted resolver
2017-01-12T12:15:07-05:00 warning watchdog[]: Restarted resolver
2017-01-12T12:25:07-05:00 warning watchdog[]: Restarted resolver
2017-01-12T12:35:07-05:00 warning watchdog[]: Restarted resolver
2017-01-12T12:45:08-05:00 warning watchdog[]: Restarted resolver
2017-01-12T12:55:07-05:00 warning watchdog[]: Restarted resolver
2017-01-12T13:05:07-05:00 warning watchdog[]: Restarted resolver
2017-01-12T13:15:08-05:00 warning watchdog[]: Restarted resolver
2017-01-12T13:25:08-05:00 warning watchdog[]: Restarted resolver
2017-01-12T13:35:07-05:00 warning watchdog[]: Restarted resolver
2017-01-12T13:45:07-05:00 warning watchdog[]: Restarted resolver
2017-01-12T13:55:08-05:00 warning watchdog[]: Restarted resolver
-
No messages in last 1000 log lines about odhcpd
30 firewall messages in last 1000 log lines seem operational, not indicative of errors:
2017-01-12T12:00:01-05:00 info /usr/sbin/cron[20875]: (root) CMD (/usr/share/firewall/turris-download)
2017-01-12T12:00:11-05:00 err turris-firewall-rules[]: (v62) Failed to download https://api.turris.cz/firewall/turris-ipsets.gz.sign
2017-01-12T12:05:01-05:00 info /usr/sbin/cron[21313]: (root) CMD (/usr/share/firewall/turris)
2017-01-12T12:05:01-05:00 info turris-firewall-rules[]: (v62) IPv4 WAN interface used - 'eth1’
2017-01-12T12:05:01-05:00 info turris-firewall-rules[]: (v62) IPv6 WAN interface used - 'lo’
2017-01-12T12:05:02-05:00 info turris-firewall-rules[]: (v62) 3403 ipv4 address(es) and 0 ipv6 address(es) were loaded (bc4b7d351917fe6864073e9991d55b9d), 0 rule(s) overriden, 0 rule(s) skipped
2017-01-12T13:00:01-05:00 info /usr/sbin/cron[26560]: (root) CMD (/usr/share/firewall/turris-download)
2017-01-12T13:00:12-05:00 err turris-firewall-rules[]: (v62) Failed to download https://api.turris.cz/firewall/turris-ipsets.gz.sign
2017-01-12T13:05:01-05:00 info /usr/sbin/cron[26999]: (root) CMD (/usr/share/firewall/turris)
2017-01-12T13:05:01-05:00 info turris-firewall-rules[]: (v62) IPv4 WAN interface used - 'eth1’
2017-01-12T13:05:01-05:00 info turris-firewall-rules[]: (v62) IPv6 WAN interface used - 'lo’
2017-01-12T13:05:02-05:00 info turris-firewall-rules[]: (v62) 3403 ipv4 address(es) and 0 ipv6 address(es) were loaded (bc4b7d351917fe6864073e9991d55b9d), 0 rule(s) overriden, 0 rule(s) skipped
2017-01-12T14:00:01-05:00 info /usr/sbin/cron[2390]: (root) CMD (/usr/share/firewall/turris-download)
2017-01-12T14:00:11-05:00 err turris-firewall-rules[]: (v62) Failed to download https://api.turris.cz/firewall/turris-ipsets.gz.sign
2017-01-12T14:01:19-05:00 info turris-firewall-rules[]: (v62) IPv4 WAN interface used - 'eth1’
2017-01-12T14:01:19-05:00 info turris-firewall-rules[]: (v62) IPv6 WAN interface used - 'lo’
2017-01-12T14:01:20-05:00 info turris-firewall-rules[]: (v62) 3403 ipv4 address(es) and 0 ipv6 address(es) were loaded (bc4b7d351917fe6864073e9991d55b9d), 0 rule(s) overriden, 0 rule(s) skipped
2017-01-12T14:02:00-05:00 notice firewall[]: Reloading firewall due to ifup of wan (eth1)
2017-01-12T14:02:00-05:00 info turris-firewall-rules[]: (v62) IPv4 WAN interface used - 'eth1’
2017-01-12T14:02:00-05:00 info turris-firewall-rules[]: (v62) IPv6 WAN interface used - 'lo’
2017-01-12T14:02:02-05:00 info turris-firewall-rules[]: (v62) 3403 ipv4 address(es) and 0 ipv6 address(es) were loaded (bc4b7d351917fe6864073e9991d55b9d), 0 rule(s) overriden, 0 rule(s) skipped
2017-01-12T14:05:01-05:00 info /usr/sbin/cron[13134]: (root) CMD (/usr/share/firewall/turris)
2017-01-12T14:05:01-05:00 info turris-firewall-rules[]: (v62) IPv4 WAN interface used - 'eth1’
2017-01-12T14:05:01-05:00 info turris-firewall-rules[]: (v62) IPv6 WAN interface used - 'lo’
2017-01-12T14:05:02-05:00 info turris-firewall-rules[]: (v62) 3403 ipv4 address(es) and 0 ipv6 address(es) were loaded (bc4b7d351917fe6864073e9991d55b9d), 0 rule(s) overriden, 0 rule(s) skipped
-
The LuCI status overview section for mwan indicated that the primary (currently only - configuring second wan is a task for today or tomorrow) wan was offline. After restarting the 3 processes above, that page once again showed primary wan as enabled rather than offline.
-
Routing functions were working (i.e. I could access the firewall machine on wan line even though external dns resolution was failing). DNS resolution from the firewall machine itself was tested and working fine.
I am leaning toward the resolver messages indicating a problem. Next time (if this happens again) I will reload only one process at a time - starting with resolver - to isolate cause. Would internal traffic from x.x.1.0 lan to firewall on x.x.10.0 route even though dns resolution was failing? I think so - checking opinions on that. Resolver problems would explain the dns resolution failures. Any thoughts on the log info or general scenario?