HaaS and minipots outages

JardaB · June 8, 2022, 12:01pm

HaaS is not running again and HaaS ports and minipots are closed
Service restarts did not help, this is a recurring and long-standing issue:

root @ Turris_JB: ~ # /etc/init.d/adblock reload
root @ Turris_JB: ~ # /etc/init.d/sentinel-proxy reload
root @ Turris_JB: ~ # /etc/init.d/sentinel-dynfw-client reload
root @ Turris_JB: ~ # /etc/init.d/sentinel-minipot reload
root @ Turris_JB: ~ # /etc/init.d/haas-proxy reload
root @ Turris_JB: ~ #

Troubleshooting or other measures (what to restart, etc) have not been mentioned anywhere

Only reinstalling the minipots in reForis will help, then the monitored ports will open.

  • Removed package sentinel-minipot
  • Removed package logc-libevent
  • Removed package base64c

  • Installed version 0.2.1-1 of package base64c
  • Installed version 0.1.0-1 of package logc-libevent
  • Installed version 2.3.0-1 of package sentinel-minipot

hagrid · June 8, 2022, 12:46pm

Can you please submit your troubles in our Issue Tracker? Issues - Turris Documentation

JardaB · June 8, 2022, 1:29pm

I tried to enter something, maybe it’s telling, I’m not an expert on some debugging problem. Gitlab is an unknown territory with me labeled “there are lions”.

hagrid · June 8, 2022, 2:05pm

Thank you very much, somebody will ask you to provide more info if needed. Cheers!

JardaB · June 9, 2022, 10:17am

Today the problem was repeated, today it was enough to turn “Enable Minipots” on and off in reForis. There are some mentions of sentinel traffic in the syslog – but I not understand that.

Pause HaaS viz

2022-06-09 14:04:50 CHN - 36.110.228.254
2022-06-08 20:14:40 CHN - 218.92.0.158

I postponed the diagnostics

Jun 9 11:50:59 Turris_JB sentinel: INFO [certgen.action_spec_init:89] Valid certificate found
Jun 9 09:50:59 Turris_JB /dhcp_host_domain_ng.py: Kresd is probably not running no socket found.

   Jun  9 11:50:59 Turris_JB dnsmasq[11583]: script process exited with status 1
    Jun  9 11:50:59 Turris_JB dnsmasq-dhcp[11583]: read /etc/ethers - 0 addresses
    Jun  9 09:51:00 Turris_JB /dhcp_host_domain_ng.py: DHCP update hostname

Souhrn

[Fujitsu,192.168.2.120]
Jun 9 09:51:00 Turris_JB /dhcp_host_domain_ng.py: Refresh kresd leases
Jun 9 09:51:00 Turris_JB procd: Instance sentinel-fwlogs::instance1 pid 17075 not stopped on SIGTERM, sending SIGKILL instead
Jun 9 09:51:00 Turris_JB /dhcp_host_domain_ng.py: DHCP update hostname [EpsonXP700,192.168.2.118]
Jun 9 09:51:00 Turris_JB /dhcp_host_domain_ng.py: Refresh kresd leases
Jun 9 09:51:00 Turris_JB foris-controller[5918]: WARNING:foris_controller_backends.collectd:Socket error occured ‘[Errno 2] No such file or directory’
Jun 9 09:51:00 Turris_JB /dhcp_host_domain_ng.py: DHCP update hostname [Pixel-4a,192.168.2.115]
Jun 9 09:51:00 Turris_JB /dhcp_host_domain_ng.py: Refresh kresd leases
Jun 9 09:51:00 Turris_JB procd: Instance sentinel-minipot::instance1 pid 18925 not stopped on SIGTERM, sending SIGKILL instead
Jun 9 09:51:01 Turris_JB /dhcp_host_domain_ng.py: DHCP update hostname [Fujitsu,192.168.2.120]
Jun 9 09:51:01 Turris_JB /dhcp_host_domain_ng.py: Refresh kresd leases
Jun 9 09:51:01 Turris_JB /dhcp_host_domain_ng.py: DHCP update hostname [EpsonXP700,192.168.2.118]
Jun 9 09:51:01 Turris_JB sentinel-fwlogs[11966]: ERROR: Packet handling failed: Resource temporarily unavailable
Jun 9 09:51:01 Turris_JB /dhcp_host_domain_ng.py: Refresh kresd leases

Jun  9 11:51:20 Turris_JB sentinel: INFO [certgen.action_spec_init:89] Valid certificate found
Jun  9 09:51:21 Turris_JB foris-controller[5918]:

Souhrn

WARNING:foris_controller_backends.collectd:Socket error occured '[Errno 2] No such file or directory'
Jun  9 09:51:22 Turris_JB procd: Instance sentinel-fwlogs::instance1 pid 11966 not stopped on SIGTERM, sending SIGKILL instead
Jun  9 09:51:23 Turris_JB foris-controller[5918]: WARNING:foris_controller_backends.collectd:Socket error occured '[Errno 2] No such file or directory'
x
x
Jun  9 09:51:56 Turris_JB sentinel-fwlogs[12571]: ERROR: Packet handling failed: Resource temporarily unavailable

x

Jun  9 21:07:24 Turris_JB sentinel-dynfw-client[17840]: ipset v7.3: Error in line 1: Element cannot be deleted from the set: it's not added
Jun  9 21:07:24 Turris_JB sentinel-dynfw-client[17840]: 2022-06-09 23:07:24,343 - WARNING - Error running ipset command: return code 1.

JardaB · June 15, 2022, 5:15pm

So again - HaaS is out and the necessary HaaS and Honeypots ports are closed. I postponed the diagnostics. I don’t see anything special in syslog

So again - HaaS is out and the necessary HaaS and Honeypots ports are closed. I postponed the diagnostics. I don’t see anything special in syslog. I won’t interfere, I’ll see if it fixes itself.

JardaB · June 16, 2022, 10:50am

So today’s outage is self-correcting without my intervention - at the time of the outage, the related ports are closed - He does what he wants

Today some records found in syslog … Diagnostics backed up

Jun 16 10:17:01 Turris_JB crond[20248]: (root) CMD (/etc/init.d/haas-proxy restart)
Jun 16 10:17:01 Turris_JB crond[20249]: (root) CMD (/usr/bin/rainbow_button_sync.sh)
Jun 16 10:17:01 Turris_JB crond[20246]: (root) CMDEND (/usr/bin/rainbow_button_sync.sh)
Jun 16 10:17:01 Turris_JB crond[20247]: (root) CMDEND (/etc/init.d/haas-proxy restart)
x
Jun 16 10:42:01 Turris_JB crond[22444]: (root) CMD (sentinel-certgen certs --hooks-dir /usr/libexec/sentinel/renew_hooks.d)
Jun 16 10:42:02 Turris_JB crond[22441]: (root) CMDEND (/usr/bin/rainbow_button_sync.sh)
Jun 16 10:42:03 Turris_JB foris-controller[5918]: WARNING:foris_controller_backends.collectd:Socket error occured '[Errno 2] No such file or directory'
Jun 16 12:42:03 Turris_JB sentinel: INFO [certgen.action_spec_init:89] Valid certificate found
Jun 16 10:42:03 Turris_JB crond[22442]: (root) CMDEND (sentinel-certgen certs --hooks-dir /usr/libexec/sentinel/renew_hooks.d)
    x
Jun 16 12:51:38 Turris_JB sentinel: INFO [certgen.action_spec_init:89] Valid certificate found

JardaB · June 17, 2022, 8:13pm

The ports are closed again

There are interesting shifts in the chronology of records in syslog - it doesn’t matter

Jun 16 22:42:01 Turris_JB crond[23668]: (root) CMD (sentinel-certgen certs --hooks-dir /usr/libexec/sentinel/renew_hooks.d)
Jun 17 00:42:03 Turris_JB sentinel: INFO [certgen.action_spec_init:89] Valid certificate found
Jun 16 22:42:03 Turris_JB crond[23666]: (root) CMDEND (sentinel-certgen certs --hooks-dir /usr/libexec/sentinel/renew_hooks.d)

x

Jun 14 22:42:01 Turris_JB crond[25341]: (root) CMD (sentinel-certgen certs --hooks-dir /usr/libexec/sentinel/renew_hooks.d)
Jun 15 00:42:03 Turris_JB sentinel: INFO [certgen.action_spec_init:89] Valid certificate found
Jun 14 22:42:03 Turris_JB crond[25340]: (root) CMDEND (sentinel-certgen certs --hooks-dir /usr/libexec/sentinel/renew_hooks.d)

JardaB · June 18, 2022, 8:21am

No intervention from me - In this time ports are opened

JardaB · June 21, 2022, 8:01am

HaaS scheduled to restart every 4 hours, in syslog no suspicious activity, I saved Diagnostics

JardaB · June 22, 2022, 6:57am

isolated record

Jun 22 08:22:34 Turris_JB sentinel: INFO [certgen.action_spec_init:89] Valid certificate found
    Jun 22 06:22:36 Turris_JB procd: Instance sentinel-fwlogs::instance1 pid 16511 not stopped on SIGTERM, sending SIGKILL instead
    Jun 22 06:22:37 Turris_JB procd: Instance sentinel-minipot::instance1 pid 16555 not stopped on SIGTERM, sending SIGKILL instead
    Jun 22 06:23:10 Turris_JB sentinel-fwlogs[19460]: ERROR: Packet handling failed: Resource temporarily unavailable

JardaB · June 22, 2022, 12:32pm

spontaneous resumption of operations

Jun 22 10:42:01 Turris_JB crond[10931]: (root) CMD (sentinel-certgen certs --hooks-dir /usr/libexec/sentinel/renew_hooks.d)
Jun 22 10:42:01 Turris_JB crond[10928]: (root) CMDEND (/usr/bin/rainbow_button_sync.sh)
Jun 22 12:42:01 Turris_JB dnsmasq-dhcp[11583]: DHCPREQUEST(br-lan) 192.168.2.118 b0:e8:92:xx:xx:xx 
Jun 22 12:42:01 Turris_JB dnsmasq-dhcp[11583]: DHCPACK(br-lan) 192.168.2.118 b0:e8:92:yy:xxf:xx EpsonXP700
Jun 22 10:42:03 Turris_JB foris-controller[5918]: WARNING:foris_controller_backends.collectd:Socket error occured '[Errno 2] No such file or directory'
Jun 22 12:42:03 Turris_JB sentinel: INFO [certgen.action_spec_init:89] Valid certificate found
Jun 22 10:42:03 Turris_JB crond[10929]: (root) CMDEND (sentinel-certgen certs --hooks-dir /usr/libexec/sentinel/renew_hooks.d)

xx

Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]: 2022-06-22T12:32:41 CRITICAL twisted Unhandled Error
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]: Traceback (most recent call last):
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]:   File "/usr/lib/python3.7/site-packages/twisted/internet/tcp.py", line 243, in doRead
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]:     
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]:   File "/usr/lib/python3.7/site-packages/twisted/internet/tcp.py", line 249, in _dataReceived

Souhrn

Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]:
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]: File “/usr/lib/python3.7/site-packages/twisted/conch/ssh/transport.py”, line 703, in dataReceived
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]:
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]: File “/usr/lib/python3.7/site-packages/twisted/conch/ssh/transport.py”, line 728, in dispatchMessage
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]:
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]: — —
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]: File “/usr/lib/python3.7/site-packages/twisted/python/log.py”, line 103, in callWithLogger
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]:
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]: File “/usr/lib/python3.7/site-packages/twisted/python/log.py”, line 86, in callWithContext
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]:
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]: File “/usr/lib/python3.7/site-packages/twisted/python/context.py”, line 122, in callWithContext
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]:
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]: File “/usr/lib/python3.7/site-packages/twisted/python/context.py”, line 85, in callWithContext
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]:
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]: File “/usr/lib/python3.7/site-packages/twisted/conch/ssh/service.py”, line 45, in packetReceived
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]:
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]: File “/usr/lib/python3.7/site-packages/twisted/conch/ssh/connection.py”, line 295, in ssh_CHANNEL_EOF
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]:
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]: builtins.KeyError: 0
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]:
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]: 2022-06-22T12:32:41 CRITICAL twisted Unhandled Error
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]: Traceback (most recent call last):
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]: File “/usr/lib/python3.7/site-packages/twisted/internet/tcp.py”, line 243, in doRead
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]:
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]: File “/usr/lib/python3.7/site-packages/twisted/internet/tcp.py”, line 249, in _dataReceived
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]:
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]: File “/usr/lib/python3.7/site-packages/twisted/conch/ssh/transport.py”, line 703, in dataReceived
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]:
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]: File “/usr/lib/python3.7/site-packages/twisted/conch/ssh/transport.py”, line 728, in dispatchMessage
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]:
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]: — —
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]: File “/usr/lib/python3.7/site-packages/twisted/python/log.py”, line 103, in callWithLogger
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]:
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]: File “/usr/lib/python3.7/site-packages/twisted/python/log.py”, line 86, in callWithContext
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]:
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]: File “/usr/lib/python3.7/site-packages/twisted/python/context.py”, line 122, in callWithContext
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]:
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]: File “/usr/lib/python3.7/site-packages/twisted/python/context.py”, line 85, in callWithContext
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]:
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]: File “/usr/lib/python3.7/site-packages/twisted/conch/ssh/service.py”, line 45, in packetReceived
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]:
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]: File “/usr/lib/python3.7/site-packages/twisted/conch/ssh/connection.py”, line 308, in ssh_CHANNEL_CLOSE
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]:
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]: builtins.KeyError: 0
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]:

JardaB · June 28, 2022, 3:18pm

2022-06-28 17:17 - Outage again, ports 21-23, 25 closed, HaaS without data.

haas

AreYouLoco · July 3, 2022, 4:24pm

Which branch are you on?

JardaB · July 3, 2022, 5:06pm

Turris OS version 5.3.10 HBS

Installed SW is Statistics, AdBlock and RIPE Atlas and NetMetr

JardaB · July 5, 2022, 5:03pm

Minipots failure recurred yesterday 4.7.22.
Restart sentinel-proxy, haas-proxy, sentinel-minipot with no result. It does what it wants, there are no indications of a problem in the log.
haas
Snímek obrazovky 2022-07-05 183944

Today I thought of looking at the firewall status and in Firewall Status I did not find an active proxy - function indication in reForis, unfortunately it does not check whether the proxy is active, but apparently only the activity of the related application.

It is running, it is possible to restart it, but without affecting the actual activity of the proxy in the firewall.

JardaB · July 5, 2022, 5:14pm