HaaS and minipots outages

HaaS is not running again and HaaS ports and minipots are closed
Service restarts did not help, this is a recurring and long-standing issue:

root @ Turris_JB: ~ # /etc/init.d/adblock reload
root @ Turris_JB: ~ # /etc/init.d/sentinel-proxy reload
root @ Turris_JB: ~ # /etc/init.d/sentinel-dynfw-client reload
root @ Turris_JB: ~ # /etc/init.d/sentinel-minipot reload
root @ Turris_JB: ~ # /etc/init.d/haas-proxy reload
root @ Turris_JB: ~ #

Troubleshooting or other measures (what to restart, etc) have not been mentioned anywhere

Only reinstalling the minipots in reForis will help, then the monitored ports will open.

  • Removed package sentinel-minipot
  • Removed package logc-libevent
  • Removed package base64c

  • Installed version 0.2.1-1 of package base64c
  • Installed version 0.1.0-1 of package logc-libevent
  • Installed version 2.3.0-1 of package sentinel-minipot

Can you please submit your troubles in our Issue Tracker? Issues - Turris Documentation

I tried to enter something, maybe it’s telling, I’m not an expert on some debugging problem. Gitlab is an unknown territory with me labeled “there are lions”.

Thank you very much, somebody will ask you to provide more info if needed. Cheers!

Today the problem was repeated, today it was enough to turn “Enable Minipots” on and off in reForis. There are some mentions of sentinel traffic in the syslog – but I not understand that.

Pause HaaS viz

2022-06-09 14:04:50 CHN - 36.110.228.254
2022-06-08 20:14:40 CHN - 218.92.0.158

I postponed the diagnostics

Jun 9 11:50:59 Turris_JB sentinel: INFO [certgen.action_spec_init:89] Valid certificate found
Jun 9 09:50:59 Turris_JB /dhcp_host_domain_ng.py: Kresd is probably not running no socket found.

   Jun  9 11:50:59 Turris_JB dnsmasq[11583]: script process exited with status 1
    Jun  9 11:50:59 Turris_JB dnsmasq-dhcp[11583]: read /etc/ethers - 0 addresses
    Jun  9 09:51:00 Turris_JB /dhcp_host_domain_ng.py: DHCP update hostname 
Souhrn

[Fujitsu,192.168.2.120]
Jun 9 09:51:00 Turris_JB /dhcp_host_domain_ng.py: Refresh kresd leases
Jun 9 09:51:00 Turris_JB procd: Instance sentinel-fwlogs::instance1 pid 17075 not stopped on SIGTERM, sending SIGKILL instead
Jun 9 09:51:00 Turris_JB /dhcp_host_domain_ng.py: DHCP update hostname [EpsonXP700,192.168.2.118]
Jun 9 09:51:00 Turris_JB /dhcp_host_domain_ng.py: Refresh kresd leases
Jun 9 09:51:00 Turris_JB foris-controller[5918]: WARNING:foris_controller_backends.collectd:Socket error occured ‘[Errno 2] No such file or directory’
Jun 9 09:51:00 Turris_JB /dhcp_host_domain_ng.py: DHCP update hostname [Pixel-4a,192.168.2.115]
Jun 9 09:51:00 Turris_JB /dhcp_host_domain_ng.py: Refresh kresd leases
Jun 9 09:51:00 Turris_JB procd: Instance sentinel-minipot::instance1 pid 18925 not stopped on SIGTERM, sending SIGKILL instead
Jun 9 09:51:01 Turris_JB /dhcp_host_domain_ng.py: DHCP update hostname [Fujitsu,192.168.2.120]
Jun 9 09:51:01 Turris_JB /dhcp_host_domain_ng.py: Refresh kresd leases
Jun 9 09:51:01 Turris_JB /dhcp_host_domain_ng.py: DHCP update hostname [EpsonXP700,192.168.2.118]
Jun 9 09:51:01 Turris_JB sentinel-fwlogs[11966]: ERROR: Packet handling failed: Resource temporarily unavailable
Jun 9 09:51:01 Turris_JB /dhcp_host_domain_ng.py: Refresh kresd leases

Jun  9 11:51:20 Turris_JB sentinel: INFO [certgen.action_spec_init:89] Valid certificate found
Jun  9 09:51:21 Turris_JB foris-controller[5918]: 
Souhrn
WARNING:foris_controller_backends.collectd:Socket error occured '[Errno 2] No such file or directory'
Jun  9 09:51:22 Turris_JB procd: Instance sentinel-fwlogs::instance1 pid 11966 not stopped on SIGTERM, sending SIGKILL instead
Jun  9 09:51:23 Turris_JB foris-controller[5918]: WARNING:foris_controller_backends.collectd:Socket error occured '[Errno 2] No such file or directory'
x
x
Jun  9 09:51:56 Turris_JB sentinel-fwlogs[12571]: ERROR: Packet handling failed: Resource temporarily unavailable

x

Jun  9 21:07:24 Turris_JB sentinel-dynfw-client[17840]: ipset v7.3: Error in line 1: Element cannot be deleted from the set: it's not added
Jun  9 21:07:24 Turris_JB sentinel-dynfw-client[17840]: 2022-06-09 23:07:24,343 - WARNING - Error running ipset command: return code 1.

So again - HaaS is out and the necessary HaaS and Honeypots ports are closed. I postponed the diagnostics. I don’t see anything special in syslog

So again - HaaS is out and the necessary HaaS and Honeypots ports are closed. I postponed the diagnostics. I don’t see anything special in syslog. I won’t interfere, I’ll see if it fixes itself.

image

So today’s outage is self-correcting without my intervention - at the time of the outage, the related ports are closed - He does what he wants

image

Today some records found in syslog … Diagnostics backed up

Jun 16 10:17:01 Turris_JB crond[20248]: (root) CMD (/etc/init.d/haas-proxy restart)
Jun 16 10:17:01 Turris_JB crond[20249]: (root) CMD (/usr/bin/rainbow_button_sync.sh)
Jun 16 10:17:01 Turris_JB crond[20246]: (root) CMDEND (/usr/bin/rainbow_button_sync.sh)
Jun 16 10:17:01 Turris_JB crond[20247]: (root) CMDEND (/etc/init.d/haas-proxy restart)
x
Jun 16 10:42:01 Turris_JB crond[22444]: (root) CMD (sentinel-certgen certs --hooks-dir /usr/libexec/sentinel/renew_hooks.d)
Jun 16 10:42:02 Turris_JB crond[22441]: (root) CMDEND (/usr/bin/rainbow_button_sync.sh)
Jun 16 10:42:03 Turris_JB foris-controller[5918]: WARNING:foris_controller_backends.collectd:Socket error occured '[Errno 2] No such file or directory'
Jun 16 12:42:03 Turris_JB sentinel: INFO [certgen.action_spec_init:89] Valid certificate found
Jun 16 10:42:03 Turris_JB crond[22442]: (root) CMDEND (sentinel-certgen certs --hooks-dir /usr/libexec/sentinel/renew_hooks.d)
    x
Jun 16 12:51:38 Turris_JB sentinel: INFO [certgen.action_spec_init:89] Valid certificate found

The ports are closed again

There are interesting shifts in the chronology of records in syslog - it doesn’t matter

Jun 16 22:42:01 Turris_JB crond[23668]: (root) CMD (sentinel-certgen certs --hooks-dir /usr/libexec/sentinel/renew_hooks.d)
Jun 17 00:42:03 Turris_JB sentinel: INFO [certgen.action_spec_init:89] Valid certificate found
Jun 16 22:42:03 Turris_JB crond[23666]: (root) CMDEND (sentinel-certgen certs --hooks-dir /usr/libexec/sentinel/renew_hooks.d)

x

Jun 14 22:42:01 Turris_JB crond[25341]: (root) CMD (sentinel-certgen certs --hooks-dir /usr/libexec/sentinel/renew_hooks.d)
Jun 15 00:42:03 Turris_JB sentinel: INFO [certgen.action_spec_init:89] Valid certificate found
Jun 14 22:42:03 Turris_JB crond[25340]: (root) CMDEND (sentinel-certgen certs --hooks-dir /usr/libexec/sentinel/renew_hooks.d)

No intervention from me - In this time ports are opened

HaaS scheduled to restart every 4 hours, in syslog no suspicious activity, I saved Diagnostics

image


isolated record

Jun 22 08:22:34 Turris_JB sentinel: INFO [certgen.action_spec_init:89] Valid certificate found
    Jun 22 06:22:36 Turris_JB procd: Instance sentinel-fwlogs::instance1 pid 16511 not stopped on SIGTERM, sending SIGKILL instead
    Jun 22 06:22:37 Turris_JB procd: Instance sentinel-minipot::instance1 pid 16555 not stopped on SIGTERM, sending SIGKILL instead
    Jun 22 06:23:10 Turris_JB sentinel-fwlogs[19460]: ERROR: Packet handling failed: Resource temporarily unavailable

spontaneous resumption of operations

Jun 22 10:42:01 Turris_JB crond[10931]: (root) CMD (sentinel-certgen certs --hooks-dir /usr/libexec/sentinel/renew_hooks.d)
Jun 22 10:42:01 Turris_JB crond[10928]: (root) CMDEND (/usr/bin/rainbow_button_sync.sh)
Jun 22 12:42:01 Turris_JB dnsmasq-dhcp[11583]: DHCPREQUEST(br-lan) 192.168.2.118 b0:e8:92:xx:xx:xx 
Jun 22 12:42:01 Turris_JB dnsmasq-dhcp[11583]: DHCPACK(br-lan) 192.168.2.118 b0:e8:92:yy:xxf:xx EpsonXP700
Jun 22 10:42:03 Turris_JB foris-controller[5918]: WARNING:foris_controller_backends.collectd:Socket error occured '[Errno 2] No such file or directory'
Jun 22 12:42:03 Turris_JB sentinel: INFO [certgen.action_spec_init:89] Valid certificate found
Jun 22 10:42:03 Turris_JB crond[10929]: (root) CMDEND (sentinel-certgen certs --hooks-dir /usr/libexec/sentinel/renew_hooks.d)

xx

Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]: 2022-06-22T12:32:41 CRITICAL twisted Unhandled Error
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]: Traceback (most recent call last):
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]:   File "/usr/lib/python3.7/site-packages/twisted/internet/tcp.py", line 243, in doRead
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]:     
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]:   File "/usr/lib/python3.7/site-packages/twisted/internet/tcp.py", line 249, in _dataReceived
Souhrn

Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]:
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]: File “/usr/lib/python3.7/site-packages/twisted/conch/ssh/transport.py”, line 703, in dataReceived
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]:
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]: File “/usr/lib/python3.7/site-packages/twisted/conch/ssh/transport.py”, line 728, in dispatchMessage
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]:
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]: — —
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]: File “/usr/lib/python3.7/site-packages/twisted/python/log.py”, line 103, in callWithLogger
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]:
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]: File “/usr/lib/python3.7/site-packages/twisted/python/log.py”, line 86, in callWithContext
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]:
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]: File “/usr/lib/python3.7/site-packages/twisted/python/context.py”, line 122, in callWithContext
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]:
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]: File “/usr/lib/python3.7/site-packages/twisted/python/context.py”, line 85, in callWithContext
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]:
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]: File “/usr/lib/python3.7/site-packages/twisted/conch/ssh/service.py”, line 45, in packetReceived
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]:
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]: File “/usr/lib/python3.7/site-packages/twisted/conch/ssh/connection.py”, line 295, in ssh_CHANNEL_EOF
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]:
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]: builtins.KeyError: 0
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]:
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]: 2022-06-22T12:32:41 CRITICAL twisted Unhandled Error
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]: Traceback (most recent call last):
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]: File “/usr/lib/python3.7/site-packages/twisted/internet/tcp.py”, line 243, in doRead
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]:
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]: File “/usr/lib/python3.7/site-packages/twisted/internet/tcp.py”, line 249, in _dataReceived
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]:
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]: File “/usr/lib/python3.7/site-packages/twisted/conch/ssh/transport.py”, line 703, in dataReceived
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]:
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]: File “/usr/lib/python3.7/site-packages/twisted/conch/ssh/transport.py”, line 728, in dispatchMessage
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]:
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]: — —
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]: File “/usr/lib/python3.7/site-packages/twisted/python/log.py”, line 103, in callWithLogger
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]:
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]: File “/usr/lib/python3.7/site-packages/twisted/python/log.py”, line 86, in callWithContext
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]:
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]: File “/usr/lib/python3.7/site-packages/twisted/python/context.py”, line 122, in callWithContext
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]:
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]: File “/usr/lib/python3.7/site-packages/twisted/python/context.py”, line 85, in callWithContext
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]:
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]: File “/usr/lib/python3.7/site-packages/twisted/conch/ssh/service.py”, line 45, in packetReceived
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]:
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]: File “/usr/lib/python3.7/site-packages/twisted/conch/ssh/connection.py”, line 308, in ssh_CHANNEL_CLOSE
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]:
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]: builtins.KeyError: 0
Jun 22 10:32:41 Turris_JB haas-proxy-start[8715]:

1 Like

2022-06-28 17:17 - Outage again, ports 21-23, 25 closed, HaaS without data.

haas