Honeypot no longer working

BluishHumility · October 27, 2022, 8:14pm

Hi folks,

I have a Turris Omnia running TurrisOS 6.0.1.

My Honeypot was working well for a while, but about five days now it has been down and I can’t quite figure out how to get it going again.

It had stopped a couple times in the past, but seemed to come back quickly enough after running /etc/init.d/haas-proxy reload. Taking a clue from this post, I set it up as a cron job in Luci → System → Scheduled Tasks, and then sort of forgot about it for a while until Monday when I noticed it was down (it looks like it has been down since last Saturday, the 22nd).

Now, /etc/init.d/haas-proxy reload is no longer bringing it back up. I also tried /etc/init.d/haas-proxy restart (I’m honestly not sure which one is the correct syntax). Other things I have tried:

Restart, stop/start, disable/enable in Luci → System → Startup
Disable/enable in Reforis → Sentinel → HaaS
Restarted firewall
Rebooted router

I did see this bug report related to the firewall logs being down: https://gitlab.nic.cz/turris/sentinel/fwlogs/-/issues/9 I am under the impression the HaaS is a different service though.

Also, the Honeypot itself is down–not just logs. I tested by attempting to SSH to the device remotely and I only get the live system–not the proxy.

Thanks for taking a look, please let me know if there is anything else I should try to get the service back up again.

JardaB · October 28, 2022, 5:25am

honeypot proxy stops working if you change port forward rules in firewall. the solution is to restart the firewall. the problem may also be on the server side

cUBAk · October 28, 2022, 6:52am

Check in Luci / Status / Processes on which port is haas_proxy running

And then set this port in Network / Firewall / Port Forwards for Haas proxy rule

JardaB · October 28, 2022, 7:12am

no such thing is needed, the rules are part of the HaaS installation process.

Skippi · October 28, 2022, 4:30pm

It is really strange.

HAAS service seems to be running - I can see records in https://haas.nic.cz/device/::

Process haas_proxy is running and listening on port 2525:

I tried to check Honeypot:

from Internet (for example https://sshwifty.herokuapp.com/) to standard port (22)
from LAN/WiFi to port 2525

But both attempts failed with the same error: connect to host haas-app.nic.cz port 10011: Connection refused.

Skippi · October 28, 2022, 6:11pm

And it is working again normally.

BluishHumility · October 29, 2022, 12:53am

I have restarted the firewall, but it does not resolve the issue. I have also rebooted the router, and the modem.

My logs stop after 2022-10-22:

It had quite a bit of activity up until that point, then suddenly stopped–I’m not sure what the change was.

I do see some errors in the log, although their significance is lost on me. Anyone have a clue?

Oct 29 00:19:31 T haas-proxy-start[5850]: 2022-10-28T20:19:31 CRITICAL twisted Unhandled Error
Oct 29 00:19:31 T haas-proxy-start[5850]: Traceback (most recent call last):
Oct 29 00:19:31 T haas-proxy-start[5850]:   File "/usr/lib/python3.9/site-packages/twisted/python/log.py", line 96, in callWithLogger
Oct 29 00:19:31 T haas-proxy-start[5850]:     
Oct 29 00:19:31 T haas-proxy-start[5850]:   File "/usr/lib/python3.9/site-packages/twisted/python/log.py", line 80, in callWithContext
Oct 29 00:19:31 T haas-proxy-start[5850]:     
Oct 29 00:19:31 T haas-proxy-start[5850]:   File "/usr/lib/python3.9/site-packages/twisted/python/context.py", line 117, in callWithContext
Oct 29 00:19:31 T haas-proxy-start[5850]:     
Oct 29 00:19:31 T haas-proxy-start[5850]:   File "/usr/lib/python3.9/site-packages/twisted/python/context.py", line 82, in callWithContext
Oct 29 00:19:31 T haas-proxy-start[5850]:     
Oct 29 00:19:31 T haas-proxy-start[5850]: --- <exception caught here> ---
Oct 29 00:19:31 T haas-proxy-start[5850]:   File "/usr/lib/python3.9/site-packages/twisted/internet/posixbase.py", line 683, in _doReadOrWrite
Oct 29 00:19:31 T haas-proxy-start[5850]:     
Oct 29 00:19:31 T haas-proxy-start[5850]:   File "/usr/lib/python3.9/site-packages/twisted/internet/tcp.py", line 248, in doRead
Oct 29 00:19:31 T haas-proxy-start[5850]:     
Oct 29 00:19:31 T haas-proxy-start[5850]:   File "/usr/lib/python3.9/site-packages/twisted/internet/tcp.py", line 253, in _dataReceived
Oct 29 00:19:31 T haas-proxy-start[5850]:     
Oct 29 00:19:31 T haas-proxy-start[5850]:   File "/usr/lib/python3.9/site-packages/twisted/conch/ssh/transport.py", line 763, in dataReceived
Oct 29 00:19:31 T haas-proxy-start[5850]:     
Oct 29 00:19:31 T haas-proxy-start[5850]:   File "/usr/lib/python3.9/site-packages/twisted/conch/ssh/transport.py", line 780, in dispatchMessage
Oct 29 00:19:31 T haas-proxy-start[5850]:     
Oct 29 00:19:31 T haas-proxy-start[5850]:   File "/usr/lib/python3.9/site-packages/twisted/conch/ssh/transport.py", line 1458, in ssh_KEXINIT
Oct 29 00:19:31 T haas-proxy-start[5850]:     
Oct 29 00:19:31 T haas-proxy-start[5850]: builtins.TypeError: 'dict_keys' object is not subscriptable
Oct 29 00:19:31 T haas-proxy-start[5850]: 
Oct 29 00:20:47 T haas-proxy-start[5850]: 2022-10-28T20:20:47 CRITICAL twisted Unhandled Error
Oct 29 00:20:47 T haas-proxy-start[5850]: Traceback (most recent call last):
Oct 29 00:20:47 T haas-proxy-start[5850]:   File "/usr/lib/python3.9/site-packages/twisted/python/log.py", line 96, in callWithLogger
Oct 29 00:20:47 T haas-proxy-start[5850]:     
Oct 29 00:20:47 T haas-proxy-start[5850]:   File "/usr/lib/python3.9/site-packages/twisted/python/log.py", line 80, in callWithContext
Oct 29 00:20:47 T haas-proxy-start[5850]:     
Oct 29 00:20:47 T haas-proxy-start[5850]:   File "/usr/lib/python3.9/site-packages/twisted/python/context.py", line 117, in callWithContext
Oct 29 00:20:47 T haas-proxy-start[5850]:     
Oct 29 00:20:47 T haas-proxy-start[5850]:   File "/usr/lib/python3.9/site-packages/twisted/python/context.py", line 82, in callWithContext
Oct 29 00:20:47 T haas-proxy-start[5850]:     
Oct 29 00:20:47 T haas-proxy-start[5850]: --- <exception caught here> ---
Oct 29 00:20:47 T haas-proxy-start[5850]:   File "/usr/lib/python3.9/site-packages/twisted/internet/posixbase.py", line 683, in _doReadOrWrite
Oct 29 00:20:47 T haas-proxy-start[5850]:     
Oct 29 00:20:47 T haas-proxy-start[5850]:   File "/usr/lib/python3.9/site-packages/twisted/internet/tcp.py", line 248, in doRead
Oct 29 00:20:47 T haas-proxy-start[5850]:     
Oct 29 00:20:47 T haas-proxy-start[5850]:   File "/usr/lib/python3.9/site-packages/twisted/internet/tcp.py", line 253, in _dataReceived
Oct 29 00:20:47 T haas-proxy-start[5850]:     
Oct 29 00:20:47 T haas-proxy-start[5850]:   File "/usr/lib/python3.9/site-packages/twisted/conch/ssh/transport.py", line 763, in dataReceived
Oct 29 00:20:47 T haas-proxy-start[5850]:     
Oct 29 00:20:47 T haas-proxy-start[5850]:   File "/usr/lib/python3.9/site-packages/twisted/conch/ssh/transport.py", line 780, in dispatchMessage
Oct 29 00:20:47 T haas-proxy-start[5850]:     
Oct 29 00:20:47 T haas-proxy-start[5850]:   File "/usr/lib/python3.9/site-packages/twisted/conch/ssh/transport.py", line 1458, in ssh_KEXINIT
Oct 29 00:20:47 T haas-proxy-start[5850]:     
Oct 29 00:20:47 T haas-proxy-start[5850]: builtins.TypeError: 'dict_keys' object is not subscriptable
Oct 29 00:20:47 T haas-proxy-start[5850]: 
Oct 29 00:21:11 T haas-proxy-start[5850]: 2022-10-28T20:21:11 CRITICAL twisted Unhandled Error
Oct 29 00:21:11 T haas-proxy-start[5850]: Traceback (most recent call last):
Oct 29 00:21:11 T haas-proxy-start[5850]:   File "/usr/lib/python3.9/site-packages/twisted/python/log.py", line 96, in callWithLogger
Oct 29 00:21:11 T haas-proxy-start[5850]:     
Oct 29 00:21:11 T haas-proxy-start[5850]:   File "/usr/lib/python3.9/site-packages/twisted/python/log.py", line 80, in callWithContext
Oct 29 00:21:11 T haas-proxy-start[5850]:     
Oct 29 00:21:11 T haas-proxy-start[5850]:   File "/usr/lib/python3.9/site-packages/twisted/python/context.py", line 117, in callWithContext
Oct 29 00:21:11 T haas-proxy-start[5850]:     
Oct 29 00:21:11 T haas-proxy-start[5850]:   File "/usr/lib/python3.9/site-packages/twisted/python/context.py", line 82, in callWithContext
Oct 29 00:21:11 T haas-proxy-start[5850]:     
Oct 29 00:21:11 T haas-proxy-start[5850]: --- <exception caught here> ---
Oct 29 00:21:11 T haas-proxy-start[5850]:   File "/usr/lib/python3.9/site-packages/twisted/internet/posixbase.py", line 683, in _doReadOrWrite
Oct 29 00:21:11 T haas-proxy-start[5850]:     
Oct 29 00:21:11 T haas-proxy-start[5850]:   File "/usr/lib/python3.9/site-packages/twisted/internet/tcp.py", line 248, in doRead
Oct 29 00:21:11 T haas-proxy-start[5850]:     
Oct 29 00:21:11 T haas-proxy-start[5850]:   File "/usr/lib/python3.9/site-packages/twisted/internet/tcp.py", line 253, in _dataReceived
Oct 29 00:21:11 T haas-proxy-start[5850]:     
Oct 29 00:21:11 T haas-proxy-start[5850]:   File "/usr/lib/python3.9/site-packages/twisted/conch/ssh/transport.py", line 763, in dataReceived
Oct 29 00:21:11 T haas-proxy-start[5850]:     
Oct 29 00:21:11 T haas-proxy-start[5850]:   File "/usr/lib/python3.9/site-packages/twisted/conch/ssh/transport.py", line 780, in dispatchMessage
Oct 29 00:21:11 T haas-proxy-start[5850]:     
Oct 29 00:21:11 T haas-proxy-start[5850]:   File "/usr/lib/python3.9/site-packages/twisted/conch/ssh/transport.py", line 1458, in ssh_KEXINIT
Oct 29 00:21:11 T haas-proxy-start[5850]:     
Oct 29 00:21:11 T haas-proxy-start[5850]: builtins.TypeError: 'dict_keys' object is not subscriptable
Oct 29 00:21:11 T haas-proxy-start[5850]: 
Oct 29 00:23:34 T haas-proxy-start[5850]: 2022-10-28T20:23:34 ERROR twisted Got remote error, code 11
Oct 29 00:23:34 T haas-proxy-start[5850]: reason: b''

JardaB · October 29, 2022, 8:09am

Try uninstalling and reinstalling the entire Advanced security & analytics - Turris Sentinel all section in reForis … Package Management - Packages Lists

BluishHumility · November 1, 2022, 7:40pm

I did uninstall and reinstall as suggested, but it did not restore the service.

A somewhat odd development: I have two more hits to the Honeypot log from Sunday, then it goes quiet again:

The Honeypot is currently still not working (I have tested just recently).

The fact that it came randomly back on and then stopped again without an appreciable change in my network or configuration settings makes me consider the issue may be on the other end, as JardaB pointed out:

Anyone else noticing prolonged downtime on their Honeypot?

iron-maiden · November 1, 2022, 10:00pm

It is not related to 6.0 update. I started to have this problem with beginning of this year.

jada4p · November 1, 2022, 10:32pm

I’m afraid that problem is on server side. I saw similar drop-outs of data recently, without any change of config or settings.

Haas log is full of messages like

2022-11-01T20:14:13 CRITICAL twisted ‘channel open failed, direct-tcpip is not allowed’

iron-maiden · November 1, 2022, 10:41pm

It seems to me issue started just after updating server side earlier this year, somehow client side is not compatible. It was working fine last year before server update. Daily was hitting about 50 attempts. Now it logs sometimes, and most often for long time doesnt log anything.