Once every few days I notice pakon-monitor stops running, of course without that process there isn’t any pakon data. Is there any debug info I can provide to help figure out the cause?
no one else seeing this? It happens to me daily.
Related to this issue below… I too have issue with Pakon not being able to load (Failed to load data) and I need to reboot the Omnia in order for it to work again…
Hello,
Unfortunately, this seems not to be related, because it’s two different things and it’s not good to revive old threads.
If you have the issue, which you described I’d like to hear more about it. Are you familiar with SSH? If not, I can describe you, how you can connect to SSH, because I need to know, if Pakon works from CLI or if it only fails to load data at Foris, but it would be cool if you can reach me on email tech.support@turris.cz or if you open your own thread.
We’d need to know, if you have a running process of Pakon, which is responsible for showing data in the command line interface.
So, you need to login to your router via SSH and send me the output from the following command:
pakon-show
run pakon-show and it gave me a very long list of data…
|datetime | dur | src MAC | hostname | dst port | proto | sent | recvd |
| | | | | | | | |
|2018-05-18 18:21:03 | 31s | xx:xx:xx:xx:xx:xx | google.com | https | | 18KiB | 5KiB|
|2018-05-18 18:21:04 | 501s | xx:xx:xx:xx:xx:xx | push.apple.com | http | http | 562B | 9KiB|Ok, I got it, pakon crapout and data only upto 6/10, hence the error…
Now the question is, why did it stop gathering data from the 10th???
Here is the last line:
|2018-06-10 10:52:28 | 59s | xx:xx:xx:xx:xx:xx | i.dailymail.co.uk | http | http | 51KiB | 172KiB|I restarted “/etc/init.d/pakon-monitor restart” and now it is working again… Thx
Pakon stopped gathering data again since:
|2018-06-16 20:46:17 | <1s | xx:xx:xx:xx:xx:xx | facebook.com | https | tls | 2KiB | 310B |
What is causing pakon to stop???
@Blowfish I’m seeing this as well, in my case pakon-monitor runs for less than an hour before crashing. /var/log/messages shows:
2018-06-18 01:56:46 info procd[]: Instance pakon-monitor::instance1 s in a crash loop 6 crashes, 106 seconds since last crash
so it is crashing, no indication why.
@Pepe I will log a tech support request on this issue.
Could you try running pakon-monitor on foreground, over SSH? This might give some better idea what’s wrong…
/etc/init.d/pakon-monitor stop
python3 /usr/libexec/pakon-light/pakon-monitor.py
Unfortunately syslog doesn’t say why it crashed, it only says it crashed.
Thanks in advance.
@mpetracek here’s what I got when I ran pakon-monitor in foreground. This is on Turris 3.10.2. It is the case that /usr/libexec/pakon-light/notify_new_device.sh is not installed, should it be?
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/libexec/pakon-light/pakon-monitor.py", line 161, in new_device_notify_thread
subprocess.call([arg.encode('utf-8') for arg in cmd])
File "/usr/lib/python3.5/subprocess.py", line 557, in call
with Popen(*popenargs, **kwargs) as p:
File "/usr/lib/python3.5/subprocess.py", line 947, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.5/subprocess.py", line 1551, in _execute_child
raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: b'/usr/libexec/pakon-light/notify_new_device.sh'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/usr/lib/python3.5/threading.py", line 862, in run
self._target(*self._args, **self._kwargs)
File "/usr/libexec/pakon-light/pakon-monitor.py", line 163, in new_device_notify_thread
logger.error("failed to create notification")
NameError: name 'logger' is not defined
Exception in thread Thread-2:
Traceback (most recent call last):
File "/usr/libexec/pakon-light/pakon-monitor.py", line 161, in new_device_notify_thread
subprocess.call([arg.encode('utf-8') for arg in cmd])
File "/usr/lib/python3.5/subprocess.py", line 557, in call
with Popen(*popenargs, **kwargs) as p:
File "/usr/lib/python3.5/subprocess.py", line 947, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.5/subprocess.py", line 1551, in _execute_child
raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: b'/usr/libexec/pakon-light/notify_new_device.sh'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/usr/lib/python3.5/threading.py", line 862, in run
self._target(*self._args, **self._kwargs)
File "/usr/libexec/pakon-light/pakon-monitor.py", line 163, in new_device_notify_thread
logger.error("failed to create notification")
NameError: name 'logger' is not defined
Exception in thread Thread-3:
Traceback (most recent call last):
File "/usr/libexec/pakon-light/pakon-monitor.py", line 161, in new_device_notify_thread
subprocess.call([arg.encode('utf-8') for arg in cmd])
File "/usr/lib/python3.5/subprocess.py", line 557, in call
with Popen(*popenargs, **kwargs) as p:
File "/usr/lib/python3.5/subprocess.py", line 947, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.5/subprocess.py", line 1551, in _execute_child
raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: b'/usr/libexec/pakon-light/notify_new_device.sh'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/usr/lib/python3.5/threading.py", line 862, in run
self._target(*self._args, **self._kwargs)
File "/usr/libexec/pakon-light/pakon-monitor.py", line 163, in new_device_notify_thread
logger.error("failed to create notification")
NameError: name 'logger' is not defined
Exception in thread Thread-4:
Traceback (most recent call last):
File "/usr/libexec/pakon-light/pakon-monitor.py", line 161, in new_device_notify_thread
subprocess.call([arg.encode('utf-8') for arg in cmd])
File "/usr/lib/python3.5/subprocess.py", line 557, in call
with Popen(*popenargs, **kwargs) as p:
File "/usr/lib/python3.5/subprocess.py", line 947, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.5/subprocess.py", line 1551, in _execute_child
raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: b'/usr/libexec/pakon-light/notify_new_device.sh'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/usr/lib/python3.5/threading.py", line 862, in run
self._target(*self._args, **self._kwargs)
File "/usr/libexec/pakon-light/pakon-monitor.py", line 163, in new_device_notify_thread
logger.error("failed to create notification")
NameError: name 'logger' is not defined
Exception in thread Thread-5:
Traceback (most recent call last):
File "/usr/libexec/pakon-light/pakon-monitor.py", line 161, in new_device_notify_thread
subprocess.call([arg.encode('utf-8') for arg in cmd])
File "/usr/lib/python3.5/subprocess.py", line 557, in call
with Popen(*popenargs, **kwargs) as p:
File "/usr/lib/python3.5/subprocess.py", line 947, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.5/subprocess.py", line 1551, in _execute_child
raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: b'/usr/libexec/pakon-light/notify_new_device.sh'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/usr/lib/python3.5/threading.py", line 862, in run
self._target(*self._args, **self._kwargs)
File "/usr/libexec/pakon-light/pakon-monitor.py", line 163, in new_device_notify_thread
logger.error("failed to create notification")
NameError: name 'logger' is not defined
WARNING:root:Error decoding json
WARNING:root:Error decoding json
Traceback (most recent call last):
File "/usr/libexec/pakon-light/pakon-monitor.py", line 325, in <module>
main()
File "/usr/libexec/pakon-light/pakon-monitor.py", line 312, in main
handle_flow_start(data, notify_new_devices, c)
File "/usr/libexec/pakon-light/pakon-monitor.py", line 211, in handle_flow_start
dev, mac=get_dev_mac(data['src_ip'])
File "/usr/lib/python3.5/site-packages/cachetools/__init__.py", line 46, in wrapper
v = func(*args, **kwargs)
File "/usr/libexec/pakon-light/pakon-monitor.py", line 143, in get_dev_mac
dev = res.groups()[0]
AttributeError: 'NoneType' object has no attribute 'groups'Ok, that’s certainly a bug. I’ll look more into why notify_new_device.sh is not installed, or when should it be installed. But the more serious problem is that exception did not catch it, because there was a typo. Fixed now.
The last error is more serious and I don’t quite understand why it happened. I added some check to prevent the crash, but I suspect there is a more serious underlying problem.
Could you my fixed version, if it crashes again, or outputs something (it might produce some warning)?
curl https://gitlab.labs.nic.cz/turris/pakon-light/raw/b09b9845b8e73e598b75b62ff7414d19f33ab2b3/monitor.py > /usr/libexec/pakon-light/pakon-monitor.py
python3 /usr/libexec/pakon-light/pakon-monitor.py@mpetracek OK I ran your patched version for about 30 minutes in foreground (before it would crash easily before that time). This time it did not crash, output is similar to this. The “failed to create notification” lines are related to notify_new_device.sh?
I will try to run it all day and see if there are further crashes.
ERROR:root:failed to create notification
WARNING:root:match for dev&mac in output of `ip neigh show 192.168.2.104`: 192.168.2.104 dev br-lan lladdr 34:12:98:79:38:93 DELAY
WARNING:root:match for dev&mac in output of `ip neigh show fe80:0000:0000:0000:14d1:3c8b:f975:6140`: fe80::14d1:3c8b:f975:6140 dev br-lan lladdr 34:12:98:79:38:93 STALE
WARNING:root:match for dev&mac in output of `ip neigh show fe80:0000:0000:0000:1ab4:30ff:fe7b:89d5`: fe80::1ab4:30ff:fe7b:89d5 dev br-lan INCOMPLETE
WARNING:root:no match for dev&mac in output of `ip neigh show fe80:0000:0000:0000:1ab4:30ff:fe7b:89d5`: fe80::1ab4:30ff:fe7b:89d5 dev br-lan INCOMPLETE
WARNING:root:match for dev&mac in output of `ip neigh show 192.168.2.132`: 192.168.2.132 dev br-lan lladdr 18:b4:30:3a:bd:21 REACHABLE
WARNING:root:match for dev&mac in output of `ip neigh show 192.168.2.125`: 192.168.2.125 dev br-lan lladdr 18:b4:30:3a:91:e8 REACHABLE
ERROR:root:failed to create notificationOk, that looks better. The only problem I see so far is that I forgot some debug outputs there 
Yes, the last line is related to notify_new_device.sh. I will have to look why it’s not bundled still. Anyway, not having the file shouldn’t result in crash, that was the thing I tried to fix.
Thanks for your cooperation.
@mpetracek pakon-monitor ran for about 30 hours, then it seems to have stopped/crashed again. This may be more similar to what @blowfish saw as it appeared to run for longer periods for him until crashing. As I was running it in the background I could not catch what caused the failure. I will run it in foreground again for a few days and see if I can determine if any messages of note are logged when it does crash.
Ok, thanks!
You can run it in screen, so you don’t need to be connected to the router the whole time…
Yes, I’m familiar with screen. Not my first time debugging something like this
I noticed at a couple points that pakon-monitor was no longer running as well. But in my case, it was after a reboot. I would have to start it by hand.
I tried the version from @mpetracek earlier in this thread, but I couldn’t get graphs after that.
It seems to run normally as long as I don’t restart. I’ll double check that the init is enabled.
[edit] I checked. It’s enabled.
This is basic, but if you’re trying the patched pakon-monitor.py, make sure it’s set executable, eg something like
chmod +x /usr/libexec/pakon-light/pakon-monitor.py
Otherwise it might not start.
Yeah, thanks.
The version of pakon-monitor not starting was from the package. I didn’t really test the new one for starting on boot since it seemed to not produce graphs for me.