Looping FORIS in AP-Mode on MOX (HBK-branch)

Software SW bugs discussion
1

Hi,
I’ve configured my MOX like a “dumb AP”, all interfaces bridged together in one LAN. Also I’ve disabled common services which are not needed in AP-mode: dnsmasq, firewall, odhcpd, resolver and kresd are disabled.

As soon as I reboot the MOX, lighttpd comes up and the FORIS part is going in a (expensive) restart loop, e.g. ps log for a few seconds:

12621 ?        R      0:02 /usr/bin/python3 /usr/bin/foris -s flup -a config -b mqtt --mqtt-host localhost --mqtt-port 11883 --mqtt-passwd-file /etc/fosquitto/credentials.plain --mqtt-controller-id 0000000D300040D1
[...]
12630 ?        R      0:00 /usr/bin/python3 /usr/bin/foris -s flup -a config -b mqtt --mqtt-host localhost --mqtt-port 11883 --mqtt-passwd-file /etc/fosquitto/credentials.plain --mqtt-controller-id 0000000D300040D1
[...]
12636 ?        R      0:00 /usr/bin/python3 /usr/bin/foris -s flup -a config -b mqtt --mqtt-host localhost --mqtt-port 11883 --mqtt-passwd-file /etc/fosquitto/credentials.plain --mqtt-controller-id 0000000D300040D1
[...]
12636 ?        R      0:01 /usr/bin/python3 /usr/bin/foris -s flup -a config -b mqtt --mqtt-host localhost --mqtt-port 11883 --mqtt-passwd-file /etc/fosquitto/credentials.plain --mqtt-controller-id 0000000D300040D1
[...]
12648 ?        R      0:00 /usr/bin/python3 /usr/bin/foris -s flup -a config -b mqtt --mqtt-host localhost --mqtt-port 11883 --mqtt-passwd-file /etc/fosquitto/credentials.plain --mqtt-controller-id 0000000D300040D1
[...]
12654 ?        R      0:00 /usr/bin/python3 /usr/bin/foris -s flup -a config -b mqtt --mqtt-host localhost --mqtt-port 11883 --mqtt-passwd-file /etc/fosquitto/credentials.plain --mqtt-controller-id 0000000D300040D1

FORIS browser access is not possible (500 server error), while the LuCI part works like a charme …:wink:
As a temporary workaround I’ve disabled the FORIS part completely with a small change in /usr/share/foris/lighttpd-dynamic.conf.

Probably this AP-mode is not a common configuration for turris devices, but FORIS should behave better. Feel free to contact me, if you need more information or testing etc. - thanks!

P.S. TurrisOS 5.0.0 286c407c3d8877e51fa90cbd830f2fedb3ad1f26 / LuCI branch git-20.105.64099-eeba108

2

Hi there,

we are still not sure what is causing this, but at least I was able to reproduce it.

I can see lots of these errors in /var/log/lighttpd/error.log

2020-04-30 20:35:26: (gw_backend.c.324) child exited: 1 unix:/tmp/fastcgi.foris-config.socket-0 
2020-04-30 20:35:26: (gw_backend.c.468) unlink /tmp/fastcgi.foris-config.socket-0 after connect failed: Connection refused

But the real culprit arises only when I start lighttpd on foreground

$ /usr/sbin/lighttpd -D -f /etc/lighttpd/lighttpd.conf
Traceback (most recent call last):
Exception in thread foris-client-reply-listener:
Traceback (most recent call last):
  File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner
  File "/usr/lib/python3.7/site-packages/foris_client/buses/mqtt.py", line 161, in run
  File "/usr/lib/python3.7/site-packages/paho/mqtt/client.py", line 839, in connect
  File "/usr/lib/python3.7/site-packages/paho/mqtt/client.py", line 962, in reconnect
  File "/usr/lib/python3.7/socket.py", line 728, in create_connection
  File "/usr/lib/python3.7/socket.py", line 716, in create_connection
ConnectionRefusedError: [Errno 111] Connection refused

Which points to mqtt not running.

$ /etc/init.d/fosquitto start
Generating fosquitto config file in /var/etc/fosquitto.generated.conf
/etc/rc.common: line 130: can't create /var/etc/fosquitto.generated.conf: nonexistent directory
/etc/rc.common: line 134: can't create /var/etc/fosquitto.generated.conf: nonexistent directory

So /var/etc is missing for some reason.

Recreating it and starting fosquitto, lighttpd and other affected services should fix it. Unfortunately, /var will be cleared after reboot, you’ll ran into this issue again after reboot.

Just to pinpoint the culprit, could you try these steps and confirm or deny that this happens in your case as well?

Dump AP mode is not the preferred way to use turris routers, but there are no intentional constrains to limit this setup, if you desire so.
We have experienced this issue only handful of times, so thanks for confirming that this is real issue.

1 Like
3

Hey, many thanks for looking into this!
I’ve just confirmed your findings - the same happens on my MOX … no “/var/etc” after reboot (unfortunately not on every reboot) … I’ll try to find the root cause in the next couple of days.
As a quick & dirty workaround I’ve added this to /etc/init.d/fosquitto

    generate_config() {
	[ ! -d "/var/etc" ] && mkdir -p "/var/etc"
        [...]

My ISP uplink connections are limited … and occupied by an Omnia and other OpenWrt devices … :wink:

1 Like
4

As it turned out, /var/etc is not created by default and every service that use it have to make sure that it exists.
By disabling services unnecessary for AP mode, you probably also disabled something, that creates /var/etc before starting fosquitto.

Just a minor tweaking… Test before mkdir is not necessary here, because mkdir won’t fail on existing directory.

We already fixed fosquitto init script and updated package should be shipped within next releases.
Thanks for confirming this bug and debugging :slight_smile:

1 Like
5

Many thanks for your support!