Today's new problem - lighthttpd not responding

So this morning I tried to log into foris/luci on my Omnia. I was greeted with a dropped connection every time.

On investigation the lighthttpd logs contained

2016-12-23 11:55:52: (connections-glue.c.289) connection closed - read failed: Function not implemented 38

over and over and over, once for every connection attempt.

Restarting lighthttpd with

/etc/init.d/lighttpd restart

… solved the problem.

My Omnia is completely vanilla in terms of software.

Model Turris Omnia
Firmware Version OpenWrt omnia 15.05 r47055 / LuCI 5ca9e5d2391f6ca149db4e53cda7c8f5d3ef6644 branch (git-16.335.29518-5ca9e5d)
Kernel Version 4.4.38-efe609c5e5f25db4116d69128330872c-1

1 Like

Yes this have been my problem for a while as well.

4.4.35 lighthttpd works and is stable.
4.4.38 lighthttpd crashes often. Especially if you edit the LED settings and then save.

Yes, problem also on my end, a restart of lighthttpd solves it. For me this has happened even before the latest update.

Have same issue on nightly branch. Did someone try to send email to tech(dot)support(at)turris(dot)cz?

I have the same issue, and as well often when configuring the leds, restart of lighttpd works every time.
And as I remember I had this issue on previous version too.

I’m having this issue after adding some custom firewall rules. Now if I try to add a new rule under traffic, it ends up crashing, but not all the time.

Does anything stand out in /var/log/lighttpd/error.log ? The log gets cleared upon router reboot, but not if you just restart lighttpd.

This issue should be fixed in TurrisOS 3.5

1 Like

No, it is happening also in 3.5

Use https://

not

http://

2 Likes

I played with the LED config and firewall rules and did not see this issue, but I was using https:// (also recommended by @hdBerretz above)

On Linux, 38 is for ENOSYS. That error is coming from the kernel after a call to read(). A configuration change by a process outside of lighttpd should not mess with a socket inside lighttpd after accept(), unless networking was being torn down and restarted on the system, or perhaps firewall connection tracking information flushed. Also, I have no idea why https would make a difference. For those seeing this, does using https make a difference? If not, would you mind posting more detailed instructions on what you did (whether in Foris, or LuCI, or command line) to trigger the issue?

Same here when editing the firewall rules

Same crash here, Version: 1.4.45-1

Same Turris OS 3.6

no further info, just

2017-03-10 20:20:20: (connections-glue.c.291) connection closed - read failed:  Function not implemented 38
2017-03-10 20:20:20: (connections-glue.c.291) connection closed - read failed:  Function not implemented 38
2017-03-10 20:20:20: (connections-glue.c.291) connection closed - read failed:  Function not implemented 38
2017-03-10 20:20:20: (connections-glue.c.291) connection closed - read failed:  Function not implemented 38
2017-03-10 20:20:20: (connections-glue.c.291) connection closed - read failed:  Function not implemented 38
2017-03-10 20:20:20: (connections-glue.c.291) connection closed - read failed:  Function not implemented 38
2017-03-10 20:20:20: (connections-glue.c.291) connection closed - read failed:  Function not implemented 38
2017-03-10 20:20:20: (connections-glue.c.291) connection closed - read failed:  Function not implemented 38
2017-03-10 20:20:20: (connections-glue.c.291) connection closed - read failed:  Function not implemented 38
2017-03-10 20:20:20: (connections-glue.c.291) connection closed - read failed:  Function not implemented 38
2017-03-10 20:20:23: (connections-glue.c.291) connection closed - read failed:  Function not implemented 38
2017-03-10 20:20:53: (connections-glue.c.291) connection closed - read failed:  Function not implemented 38
2017-03-10 20:20:54: (connections-glue.c.291) connection closed - read failed:  Function not implemented 38
2017-03-10 20:20:56: (connections-glue.c.291) connection closed - read failed:  Function not implemented 38
2017-03-10 20:20:56: (connections-glue.c.291) connection closed - read failed:  Function not implemented 38
2017-03-10 20:20:56: (connections-glue.c.291) connection closed - read failed:  Function not implemented 38
2017-03-10 20:20:56: (connections-glue.c.291) connection closed - read failed:  Function not implemented 38
2017-03-10 20:20:58: (connections-glue.c.291) connection closed - read failed:  Function not implemented 38
2017-03-10 20:20:59: (connections-glue.c.291) connection closed - read failed:  Function not implemented 38
2017-03-10 20:21:05: (server.c.1828) server stopped by UID = 0 PID = 28707
2017-03-10 20:21:06: (log.c.217) server started
2017-03-10 20:21:06: (server.c.1295) WARNING: unknown config-key: setenv.set-environment (ignored)
2017-03-10 20:21:40: (connections-glue.c.291) connection closed - read failed:  Function not implemented 38
2017-03-10 20:21:43: (connections-glue.c.291) connection closed - read failed:  Function not implemented 38
2017-03-10 20:22:03: (connections-glue.c.291) connection closed - read failed:  Function not implemented 38
2017-03-10 20:22:10: (server.c.1828) server stopped by UID = 0 PID = 29736
2017-03-10 20:22:11: (log.c.217) server started
2017-03-10 20:22:11: (server.c.1295) WARNING: unknown config-key: setenv.set-environment (ignored)

I am thinking if this because new set env module, or lighttpd as itself.

Edit: And I was right :-]

I have same issue

2017-03-16 18:37:02: (server.c.1828) server stopped by UID = 0 PID = 10580
2017-03-16 18:37:03: (log.c.217) server started
2017-03-16 18:37:03: (server.c.1295) WARNING: unknown config-key: setenv.set-environment (ignored)
2017-03-16 18:44:31: (connections-glue.c.291) connection closed - read failed: Function not implemented 38
2017-03-16 18:44:34: (connections-glue.c.291) connection closed - read failed: Function not implemented 38
2017-03-16 18:44:35: (connections-glue.c.291) connection closed - read failed: Function not implemented 38
2017-03-16 18:44:36: (connections-glue.c.291) connection closed - read failed: Function not implemented 38
2017-03-16 18:44:37: (connections-glue.c.291) connection closed - read failed: Function not implemented 38
2017-03-16 18:44:37: (connections-glue.c.291) connection closed - read failed: Function not implemented 38
2017-03-16 18:44:37: (connections-glue.c.291) connection closed - read failed: Function not implemented 38
2017-03-16 18:44:37: (connections-glue.c.291) connection closed - read failed: Function not implemented 38
2017-03-16 18:44:37: (connections-glue.c.291) connection closed - read failed: Function not implemented 38
2017-03-16 18:44:37: (connections-glue.c.291) connection closed - read failed: Function not implemented 38
2017-03-16 18:44:37: (connections-glue.c.291) connection closed - read failed: Function not implemented 38
2017-03-16 18:44:37: (connections-glue.c.291) connection closed - read failed: Function not implemented 38
2017-03-16 18:44:37: (connections-glue.c.291) connection closed - read failed: Function not implemented 38
2017-03-16 18:44:37: (connections-glue.c.291) connection closed - read failed: Function not implemented 38
2017-03-16 18:44:37: (connections-glue.c.291) connection closed - read failed: Function not implemented 38
2017-03-16 18:44:37: (connections-glue.c.291) connection closed - read failed: Function not implemented 38
2017-03-16 18:44:37: (connections-glue.c.291) connection closed - read failed: Function not implemented 38
2017-03-16 18:44:37: (connections-glue.c.291) connection closed - read failed: Function not implemented 38
2017-03-16 18:44:38: (connections-glue.c.291) connection closed - read failed: Function not implemented 38
2017-03-16 18:44:38: (connections-glue.c.291) connection closed - read failed: Function not implemented 38
2017-03-16 18:44:38: (connections-glue.c.291) connection closed - read failed: Function not implemented 38
2017-03-16 18:44:38: (connections-glue.c.291) connection closed - read failed: Function not implemented 38
2017-03-16 18:44:43: (connections-glue.c.291) connection closed - read failed: Function not implemented 38
2017-03-16 18:44:49: (server.c.1828) server stopped by UID = 0 PID = 15517
2017-03-16 18:44:49: (log.c.217) server started
2017-03-16 18:44:49: (server.c.1295) WARNING: unknown config-key: setenv.set-environment (ignored)
2017-03-16 19:30:48: (server.c.1828) server stopped by UID = 0 PID = 2339
2017-03-16 19:30:49: (log.c.217) server started
2017-03-16 19:30:49: (server.c.1295) WARNING: unknown config-key: setenv.set-environment (ignored)
2017-03-16 19:33:55: (server.c.1828) server stopped by UID = 0 PID = 7885
2017-03-16 19:33:56: (log.c.217) server started
2017-03-16 19:33:56: (server.c.1295) WARNING: unknown config-key: setenv.set-environment (ignored)
2017-03-16 19:38:14: (server.c.1828) server stopped by UID = 0 PID = 14989
2017-03-16 19:38:15: (log.c.217) server started
2017-03-16 19:38:15: (server.c.1295) WARNING: unknown config-key: setenv.set-environment (ignored)
2017-03-16 19:40:48: (server.c.1828) server stopped by UID = 0 PID = 17592
2017-03-16 19:40:49: (log.c.217) server started
2017-03-16 19:40:49: (server.c.1295) WARNING: unknown config-key: setenv.set-environment (ignored)
2017-03-16 19:41:05: (server.c.1828) server stopped by UID = 0 PID = 17774
2017-03-16 19:41:06: (log.c.217) server started
2017-03-16 19:41:06: (server.c.1295) WARNING: unknown config-key: setenv.set-environment (ignored)

Same issue in Turris OS 3.6.1 (lighttpd 1.4.45-1) still…

2017-03-18 07:31:33: (connections-glue.c.291) connection closed - read failed: Function not implemented 38
2017-03-18 07:31:33: (connections-glue.c.291) connection closed - read failed: Function not implemented 38
2017-03-18 07:33:12: (connections-glue.c.291) connection closed - read failed: Function not implemented 38
2017-03-18 07:33:13: (connections-glue.c.291) connection closed - read failed: Function not implemented 38
2017-03-18 07:36:43: (server.c.1828) server stopped by UID = 0 PID = 16492
2017-03-18 07:36:44: (log.c.217) server started
2017-03-18 07:36:44: (server.c.1295) WARNING: unknown config-key: setenv.set-environment (ignored)

Today I experienced this problem for the first time.

2017-03-18 11:25:16: (connections-glue.c.291) connection closed - read failed: Function not implemented 38

I was in LuCI and lighttpd stopped working when I switched to another menu item.
Omnia 3.6.1; lighttpd - 1.4.45-1

Thank you @pav for trying to provide additional information.

I (lighttpd developer) would love to track this down, but have been unable to reproduce this on my Turris Omnia. For everyone saying “me, too”, please post more details of precisely what you were doing when this occurs. Again, if I can’t reproduce this, I can’t help provide a solution. ENOSYS “Function not implemented” is something which should not happen on a read() of an already opened file descriptor, so something is probably happening in firewall or kernel.

There are known bugs in opkg for which I provided patches. The patches have been accepted into LEDE, and I mentioned them to turris support, but I don’t believe they have been integrated into Turris yet. https://github.com/openwrt/packages/issues/1922

For those having this problem, please edit /etc/lighttpd/conf.d/luci.conf and change “setenv.set-environment” to be “setenv.add-environment”. Then, restart lighttpd and see if you can reproduce the “connection closed - read failed” issue.

For those getting this error, how are you connecting to the router? Wired or wireless connection? If everyone having this issue is running LuCI via a wireless connection to the router, that would be useful information. When you get this error message from lighttpd error.log, is there any information in ‘dmesg’ about the same time, perhaps related to network interfaces going down and up?