Foris keeps disappearing ... why?

Today is the second time in recent weeks I’ve found foris missing. Here’s why this time at a guess (something I see when reinstalling foris from the Luci software page):

Update from 2017/12/27 21:06:56
• Installed version 2.1 of package dhparam
• Removed package foris-wizard
• Removed package foris-config
• Removed package foris-common
• Removed package python-bottle-i18n
• Removed package python-pbkdf2
• Removed package python-flup
• Removed package python-ncclient
• Removed package lighttpd-mod-redirect
• Removed package foris-ws
• Removed package python-websocket-server
• Removed package lighttpd-https-cert
• Removed package sqm-scripts
• Removed package kmod-sched
• Removed package kmod-ifb
• Removed package iptables-mod-ipopt
• Removed package kmod-ipt-ipopt
• Removed package iptables-mod-conntrack-extra
• Removed package kmod-ipt-conntrack-extra
• Removed package foris-client
• Removed package foris-controller
• Removed package python-ubus
• Removed package python-prctl
• Removed package foris-schema
• Removed package python-jsonschema
• Removed package python-functools32
• Removed package server-uplink
• Removed package l10n_supported
• Removed package python-bottle
• Removed package tc
• Removed package kmod-sched-core 

What gives? Why is foris being uninstalled? Is this what updates are supposed to do? uninstall the web interface?

Actually, I’m bamboozled. It just disappeared again … minutes after posting this, and I installed it again, but no news this time ;-). So tried this:

# grep foris /var/log/messages
2017-12-30T17:21:03+11:00 warning updater[29684]: planner.lua:465 (Globals): Request not satisfied to install package: foris
2017-12-30T17:21:03+11:00 warning updater[29684]: planner.lua:465 (Globals): Request not satisfied to install package: foris-diagnostics-plugin
2017-12-30T17:21:03+11:00 warning updater[29684]: planner.lua:465 (Globals): Request not satisfied to install package: foris-openvpn-plugin
2017-12-30T17:21:03+11:00 warning updater[29684]: planner.lua:465 (Globals): Request not satisfied to install package: foris-tls-plugin
2017-12-30T17:21:03+11:00 warning updater[29684]: planner.lua:465 (Globals): Request not satisfied to install package: foris
2017-12-30T21:41:21+11:00 warning updater[29978]: planner.lua:465 (Globals): Request not satisfied to install package: foris
2017-12-30T21:41:21+11:00 warning updater[29978]: planner.lua:465 (Globals): Request not satisfied to install package: foris-diagnostics-plugin
2017-12-30T21:41:21+11:00 warning updater[29978]: planner.lua:465 (Globals): Request not satisfied to install package: foris-openvpn-plugin
2017-12-30T21:41:21+11:00 warning updater[29978]: planner.lua:465 (Globals): Request not satisfied to install package: foris-tls-plugin
2017-12-30T21:41:21+11:00 warning updater[29978]: planner.lua:465 (Globals): Request not satisfied to install package: foris
2017-12-31T01:35:49+11:00 warning updater[2400]: planner.lua:465 (Globals): Request not satisfied to install package: foris
2017-12-31T01:35:49+11:00 warning updater[2400]: planner.lua:465 (Globals): Request not satisfied to install package: foris-diagnostics-plugin
2017-12-31T01:35:49+11:00 warning updater[2400]: planner.lua:465 (Globals): Request not satisfied to install package: foris-openvpn-plugin
2017-12-31T01:35:49+11:00 warning updater[2400]: planner.lua:465 (Globals): Request not satisfied to install package: foris-tls-plugin
2017-12-31T01:35:49+11:00 warning updater[2400]: planner.lua:465 (Globals): Request not satisfied to install package: foris
2017-12-31T05:05:29+11:00 info procd[]: Instance foris-ws::instance1 s in a crash loop 6 crashes, 1 seconds since last crash
2017-12-31T05:28:39+11:00 warning updater[25984]: planner.lua:465 (Globals): Request not satisfied to install package: foris
2017-12-31T05:28:39+11:00 warning updater[25984]: planner.lua:465 (Globals): Request not satisfied to install package: foris-diagnostics-plugin
2017-12-31T05:28:39+11:00 warning updater[25984]: planner.lua:465 (Globals): Request not satisfied to install package: foris-openvpn-plugin
2017-12-31T05:28:39+11:00 warning updater[25984]: planner.lua:465 (Globals): Request not satisfied to install package: foris-tls-plugin
2017-12-31T05:28:39+11:00 warning updater[25984]: planner.lua:465 (Globals): Request not satisfied to install package: foris-netmetr-plugin
2017-12-31T05:28:39+11:00 warning updater[25984]: planner.lua:465 (Globals): Request not satisfied to install package: foris
2017-12-31T05:28:39+11:00 info updater[25984]: updater.lua:119 (Globals): Queue removal of foris-wizard
2017-12-31T05:28:39+11:00 info updater[25984]: updater.lua:119 (Globals): Queue removal of foris-config
2017-12-31T05:28:39+11:00 info updater[25984]: updater.lua:119 (Globals): Queue removal of foris-common
2017-12-31T05:28:39+11:00 info updater[25984]: updater.lua:119 (Globals): Queue removal of foris-ws
2017-12-31T05:28:39+11:00 info updater[25984]: updater.lua:119 (Globals): Queue removal of foris-client
2017-12-31T05:28:39+11:00 info updater[25984]: updater.lua:119 (Globals): Queue removal of foris-controller
2017-12-31T05:28:39+11:00 info updater[25984]: updater.lua:119 (Globals): Queue removal of foris-schema
2017-12-31T05:31:27+11:00 info procd[]: Instance foris-ws::instance1 s in a crash loop 6 crashes, 1 seconds since last crash
# date
Sun Dec 31 16:35:09 AEDT 2017

Huh? What’s with updater.lua?

Please post updater’s configuration. You clearly have some Unistall command with higher priority in it for package that Foris depends on.

Dang it, it did it yet again since last posting. Grrr. Here’s the configs if I understand you right:

# ls /etc/config/updater*
/etc/config/updater       /etc/config/updater-opkg
# cat /etc/config/updater

config pkglists 'pkglists'
	list lists 'openvpn'
	list lists 'webcam'
	list lists 'tor'
	list lists 'netutils'
	list lists 'api-token'
	list lists 'nas'
	list lists 'lxc'
	list lists 'majordomo'
	list lists 'netmetr'
	list lists 'dev-detect'
	list lists 'luci-controls'
	list lists 'automation'

config override 'override'
	option disable '0'

config l10n 'l10n'

# cat /etc/config/updater-opkg 
config pkglists pkglists
	list lists 'cacerts'
	list lists 'luci-controls'
	list lists 'nas'
	list lists 'netutils'

config l10n 'l10n'
	list langs 'cs'
	list langs 'de'

I can’t see an issue there alas. Can you? Or is there some other config I’m looking for?

I reinstalled it again and then ran it, and this news item is at top:

Update from 2017/12/31 17:59:05
• Removed package foris-wizard
• Removed package foris-config
• Removed package foris-common
• Removed package python-pbkdf2
• Removed package python-flup
• Removed package python-ncclient
• Removed package lighttpd-mod-redirect
• Removed package foris-ws
• Removed package python-websocket-server
• Removed package lighttpd-https-cert
• Removed package sqm-scripts
• Removed package kmod-sched
• Removed package kmod-ifb
• Removed package iptables-mod-ipopt
• Removed package kmod-ipt-ipopt
• Removed package foris-client
• Removed package foris-controller
• Removed package python-ubus
• Removed package python-prctl
• Removed package server-uplink
• Removed package foris-schema
• Removed package python-jsonschema
• Removed package python-functools32
• Removed package l10n_supported
• Removed package tc
• Removed package kmod-sched-core
• Removed package python-bottle-i18n
• Removed package python-bottle 

What on earth is causing updater to keep removing all this? Hmmmm.

No I don’t want that, sorry for not specifying it exactly. I want to see all files from /etc/updater/conf.d (you can ignore base.lua, localrepo.lua, opkg.lua. Anything else I want to see).

Thanks for clarifying. How’s this:

# ll /etc/updater/conf.d
-rw-------    1 root     root          1796 Dec 20 04:36 base.lua
-rw-------    1 root     root          2882 Dec 20 04:36 example.lua
-rw-------    1 root     root           238 Dec 20 04:36 localrepo.lua
-rw-------    1 root     root           438 Dec 31 20:17 opkg-auto.lua
-rw-------    1 root     root           924 Dec 20 04:36 opkg.lua
-rw-------    1 root     root           213 Sep 17 19:48 user.lua
# cat /etc/updater/conf.d/example.lua 
--[[
This is example configuration file. You can use it as quick reference for your own
configurations.

Note that this is Lua 5.1 so you can write any Lua code to these configuration
files and it will be executed as part of updater execution.

Full reference to language it self can be found here:
https://turris.pages.labs.nic.cz/updater/docs/language.html
]]

--[[
To add repository you have to call Repository command.
First argument must be name of repository (duplicate names are not allowed).
Second argument must be URL to repository. Use `file://` for local path.
Third argument is optional table with options for this repository.

Most important options are following:
priority: This allows you to specify priority of this repository. Packages are
  preferably taken from repository with higher priority no matter even if there
  is potentially newer version of package. In default all repositories has
  priority 50 and you can specify any value from 0 to 100.
verification: Specifies verification level. You can specify "none" to have do no
  repository verification or "sig" to check only repository signature or "cert" to
  not check signature but check ssl certificate if https is used. Or use "both" to
  verify both signature  and ssl vertificate.
pubkey: URL to public key used for signature verification.
]]
--Repository("repo-name", "https://example.com/", { pubkey = "file:///etc/repo.pubkey" })

--[[
To specify that you want to have some package installed you have to use function
Install.
It expect packages names as arguments and optional table with additional options.
You can pass multiple arguments at once to this function and additional options
will be applied on all of those.

Most important options are following:
priority: This allows you to specify how much you want this package. It has to be
  number between 0 and 100 and in default if not specified is set to 50. Updater
  can ignore given package if it's not possible to install it because it collides
  with some other package that is required with higher priority or with Uninstall
  command with higher priority.
version: Explicitly states that we want to install given packages with some
  specific version. Value has to be a string starting with at least one of these
  characters "=<>". So for example if you want anything newer than version 2.0
  then you can specify ">=2.0" to this option.
repository: Explicitly states that we want to install given packages from one of
  specified repositories.
]]
--Install("pkg1mame", "pkg2name", { repository = {"repo-name"} })


--[[
To specify that you don't want to have some package installed you have to use
function Uninstall.
It expect packages names as arguments and optional table with additional options.

Most important options are following:
priority: See Install command for more information.
]]
--Uninstall("pkg1name", "pkg2name")
# cat /etc/updater/conf.d/opkg-auto.lua 
-- A file with auto-generated Install commands. Do not edit.
Install("privoxy")
Install("luci-app-privoxy")
Install("lighttpd-mod-proxy")
Install("lighttpd-mod-accesslog")
Install("lighttpd-mod-access")
Install("kmod-sched-cake")
Install("nmap")
Install("less")
Install("vnstat")
Install("vnstati")
Install("luci-app-vnstat")
Install("lighttpd-mod-fastcgi")
Install("python")
Install("pakon")
Install("pakon-dev-detect")
Install("foris")
# cat /etc/updater/conf.d/user.lua 
--[[
A place for user definitions.

Repository "name" "URI" { ca = "file:///etc/ssl/ca.pem", pubkey = "file:///etc/repo.pubkey" }
Install "pkgname" "other"
]]

Uninstall("lighttpd-https-cert", { priority = 60 })

IIRC Foris depends on this package.

foris-ws depends on it, yes. And that package is a dependency of foris-common.
Did you add that line yourself?

Thanks guys, I’ll remove that line. Alas no idea where it came from. Most certainly did not add it myself as I had no idea these config files even existed. My best guess is lighttpd added it! Bizarre, but I can find no other candidate to explain it. As the only changes I have made that I can recall, over recent times was to change lighttpd confs to serve SSL certificates that are provided by letsencrypt, specifically adding this to my lighttpd confs:

$HTTP["host"] == "leaderboard.space" {
      $SERVER["socket"] == ":443" {
          ssl.engine              = "enable"
          ssl.ca-file             = "/etc/letsencrypt/live/leaderboard.space/chain.pem"
          ssl.pemfile             = "/etc/letsencrypt/live/leaderboard.space/combined.pem"
          ssl.honor-cipher-order  = "enable"
          ssl.use-sslv2           = "disable"
          ssl.use-sslv3           = "disable"
      }

      proxy.server  = ( "" => ( ( "host" => "192.168.0.14" ) ) )
}

I don’t eve manage the certs on the router, but on my webserver, and copy them over to the router, and I’m not even serving the site publicly yet with intention either, just experimenting with it (though it does seem to work well enough I’m working on security and other issues with that site.

And that’s the only change that lighttpd-https-cert reminds me of. I certainly don’t recall installing any packages for this, nothing, just changing the lighttpd confs to serve the certs before forwarding requests on my LAN to the webserver, because lighttpd has no support for proxying the SSL cert handshake and delivery.

So I’ve removed that line for user.lua for now and await what happens. Because foris has been being removed at regular intervals it seems in recent days. As an aside I don’t think I made a single change to the route over the days since I noticed foris disappearing. But that’s by the by, I may not have noticed it gone for a while as I don’t use it much anyhow I tend to use luci.

You most certainly did added it by your self. I suspect that you did it because you wanted to change path to certificate and your configuration file was in collision with the one provided by lighttps-https-cert. That is probably why you did it. So most probably your https won’t work any more after lighttpd-https-cert is installed. Unfortunately I don’t have solution for this for you (except providing dummy empty lighttpd-https-cert package). You are not the fist one who did reconfigured its https this way and we might have to think about how to allow such change but for now you have to choose: no foris or no letsencrypt.

Hmmm, I admit I have no recollection of ever adding such a thing and no awareness that the /etc/updater/conf.d folder even exists ;-). So if I did add it myself I would have to assume it was by proxy (by which I mean running something else that add it without my knowing) though I can’t recall or imagine that either as I don’t play around with the router much at all, intentionally trying to keep stuff off of it (like my webserver, NAS etc,).

Oddly right now I can see that lighttpd-https-cert is installed, foris is working fine and the Omnia is serving my letsencrypt certificate for leaderboard.space just fine. Which is the way I like it ;-),

But yes, I would be surprised if I were the first at many things as I’m a rather slow adopter working mainly off the experiences of early adopters in most of what I do I guess. So I put this django site on a Raspberry for now under lighttpd and uwsgi based mainly on experiences others have described and documented, so I’m not at the cutting edge in that space no. The lighttpd community knows the cert issue (it’s not supporting SSL cert proxy and hence the need for the proxy lighttpd server to serve the certs itself on behalf of the downstream site. Not a crisis albeit unclean, I just currently have notes and scripts to publish the certs tot he router when renewed (as they have a lifepspan of 90 days).

Wow, I must have had a wild Christmas break! I stand so corrected and so grateful to you cynerd. I did indeed make that mod and totally forgot. Check this out:

I owe you an apology for my louse memory on the matter (I’ll blame three kids, a day job, a handful of other projects and a holiday period, but blame aside, I goofed memory-wise ;-).

Which does mean sadly right now that it seems it’s foris or SSL … methinks and desires foris needs a fix here.

But check this out:

https://www.ssllabs.com/ssltest/analyze.html?d=leaderboard.space

It seems that the Omnia is deliverying two certificates, my trusted one and an Omnia untrusted one!

I wish I could find a way to configure it so it delivers just the first one!

Hmm interesting. Well at least it some what works but is no way perfect. May I have a question? Do you need access to Foris or Luci from outside? Can’t you just forward ports from wan to your server?

It’s common to choose certificates based on the hostname desired by the client. Quick search indicates that even lighttpd should support that.

I have same issue/messages in log as OP.
… but i know I’ve decided to pick some package which ‘remove’ Foris completelly and gave me full control in LUCI. (i really forgot where and when it was…and if that was part of OS update or community trick&tips in ssh :slight_smile: )

I do not actually need Foris to manage my router, but sometimes it is handy --> mobile app token refresh, to force some packages to be re-installed). So i have to manually install it, use it for my stuff and next updater run removes it again. (to fix Haas vs SSH Honeypot issue it took me two times to install foris and two updater runs :)) …

I am now checking with opkg what depends on foris and if i have it in my user.lua file or in some config file. Will see.

Actually, I was wrong again :-(. Man, I’m stumbling around this. The test I did was from home via a proxy (hide.me) and it loaded without an warning. But as it happens I tried it from the office today and alas it’s not working, because the Omnia is servinge two certs, mine and the Turris one and Turris one is not trusted so the broswer warning pops up. I surmise that hid.me simply suppresses that browser warning (ignores the bad cert and doesn’t show me the browser warning). Grrr. So much to learn.

So back to the drawing board I guess.

I don’t access Foris or Luci form outside no. I open the SSH and VNC ports to one IP which the IP of my office PC, so I can VNC into my dev box from work, and check Foris or Luci from the dev box (on my home LAN).

I’ll have to see how I can work around this. I don’t need Foris much, but I do like to have it available because of the News posts because it’s the official Turris web interface I guess But there may be a way to convince lighttp not to serve the Turris cert. Alas too late tonight to do this, as I gotta hit the sack. But I’ll try and look at it tomorrow again but sifting through the lighttp config file (like the -p option which gets lightttpd to spew out a fully parsed config file). A cursor (pre sleep check) suggest it’s:

/etc/lighttpd-self-signed.pem

that’s being served via config in:

/etc/lighttpd/conf.d/ssl-enable.conf

And tomorrow I’ll see if suppressing this affects foris in any way, and if so whether I can serve this only for foris and not universally on port 443 as is currently configged (conceivable as I know the TLD I access forris through (cerberus.lan on my LAN) and could conditionally serve it conditionally on that host name say and my other cert on leaderboad.space.

That’s feeling warm, but tests will await tomorrow. I’m a tad bamboozled why foris would rely on an untrusted cert anyhow. But I clearly have learning to do and memory muscles to exercise (given I couldn’t remember making a config file change that I did in fact make - the reason good records are kick ass awesome and one hand side effect of doing things openly).

What puzzling me also mildly is that this isn’t a common issue (i.e. that there aren’t already a good number of folk serving https sites form behind an Omnia gateway and run into these issues. But perhaps they are OK with not having foris? Or perhaps I am sadly at the cutting edge … (not where I like to be or like to think I am).

Because in common it is not on any domain so we can’t issue any valid certificate for it. That is why it have to only use self signed certificate.

Well what I mean is that you are probably doing it little bit too complicated. If you don’t need access from outside to any website running on router then why bother with it. Why you just don’t nat port 80 and 443 to your server serving that website (this way you don’t have to copy certificates and so on). And if you are concerned with access to your website from internal network then I would suggest you to use split dns (add static hint to kresd to resolve your domain to internal ip address). Just cut out the middle man if you don’t need it.

One reason I went this way I guess is that I don’t serve from just one machine, but I have a webserver and a NAS that are exposed to the outside world and sort of envisaged maybe having more, as I roll various projects along. Not that that’s a crisis, I can have a webserver do all the forwarding and indeed NAT the ports to it.

I’d have to experiment if I went that way as I’m an active learner in many ways but surmise from your suggestion that if I NAT forward those ports that only affects requests from the WAN, and requests on the Omnia from the LAN would not be forwarded.

But as mentioned made time for a test. All I did was wrap the whole of:

/etc/lighttpd/conf.d/ssl-enable.conf

in

$HTTP["host"] != "leaderboard.space" {

}

and voila,

https://www.ssllabs.com/ssltest/analyze.html?d=leaderboard.space

confirms only one cert is now served, and I VPNed and remoted into my office PC now and loaded the page from there (kind of a round trip really :wink: and true enough the page no loads with no trust issues. So therein lies a solution and the learning of the day which I should document on the sister thread too is that:

Foris needs a generic SSL cert to work. If you want the Omnia to serve certs for other domains it serves and not run into issues, you need to wrap /etc/lighttpd/conf.d/ssl-enable.conf in

 $HTTP["host"] != "mydomain.tld" {

}

to prevent this generic untrusted cert from being served as a second one.

Yes that is valid argument and it makes sense to do it this way.

I am afraid that this won’t survive lighttpd-https-cert package update. You can check using pkg_check but I am pretty much sure about that.

Thanks for the heads up! Does seem a tad complicate that be sure.

Pkg_check returns odd and concerning results (a load of failures) but does confirm that file I tampered with is changed:

# pkg_check
 * Changes found in package dnssec-rootkey:
   - /etc/root.keys: FAILED

 * Changes found in package knot-resolver:
   - /etc/kresd/kresd.postinst.sh: FAILED
   - /etc/kresd/convert_config.sh: FAILED

 * Changes found in package lighttpd-https-cert:
   - /etc/lighttpd/conf.d/ssl-enable.conf: FAILED

 * Changes found in package lighttpd-mod-access:
   - /etc/lighttpd/modules.d/30-access.load: FAILED

 * Changes found in package lighttpd-mod-accesslog:
   - /etc/lighttpd/modules.d/30-accesslog.load: FAILED

 * Changes found in package lighttpd-mod-alias:
   - /etc/lighttpd/modules.d/30-alias.load: FAILED

 * Changes found in package lighttpd-mod-cgi:
   - /etc/lighttpd/modules.d/30-cgi.load: FAILED

 * Changes found in package lighttpd-mod-fastcgi:
   - /etc/lighttpd/modules.d/30-fastcgi.load: FAILED

 * Changes found in package lighttpd-mod-proxy:
   - /etc/lighttpd/modules.d/30-proxy.load: FAILED

 * Changes found in package lighttpd-mod-setenv:
   - /etc/lighttpd/modules.d/30-setenv.load: FAILED

 * Changes found in package luci-base:
   - /etc/config/ucitrack: FAILED
   - /etc/config/luci: FAILED

 * Changes found in package netmetr:
   - /etc/config/netmetr: FAILED

Some packages contain changed files!
Maybe something worth looking into?
Here is the list of packages and changed files:

 - dnssec-rootkey: /etc/root.keys
 - knot-resolver: /etc/kresd/kresd.postinst.sh
 - knot-resolver: /etc/kresd/convert_config.sh
 - lighttpd-https-cert: /etc/lighttpd/conf.d/ssl-enable.conf
 - lighttpd-mod-access: /etc/lighttpd/modules.d/30-access.load
 - lighttpd-mod-accesslog: /etc/lighttpd/modules.d/30-accesslog.load
 - lighttpd-mod-alias: /etc/lighttpd/modules.d/30-alias.load
 - lighttpd-mod-cgi: /etc/lighttpd/modules.d/30-cgi.load
 - lighttpd-mod-fastcgi: /etc/lighttpd/modules.d/30-fastcgi.load
 - lighttpd-mod-proxy: /etc/lighttpd/modules.d/30-proxy.load
 - lighttpd-mod-setenv: /etc/lighttpd/modules.d/30-setenv.load
 - luci-base: /etc/config/ucitrack
 - luci-base: /etc/config/luci
 - netmetr: /etc/config/netmetr

Wonder if I can find or build a hook that survives updates and reapplies the patch? On thought is to reconsider the line:

include "/etc/lighttpd/conf.d/*.conf"

in:

/etc/lighttpd/lighttpd.conf

and find a way to wrap that particular file at this level which survives updates. Just musing before breakfast.