Let's Encrypt servers are blocked since Turris OS 7.1

A certbot instance inside a LXC container cannot renew certificates (with the webroot method) since the update to TOS 7.1.

Let’s Encrypt try to contact my http server (running in the same container) for the webroot challenge, from the server outbound2p.letsencrypt.org (23.178.112.215) or outbound2l.letsencrypt.org (23.178.112.211), but the router firewall contains this in the set dynfw_4 (“IPv4 addresses blocked by Turris Sentinel”):

23.178.112.101, 23.178.112.103,
23.178.112.211, 23.178.112.215,

certbot error:

Timeout during connect (likely firewall problem)

Sentinel and Dynamic Firewall are enabled on Turris.

2 Likes

This is surely a problem in Sentinel. We filter out LE accesses (based on metadata, because IP addresses may vary), but it obviously fails due to an unknown reason. We will investigate and fix it.

4 Likes

Hi,

since I am having also issues, where can I find a complete list of that IPs?

Found that, but can it be they are only using 4 IPs?

Thx,
Vienna

They use much more addresses. The most of them are from the 23.178.112.0/24 range, but we have already detected some different addresses (probably related to their “multi-perspective validation”). Because of this, we don’t use IP addresses for LE detection at all, we use HTTP request metadata instead.

2 Likes

The implementation of the Let’s Encrypt filter in Sentinel has been fixed and improved. It should no longer block IP addresses used by Let’s Encrypt. We apologize for the problems.

5 Likes

Hi @ljelinek,

Thanks for your answer - how can I differentiate between normal and LE request HTTP? Since I only want to use SFTP with correct LE certificate.

Thanks,
Vienna

LE uses its specific User-Agent header and accesses only specific URLs (/.well-known/acme-challenge/<generated-string>).

1 Like

LE addresses are not in the blocklist anymore, but it’ still fails. LE requests don’t event reach my web server.

Does your container have a public address or forwarded ports (80, 443) from the router (that must have a public address of course)? Does your ISP allow to access the given ports from the Internet?

1 Like

Well yes, it worked fine for years, and I haven’t changed anything.

Please generate the diagnostics and send them to tech.support@turris.cz.

It worked when I disabled the http minipot in sentinel. That means the minipot was hijacking LE requests to my server whereas other http requests were correctly forwarded to my server.

2 Likes

This sounds strange. Please send us the diagnostics mentioned above.

There is too much private information, which modules are really needed?

I posted about the same issue here before: Dynamic firewall blocks Let's Encrypt renewals - #13 by Scrumplex

We need especially “firewall” and “network”.

Since TOS 7.1, iptables have been replaced by nftables and the Sentinel components have been modified a bit. Therefore it need not be the same problem.

Hi! I’m just commenting to report that disabling the HTTP minipot in Sentinel solved my problems.

I have a server behind my Turris router, it was having problems with the certbot renew command. The renewal response from the Let’s Encrypt servers was a 401, “unauthorized” status. It couldn’t complete the check request to my router on port 80.

Thank you for providing the solution @salty!