Discourse failure to renew certificate

[RBoy]

Continuing the conversation from here:

Letsencrypt certificate failure to renew

I think it’s been years since I’ve seen a let’s encrypt certificate fail to renew, but I’ve had three sites in the past week or two that have stale certificates. A rebuild fixes it and I have heretofore failed to look for clues in the logs. Next time I’ll do a bit more to diagnose the problem before fixing it for the site.

I got a reminder from Redsift that my certificates are going to expire in a week. Usually discourse will renew the certificates well ahead of time. This time not so, before I start doing a rebuild (which is supposed to solve the issue), @Falco is there anything you want me to check and post back here to help get to the root of why the certificates did not renew?

The root certificate is ISRGX1 and here is the expiring certificate information:

Common Name (CN)	E6
Organization (O)	Let’s Encrypt
Organizational Unit (OU)
Issued On	Wednesday, July 16, 2025 at 7:36:45 PM
Expires On	Tuesday, October 14, 2025 at 7:36:44 PM

The current build is 3.6.0.beta1-dev (7ee52c8f85)

[pfaffman (Jay Pfaffman)]

There was a period of time that the endpoint that let’s encrypt needed was redirected. That’s fixed if you rebuild.

[Dannii]

About how long after I update should the certificate be renewed?

[featheredtoast (Jeff Wong)]

If memory serves the certificates are valid for 3 months, and they will now attempt to auto renew before then.

[Dannii]

Yes I know how it’s meant to work.

But it’s been over a day since I updated the forum software and the certificate doesn’t appear to have been updated yet. It’s got 5 days before it expires so it really needs to be renewed soon.

I’m on the Discouse stable branch if that makes a difference. Is it possible the endpoint fix hasn’t been backported?

[RBoy]

For me the certificate updated immediately after the rebuild

[Dannii]

Did you rebuild through the web or via the command line?

[Canapin (Coin-coin le Canapin)]

Yes, the explanation was here:

Let's Encrypt certificate doesn't automatically renew, but rebuilding does trigger the renewal

I recently worked on the le cert renewal. It is the http to https redirect - acme does not handle being told to redirect well at all and by default attempts to connect on the same protocol (http) as it did when it was setup initially Recent updates to the let’s encrypt template should fix these renewals going forward.

[Dannii]

My forum has finally updated its certificate after I did a command line rebuild.

[lessLost (Michael)]

We have had the same experience of SSL not renewing.

It would be great if someone could double check that web.ssl.template is behaving correctly on discourse-docker, it appeared to me that port 80 was not actually serving any /.well-known/ URLs used by Let’s Encrypt, all URLs were forwarding to SSL including test files I manually placed into /var/www/discourse/public/.well-known/ . I had to edit /etc/nginx/conf.d/outlets/before-server/20-redirect-http-to-https.conf directly inside the app container.

Perhaps this started after commit ae4887a of discourse-docker?

[pfaffman (Jay Pfaffman)]

There was another error with the well known route in recent memory.

When’s the last time you did a rebuild?

[giuseppe.lanzi]

Same here. I didn’t get a warning abount the cert’s expiration. Enterine the server and launching a rebuild /var/discourse % ./launcher rebuild did the trick.

[lessLost (Michael)]

In my testing on a vanilla nginx install (1.18.0 but I think it’s the same for 1.26.3), an nginx config line return 301 https://thehostname$request_uri; outside of a location completely overrides any earlier location block before it, rather than being a catch-all. I believe /.well-known/ simply isn’t served on port 80 unless the 301 redirect is specifically for another location such as / at the end of the server block. Could be the same problem as this stackoverflow post?

Glad rebuild works, but since the cert had already renewed for me, I couldn’t confirm that a rebuild would allow the Let’s Encrypt validation servers to get there if a cert had expired. Maybe a rebuild kicks off the cert renewal before that template line is in place or similar rather than fixing the templates, but I’m not able to confirm if that’s why rebuild gets the renewal to work.

[pfaffman (Jay Pfaffman)]

If you think this is a Discourse but then perhaps you should reply on the github commit or open a new big report.

[cedric_chin]

@lessLost

In my testing on a vanilla nginx install (1.18.0 but I think it’s the same for 1.26.3), an nginx config line return 301 https://thehostname$request_uri; outside of a location completely overrides any earlier location block before it, rather than being a catch-all. I believe /.well-known/ simply isn’t served on port 80 unless the 301 redirect is specifically for another location such as / at the end of the server block. Could be the same problem as this stackoverflow post?

I can confirm that letsencrypt renewal fails. I’ve been running a self-hosted Discourse install for years, and very strangely renewal failed for me two times in a row over the past couple of months. The second time was this morning, and so I started investigating.

I’ve traced it to the following two commits:

Use Nginx config outlets in templates

And relevant line linked:

https://github.com/discourse/discourse_docker/commit/c9064be6b7a743e3d86dbc69ddaa80701766aa87#diff-b8c95dfe3760424eecc60d21538dbd785f096d46ef95764c60f4b608f40dd536R26

There are two issues, I think.

First, return 301 https://${DISCOURSE_HOSTNAME}$request_uri; gets turned into return 301 https://<MY SERVER NAME> without a $request_uri at the end. I have verified on my self-hosted install, and also on a friend’s self-hosted install that was set up in the past week. I don’t understand how Discourse template works, so I don’t know why it gets dropped.

Second, as @lessLost mentioned, the 301 redirect is outside of the location block. I believe a server level redirect overrides all location blocks. LetsEncrypt uses http for renewals. However, any attempt to curl -I http://YOUR_DOMAIN/.well-known/acme-challenge/test will return a 301 to https, instead of a 404 (which is expected behaviour; we want a 404 not a 301).

I’ve fixed this manually on my self-hosted install, but I expect any update will override my changes. Unfortunately I don’t understand the templates enough to submit a pull request @pfaffman — or I’d do that too.

Edited to add:

I believe this is mistaken —

Let's Encrypt certificate doesn't automatically renew, but rebuilding does trigger the renewal

I recently worked on the le cert renewal. It is the http to https redirect - acme does not handle being told to redirect well at all and by default attempts to connect on the same protocol (http) as it did when it was setup initially Recent updates to the let’s encrypt template should fix these renewals going forward.

I’m fairly certain LetsEncrypt uses http by default (for obvious reasons, if the cert is expired then it can’t renew!) But placing the 301 at the server block level forces all requests to 301 to https, which is inconsistent with this renewal strategy.

Edit 2: Evidence for the http renewal strategy, but you may also Google around to verify this.

[pfaffman (Jay Pfaffman)]

When did you last rebuild?

[cedric_chin]

This morning, roughly 10 minutes after I woke up, visited my forum, and realised the cert had expired again. (Rebuilding was what renewed the last time it expired on me — roughly 3 months ago?)

[pfaffman (Jay Pfaffman)]

@cedric_chin

(Rebuilding was what renewed the last time it expired on me — roughly 3 months ago?)

I think that they’ve committed changes since then that have fixed this problem, but since it takes 3 months to find out, the jury is still out. You might set a reminder a couple weeks before the current cert expires.

[cedric_chin]

This is not true.

1. The links I attached in the comment above was from a git blame. Here is the latest version of the file (relevant line linked): discourse_docker/templates/web.ssl.template.yml at 247c71a1e45d32b0b814a8e9d5fdaa4faaf727b9 · discourse/discourse_docker · GitHub
2. My friend’s new site install was from a week ago. Line 37 of the template above reads: return 301 https://${DISCOURSE_HOSTNAME}$request_uri; but in the Discourse Docker container, both her (and my) /etc/nginx/conf.d/outlets/before-server/20-redirect-http-to-https.conf reads return 301 https://<our_discourse_site>; Notice how $request_uri is stripped. Something is causing that to vanish! (I don’t know what).
3. I did a simulated forced renewal this morning as part of my investigation. It failed. I then changed /etc/nginx/conf.d/outlets/before-server/20-redirect-http-to-https.conf. It succeeded!

This is actually ok; I’ll just manually edit 20-redirect-http-to-https.conf every time I update Discourse. For those stumbling on this comment, the command to run is:

cat > /etc/nginx/conf.d/outlets/before-server/20-redirect-http-to-https.conf << 'EOF'
server {
  listen 80;
  listen [::]:80;

  location ~ /.well-known {
    root /var/www/discourse/public;
    allow all;
  }

  location / {
    return 301 https://<YOUR_FORUM_ADDRESS>$request_uri;
  }
}
EOF

I’m not entirely sure what is causing this failure, but I know modifying the conf above fixes it. But I’ve also modified notifs so that letsencrypt renewals no longer fails silently — so I can have some advanced warning. Just thought you’d like to know!

[nat (Natalie T)]

Thanks for the report, I think this is legit.

(fyi @featheredtoast)