My forum went down this night, between 10 and 17h ago (can’t tell precisely). The pages were apparently loading cached versions and a lot of resources didn’t load.
I stopped and started the container with no success.
I rebuilt it (once), and have now an expired certificate message.
Create new order error. Le_OrderFinalize not found. {
"type": "urn:ietf:params:acme:error:rateLimited",
"detail": "Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/rate-limits/",
"status": 429
}
I looked at the log more carefully and saw previously multiple times, once a day from March (not enough to trigger Letsencrypt rate limit, regarding my logs):
unicyclist.com:Verify error:Fetching http://unicyclist.com/.well-known/acme-challenge/uSv1JIUxVs-Nn7Zn2cIZO355KGaqrtutELs4pgw67_Y: Timeout during connect (likely firewall problem)
I set this firewall rules on Hetzner at the beginning of March:
Could these Hetzner Firewall rules be the issue? If so, what rule(s) are missing that could cause the issue? I removed all these rules right now, by the way.
Why did I had unicyclist.com:Verify error:Fetching […] Timeout during connect (likely firewall problem) message 10 times in a single rebuild this morning? Could a single rebuilt trigger Letsencrypt rate limit?
Since I have reached a rate limit, does that mean that my forum is basically down for a week and I can do nothing about it?
The solution that I have used is to add a second hostname (e.g., www, but it could be anything) as suggested in Setting up Let’s Encrypt with Multiple Domains, but I believe that there have been some changes to the templates that cause those instructions not to work. What I did for another site a couple of days ago was to edit /etc/runit/1.d/letsencrypt and add -d newdomain in the places that you see -d realdomain. Then make sure that the new domain is pointed to your server and then run /etc/runit/1.d/letsencrypt.
You may also need to open port 80. My understanding is that it’s not needed when there is a valid cert, but you don’t have a valid cert, so I think it needs to read from port 80 to get things started. There is no downside to having port 80 open, as some people might try to access with http:// and if you have port 80 open they can be redirected to https.
I’ll work on updating those instructions, but I have a moving truck arriving tomorrow and should really be getting ready for that rather than posting on meta anyway.
Thank you for your reply!
Does using a new domain/subdomain would require a rebake of all posts though?
I have 1,6M posts. A rebake would also trigger Youtube embedding rate limit, as I came across when I imported this forum from vBulletin.
The issue with a temporary new domain/subdomain is that I’m not the owner of the domain my forum is hosted on, which is quite annoying. The owner’s replies can be slow, and if anything goes wrong, it’s other emails going back and forth… Not very practical
I indeed saw in Challenge Types - Let's Encrypt that the port 80 should be available in some “challenges” (I don’t know what kind of challenge is used when renewing a certificate from Discourse).
Besides this, I’m really interested to know a bit more about what seems to be 10 failed challenges (is that equal to a certificate request?) during a single rebuild.
Maybe an official reply about this could be interesting?
Because if for some reason this triggers more certificate requests that it is permitted by Let’s encrypt’s rate limit, then it shouldn’t behave like this
Excerpt of acme.sh.log:
[Thu 14 Apr 2022 10:29:01 AM UTC] payload
[Thu 14 Apr 2022 10:29:01 AM UTC] POST
[Thu 14 Apr 2022 10:29:01 AM UTC] _post_url='https://acme-v02.api.letsencrypt.org/acme/chall-v3/98157717290/JXMDvA'
[Thu 14 Apr 2022 10:29:01 AM UTC] _CURL='curl --silent --dump-header /shared/letsencrypt/http.header -L '
[Thu 14 Apr 2022 10:29:01 AM UTC] _ret='0'
[Thu 14 Apr 2022 10:29:01 AM UTC] code='200'
[Thu 14 Apr 2022 10:29:01 AM UTC] Pending
[Thu 14 Apr 2022 10:29:01 AM UTC] sleep 2 secs to verify
[Thu 14 Apr 2022 10:29:03 AM UTC] checking
[Thu 14 Apr 2022 10:29:03 AM UTC] url='https://acme-v02.api.letsencrypt.org/acme/chall-v3/98157717290/JXMDvA'
[Thu 14 Apr 2022 10:29:03 AM UTC] payload
[Thu 14 Apr 2022 10:29:03 AM UTC] POST
[Thu 14 Apr 2022 10:29:03 AM UTC] _post_url='https://acme-v02.api.letsencrypt.org/acme/chall-v3/98157717290/JXMDvA'
[Thu 14 Apr 2022 10:29:03 AM UTC] _CURL='curl --silent --dump-header /shared/letsencrypt/http.header -L '
[Thu 14 Apr 2022 10:29:04 AM UTC] _ret='0'
[Thu 14 Apr 2022 10:29:04 AM UTC] code='200'
[Thu 14 Apr 2022 10:29:04 AM UTC] unicyclist.com:Verify error:Fetching http://unicyclist.com/.well-known/acme-challenge/uSv1JIUxVs-Nn7Zn2cIZO355KGaqrtutELs4pgw67_Y: Timeout during connect (likely firewall problem)
[Thu 14 Apr 2022 10:29:04 AM UTC] pid
Then a few seconds later:
[Thu 14 Apr 2022 10:29:18 AM UTC] payload
[Thu 14 Apr 2022 10:29:19 AM UTC] POST
[Thu 14 Apr 2022 10:29:19 AM UTC] _post_url='https://acme-v02.api.letsencrypt.org/acme/chall-v3/98157816830/tzknWw'
[Thu 14 Apr 2022 10:29:19 AM UTC] _CURL='curl --silent --dump-header /shared/letsencrypt/http.header -L '
[Thu 14 Apr 2022 10:29:19 AM UTC] _ret='0'
[Thu 14 Apr 2022 10:29:19 AM UTC] code='200'
[Thu 14 Apr 2022 10:29:19 AM UTC] unicyclist.com:Verify error:Fetching http://unicyclist.com/.well-known/acme-challenge/UUi8goql9f4QjXwqdk_CUISDmwUpLHqhrSqwbr5D2aY: Timeout during connect (likely firewall problem)
[Thu 14 Apr 2022 10:29:19 AM UTC] pid
And like this, 10 times during this rebuild.
I don’t know much about this, so maybe I completely misunderstand things here.
You should open port 80. It’s required for renewing the certificate. Also, your site won’t work for people trying to access it over HTTP for the first time as they won’t get the permanent redirect to HTTPS.
Glad that did it! (For my work-around for too many let’s encrypt requests, you don’t change the forum URL, just add a second domain to the cert request. It can be any other domain, so you could point some domain that you do own at the forum and rebuild as I suggested, but thankfully you didn’t need that.)
