# Let's Encrypt X3 nightmare

**URL:** https://meta.discourse.org/t/lets-encrypt-x3-nightmare/204918
**Category:** Support
**Created:** [September 30, 2021, 10:51pm UTC](https://meta.discourse.org/t/lets-encrypt-x3-nightmare/204918 "2021-09-30T22:51:56Z")
**Posts on this page:** 14
**Page:** 1

<div class="post-metadata">

### Author: ![wlandgraf](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/wlandgraf/32/154660_2.png) [@wlandgraf](https://meta.discourse.org/u/wlandgraf)
#### Post date: [September 30, 2021, 10:51pm UTC](https://meta.discourse.org/t/lets-encrypt-x3-nightmare/204918/1 "2021-09-30T22:51:57Z")

</div>

I’m getting the following error when using the mail-receiver app as of today. It was working fine before.

It’s trying to send the e-mail to Discourse via API. Is it possible it’s caused by [DST Root CA X3](https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/) certificate expiration?

If yes, anyone has any idea about how to solve it? I tried to rebuild both app and mail-receiver, recreating Let’s Encrypt certificates, etc..

> \<19\>Sep 30 22:07:26 receive-mail[96]: Failed to POST the e-mail to [https://forum.validadortiss.com.br/admin/email/handle\_mail:](https://forum.validadortiss.com.br/admin/email/handle_mail:) SSL\_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (OpenSSL::SSL::SSLError)

---

<div class="post-metadata">

### Author: ![david](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/david/32/157490_2.png) [@david](https://meta.discourse.org/u/david)
#### Post date: [September 30, 2021, 11:12pm UTC](https://meta.discourse.org/t/lets-encrypt-x3-nightmare/204918/2 "2021-09-30T23:12:49Z")

</div>

Thanks for the report @wlandgraf - we’ll get a new mail-receiver image pushed up to Dockerhub ASAP

---

<div class="post-metadata">

### Author: ![david](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/david/32/157490_2.png) [@david](https://meta.discourse.org/u/david)
#### Post date: [September 30, 2021, 11:24pm UTC](https://meta.discourse.org/t/lets-encrypt-x3-nightmare/204918/3 "2021-09-30T23:24:56Z")

</div>

The `release` tag on Dockerhub has now been updated. Assuming you followed the official setup instructions, you can pull the new image by running

```plaintext
cd /var/discourse
./launcher rebuild mail-receiver

```

> **[Image Layer Details - discourse/mail-receiver:release](https://hub.docker.com/layers/discourse/mail-receiver/release/images/sha256-67222bded8651b2aefae113c57f183bba5d82b78e9ea60630a996d14833657dc)**

---

<div class="post-metadata">

### Author: ![wlandgraf](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/wlandgraf/32/154660_2.png) [@wlandgraf](https://meta.discourse.org/u/wlandgraf)
#### Post date: [September 30, 2021, 11:31pm UTC](https://meta.discourse.org/t/lets-encrypt-x3-nightmare/204918/4 "2021-09-30T23:31:29Z")

</div>

Hi @david, your prompt response is really appreciated. However, that didn’t solve my problem. It still persists.

---

<div class="post-metadata">

### Author: ![david](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/david/32/157490_2.png) [@david](https://meta.discourse.org/u/david)
#### Post date: [September 30, 2021, 11:36pm UTC](https://meta.discourse.org/t/lets-encrypt-x3-nightmare/204918/5 "2021-09-30T23:36:01Z")

</div>

In `/var/discourse/containers/mail-receiver.yml`, what do you have on the `base_image:` line? It should be:

```plaintext
base_image: discourse/mail-receiver:release

```

(More info on that [here](https://meta.discourse.org/t/how-to-update-mail-receiver-to-the-release-version/133491))

---

<div class="post-metadata">

### Author: ![wlandgraf](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/wlandgraf/32/154660_2.png) [@wlandgraf](https://meta.discourse.org/u/wlandgraf)
#### Post date: [September 30, 2021, 11:36pm UTC](https://meta.discourse.org/t/lets-encrypt-x3-nightmare/204918/6 "2021-09-30T23:36:39Z")

</div>

Yes, that’s what I have.

---

<div class="post-metadata">

### Author: ![wlandgraf](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/wlandgraf/32/154660_2.png) [@wlandgraf](https://meta.discourse.org/u/wlandgraf)
#### Post date: [September 30, 2021, 11:38pm UTC](https://meta.discourse.org/t/lets-encrypt-x3-nightmare/204918/7 "2021-09-30T23:38:17Z")

</div>

This is the beginning of my file:

```plaintext
## this is the incoming mail receiver container template
##
## After making changes to this file, you MUST rebuild
## /var/discourse/launcher rebuild mail-receiver
##
## BE *VERY* CAREFUL WHEN EDITING!
## YAML FILES ARE SUPER SUPER SENSITIVE TO MISTAKES IN WHITESPACE OR ALIGNMENT!
## visit http://www.yamllint.com/ to validate this file as needed

base_image: discourse/mail-receiver:release
update_pups: false

expose:
  - "25:25" # SMTP

```

And here is the (partial) output of the rebuild operation (omitted the parts that have API keys):

```plaintext
Ensuring launcher is up to date
Fetching origin
Launcher is up-to-date
Stopping old container
+ /usr/bin/docker stop -t 60 mail-receiver
mail-receiver
cd /pups && /pups/bin/pups --stdin
sha256:5f123a8eb11784828d5195ee0f328a0ea5a5d2ce36eeae1760e3d47b0dbeb15c
165ebaa91836a07696924f95d3746cbd1cc14412f478ba715ee40f502780ab7a
Removing old container
+ /usr/bin/docker rm mail-receiver
mail-receiver
``
```

---

<div class="post-metadata">

### Author: ![david](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/david/32/157490_2.png) [@david](https://meta.discourse.org/u/david)
#### Post date: [September 30, 2021, 11:40pm UTC](https://meta.discourse.org/t/lets-encrypt-x3-nightmare/204918/8 "2021-09-30T23:40:58Z")

</div>

Can you try

```plaintext
docker pull discourse/mail-receiver:release
cd /var/discourse
./launcher rebuild mail-receiver

```

---

<div class="post-metadata">

### Author: ![wlandgraf](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/wlandgraf/32/154660_2.png) [@wlandgraf](https://meta.discourse.org/u/wlandgraf)
#### Post date: [September 30, 2021, 11:45pm UTC](https://meta.discourse.org/t/lets-encrypt-x3-nightmare/204918/9 "2021-09-30T23:45:13Z")

</div>

That did it! Thank you very much! A few questions if you don’t mind:

1. Why was that necessary? Should I always manually pull the latest image before building?
2. Do I have to apply the same procedure (docker pull) when rebuilding the app?
3. I see e-mails sent after the rebuild arrived, but not the e-mails previously sent. Were those e-mails lost?

---

<div class="post-metadata">

### Author: ![david](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/david/32/157490_2.png) [@david](https://meta.discourse.org/u/david)
#### Post date: [September 30, 2021, 11:49pm UTC](https://meta.discourse.org/t/lets-encrypt-x3-nightmare/204918/10 "2021-09-30T23:49:04Z")

</div>

It looks like your docker installation had cached the `:release` tag. To avoid the need for a docker pull in future, we might be able to add some logic to our `launcher` script.

The `app` images are handled differently, so there won’t be any caching issue there.

Emails sent during the outage should have been returned to the sending server with a “temporary fail”. Those servers should retry the mail periodically, so hopefully you will see the missing emails arrive in the next few hours.

---

<div class="post-metadata">

### Author: ![wlandgraf](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/wlandgraf/32/154660_2.png) [@wlandgraf](https://meta.discourse.org/u/wlandgraf)
#### Post date: [September 30, 2021, 11:50pm UTC](https://meta.discourse.org/t/lets-encrypt-x3-nightmare/204918/11 "2021-09-30T23:50:56Z")

</div>

Excelent. Thank you again for the fast response. The world seems to be suffering from this Root CA X3 mess, you have now made your contribution to make the world a little bit better now. 😃

---

<div class="post-metadata">

### Author: ![pfaffman](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/pfaffman/32/120154_2.png) [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [October 1, 2021, 12:09am UTC](https://meta.discourse.org/t/lets-encrypt-x3-nightmare/204918/12 "2021-10-01T00:09:18Z")

</div>

Any clues on how to detect whether a mail receiver needs to be updated and whether the docker pull will be necessary?

---

<div class="post-metadata">

### Author: ![david](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/david/32/157490_2.png) [@david](https://meta.discourse.org/u/david)
#### Post date: [October 1, 2021, 12:12am UTC](https://meta.discourse.org/t/lets-encrypt-x3-nightmare/204918/13 "2021-10-01T00:12:33Z")

</div>

All mail-receivers installed before today will need to be updated. (technically, any install using a base image earlier than `67222bded865`)

The `docker pull` will be necessary until we update `launcher` (which is unlikely to be today/tomorrow). I just pushed an announcement topic here, because I know this will be disruptive to many sites. It includes update instructions, including the `docker pull`:

> [@Self-hosted mail-receiver update following Let's Encrypt root certificate change](https://meta.discourse.org/t/self-hosted-mail-receiver-update-following-letsencrypt-root-certificate-change/204924):
>
> This announcement only affects self-hosters who are running [Configure direct-delivery incoming email for self-hosted sites with Mail-Receiver](https://meta.discourse.org/t/direct-delivery-incoming-email-for-self-hosted-sites/49487). Managed hosting customers, and self-hosters using POP3 for incoming mail are not affected. As you may be aware, Let’s Encrypt recently [changed their root certificate](https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/). The old root certificate expired today, and has caused a number of issues for out-of-date clients across the web. All of our systems at CDCK are up-to-date, and were not affected by today’s…

---

<div class="post-metadata">

### Author: ![david](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/david/32/157490_2.png) [@david](https://meta.discourse.org/u/david)
#### Post date: [October 2, 2021, 7:00am UTC](https://meta.discourse.org/t/lets-encrypt-x3-nightmare/204918/18 "2021-10-02T07:00:10Z")

</div>

This topic was automatically closed after 30 hours. New replies are no longer allowed.
