Configure direct-delivery incoming email for self-hosted sites with Mail-Receiver

[Ryan_Hyer]

Hello! I have a weird issue where I’ve set this up following the guide, and it works great! However, something has gone wrong with outbound email, which I thought wouldn’t be affected by any of this. Sidekiq gives the following error for every attempted email (all of them stuck in the Retries list) since I turned on mail-receiver:

Jobs::HandledExceptionWrapper: Wrapped OpenSSL::SSL::SSLError: SSL_read: unexpected eof while reading

My searching leads me to believe that this is related to TLS somehow. I had uncommented the TLS-related lines in the .yml file, but re-commenting them didn’t fix the issue either. I tried the instructions in the guide for resolving Postfix conflicts, but I apparently don’t have Postfix? (The /etc/postfix directory in the guide doesn’t exist on my instance, nor does it recognize postfix as a service.) And according to the netstat results, only docker-proxy is using port 25.

We’re using Gmail as the outbound SMTP service, and in fact we were using Gmail for inbound POP3 polling before this. I did delete a bunch of MX records pointing to Google, but the guide said to do that.

This is my mail-receiver.yml, with certain details redacted, of course:

## this is the incoming mail receiver container template
##
## After making changes to this file, you MUST rebuild
## /var/discourse/launcher rebuild mail-receiver
##
## BE *VERY* CAREFUL WHEN EDITING!
## YAML FILES ARE SUPER SUPER SENSITIVE TO MISTAKES IN WHITESPACE OR ALIGNMENT!
## visit http://www.yamllint.com/ to validate this file as needed

base_image: discourse/mail-receiver:release
update_pups: false

expose:
  - "25:25"   # SMTP

env:
  LC_ALL: en_US.UTF-8
  LANG: en_US.UTF-8
  LANGUAGE: en_US.UTF-8

  ## Where e-mail to your forum should be sent.  In general, it's perfectly fine
  ## to use the same domain as the forum itself here.
  MAIL_DOMAIN: discourse.[mydomain].org
# uncomment these (and the volume below!) to support TLS
  POSTCONF_smtpd_tls_key_file:  /letsencrypt/discourse.[mydomain].org/discourse.[mydomain].org.key
  POSTCONF_smtpd_tls_cert_file:  /letsencrypt/discourse.[mydomain].org/fullchain.cer
  POSTCONF_smtpd_tls_security_level: may


  ## The base URL for this Discourse instance.
  ## This will be whatever your Discourse site URL is. For example,
  ## https://discourse.example.com. If you're running a subfolder setup,
  ## be sure to account for that (ie https://example.com/forum).
DISCOURSE_BASE_URL: 'https://discourse.[mydomain].org'

  ## The master API key of your Discourse forum.  You can get this from
  ## the "API" tab of your admin panel.
  DISCOURSE_API_KEY: [myapikey]

  ## The username to use for processing incoming e-mail.  Unless you have
  ## renamed the `system` user, you should leave this as-is.
  DISCOURSE_API_USERNAME: system

volumes:
  - volume:
      host: /var/discourse/shared/mail-receiver/postfix-spool
      guest: /var/spool/postfix
# uncomment to support TLS
  - volume:
      host: /var/discourse/shared/standalone/letsencrypt
      guest: /letsencrypt

Email tech is a bit out of my expertise, so I appreciate any advice, even if it’s to point out that I missed something stupid when I was setting this up. Thanks!

[pfaffman (Jay Pfaffman)]

As you thought, it’s nothing to do with the neck receiver. The host that you’re sending mail through has a broken ssl cert.

[Ryan_Hyer]

Well, I figured it out after much troubleshooting. The issue likely came from the fact that the domain we host our Discourse instance on is not the same as the domain our MX records were on. Once I got past that confusion, it all came together.

It’s definitely my own stupid mistake, but the guide contributed to my confusion a bit with this:

@Discourse

• DNS Name: forum.example.com (this is the MAIL_DOMAIN)
• Type: MX
• Priority: 10
• Value: forum.example.com (this is the domain of your forum)

It’s not super clear that the two forum.example.com entries don’t have to be identical, and in my case they needed to be different. That may be something that people using this guide should be experienced enough to know, but I was not. So I’m leaving this here for anyone else who may run into a similar problem. I learned a few things about DNS that I didn’t know, so this was a good learning experience, and it’s all working great now.

[Ryan_Hyer]

Well, I spoke too soon. Outbound email works fine, inbound replies seem to work fine, but posting to a category’s email address is failing silently. I copy/pasted the address straight from the settings into a new email, so I know there are no typos.

My mail-receiver’s logs basically have three types of entries. The successful one, which was an emailed reply to an existing post, looks like this:

Sep 20 16:59:44 discourse-mail-receiver postfix/smtpd[277]: connect from server168-1.web-hosting.com[68.65.122.144]
Sep 20 16:59:45 discourse-mail-receiver postfix/smtpd[277]: NOQUEUE: reject: RCPT from server168-1.web-hosting.com[68.65.122.144]: 454 4.7.1 <[category]@discourse.[domain].org>: Relay access denied; from=<ryan@[redacted].com> to=<[category]@discourse.[domain].org> proto=ESMTP helo=<server168-1.web-hosting.com>
<22>Sep 20 16:59:45 policyd-spf[288]: : prepend Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=[redacted]; helo=server168-1.web-hosting.com; envelope-from=ryan@[redacted].com; receiver=discourse.[domain].org Sep 20 16:59:45 discourse-mail-receiver postfix/cleanup[281]: 4CCED114200: message-id=<20240920165945.4CCED114200@discourse-mail-receiver.localdomain>
Sep 20 16:59:45 discourse-mail-receiver postfix/smtpd[277]: disconnect from server168-1.web-hosting.com[68.65.122.144] ehlo=1 starttls=0/1 mail=1 rcpt=0/1 data=0/1 quit=1 commands=3/6

Other than that, there are two types of (what I assume are) errors, each of which repeats quite a lot. The first one looks like:

Sep 20 17:00:23 discourse-mail-receiver postfix/qmgr[124]: 5D162FC26D: from=<double-bounce@discourse-mail-receiver.localdomain>, size=960, nrcpt=1 (queue active)

And the other:

Sep 20 17:00:23 discourse-mail-receiver postfix/error[293]: 8DC3BFC141: to=<postmaster@discourse-mail-receiver.localdomain>, orig_to=<postmaster>, relay=none, delay=126622, delays=126622/0.05/0/0, dsn=4.4.3, status=deferred (delivery temporarily suspended: Host or domain name not found. Name service error for name=discourse-mail-receiver.localdomain type=MX: Host not found, try again)

And here’s what my mailq looks like, just entries like this one over and over and over:

3D07BFC23D      960 Fri Sep 20 06:42:23  double-bounce@discourse-mail-receiver.localdomain
(delivery temporarily suspended: Host or domain name not found. Name service error for name=discourse-mail-receiver.localdomain type=MX: Host not found, try again)
                                         postmaster@discourse-mail-receiver.localdomain

Some of this looks like it has to do with emails that Discourse sends, which then get bounced back for whatever reason. Does mail-receiver have any sort of functionality to processes these bounces, or are they going to sit in the mailq forever?

Secondly, why do replies work, but email posting directly to a category does not? Thanks again for your help and your patience.

[Simon_Manning]

@Ryan_Hyer

The successful one, which was an emailed reply to an existing post, looks like this:

...
454 4.7.1 <[category]@discourse.[domain].org>: Relay access denied;
...

That looks like the log entries for failure to a category’s address rather than success replying to a post.

@Ryan_Hyer

Secondly, why do replies work, but email posting directly to a category does not?

I’m not 100% sure but I think relay access denied suggests discourse.[domain].org is probably not the domain used for MAIL_DOMAIN in mail-receiver.yml. Possibly the reply address is allowed through other means.

I know what’s used in MAIL_DOMAIN ends up in at least one place in a postfix config file so changing it probably requires rebuilding the container. Have you changed MAIL_DOMAIN and if so, did you run ./launcher rebuild mail-receiver afterwards?

[Ryan_Hyer]

[Sorry for accidentally hitting enter prematurely before finishing my previous post]

I am still banging my head against this issue. But I have a new idea on what the issue might be. I am working with two domains, let’s call them [domain1] and [domain2]. My Gmail SMTP relay is hosted on [domain1]. My Discourse instance, as well as my mail-receiver is hosted on [domain2].

How do I set the reply-by-email-address setting in Discourse to force a reply-to address in [domain2], when the email is being sent out from [domain1]? I get the SSL EOF error mentioned above when trying to do this. I assume there is some DNS authentication trickery or something I’m missing.

[Ryan_Hyer]

Looks like I’ve figured this out finally. In order to make the ‘reply-to’ address be in a different domain than the SMTP relay, there were some settings I needed to loosen up in Google Workspace. Everything seems to be working as intended in both directions.

[Ryan_Hyer]

One final question here. Even though everything works correctly now, I still have a bunch of old entries in my mailq. These are most likely emails that were generated with the wrong settings and so will be stuck in limbo forever. I’d rather just delete them and move on. So, how do I clear the mailq?

[Jagster (Jakke Lehtonen)]

@Ryan_Hyer

SSL EOF error

A bit old post, but perhaps the most common reason for SSL eof error is a conflict between OpenSSL versions: v1.1.1f vs v3. Upgrading that old 1.1.1f will be solution. And bad news are that for example Ubuntu 20.x can’t use newer one, so whole Ubunto mist ne upgraded.

[denvergeeks]

At my wits end on this one, after spending hours. I can’t seem to get past this Launcher error, even though my config file names are all lowercase.

“ERROR: Config name must not contain upper case characters, spaces or special characters. Correct config name and rerun ./launcher.”

while running

./launcher rebuild mail-receiver

or

./launcher bootstrap mail-receiver

or

./launcher start mail-receiver

I can see it here in the launcher codebase

Grrrrrrrrrr – Please help!

Tried everything on the linked post above related to locale, and everything I could find elsewhere.

./launcher rebuild app___ are all working fine!

[denvergeeks]

I do have one possible clue: this started happening immediately after I had accidentally hit the CapsLock (but only capitalized 2 of the letters) whilst naming the config file, which I had then immediately disabled the CapsLock and retyped the 2 letters before even saving it.

Hard to imagine how this brief typo/correction could have caused this, but maybe the caps are stuck in a buffer somewhere, or ???

[Canapin (Coin-coin le Canapin)]

It’s far above my knowledge, but I’m surprised the error message doesn’t output the $config variable
It would help debugging for sure.

[denvergeeks]

Thanks @Canapin ! - this is what I’m trying to set up:

https://www.perplexity.ai/search/provide-the-code-lJcI4BrFQ2auuD42ehYFwA

[Canapin (Coin-coin le Canapin)]

Can you copy-paste all the content of your command line?

From your ./launcher start mail-receiver to the error message, as well as the exact .yml filename?

If I rename the config file as Mail-receiver.yml, ./launcher start Mail-receiver will output

ERROR: Config name 'Mail-receiver' must not contain upper case characters, spaces or special characters. Correct config name and rerun ./launcher.

Here the error message contains the file name.

Also, if you run ./launcher start aaa, it won’t find any corresponding file and will list the available ones. It only picks them from the folder so there’s no magic here, but maybe it will output something interesting

ERROR: containers/aaa.yml does not exist or is not readable.

Available configs ( app, mail-receiver )

[denvergeeks]

Hey thanks a lot – I got it sorted and working.

[Canapin (Coin-coin le Canapin)]

What was the issue in the end? Could help others

[denvergeeks]

There was no issue, it was really just a learning curve to understand how the various server components interact for routing domains and email. I’d never before dipped into learning about postfix. It was fun and I learned a lot.

The recipe I finally arrived at is to use a mail-receiver.yml (a docker container) paired to each instance of Discourse, all sharing port 25 using the transport feature in postfix to handle the routing.

[denvergeeks]

On my dedicated server (running Ubuntu 22.04, with Postfix installed) I use a separate mail-receiver.yml paired to each instance of Discourse where I have enabled the mail-in posting feature.

This setup creates a separate container for each instance of Discourse on my server (alongside the typical app container) which receives and processes the e-mails for its corresponding Discourse instance.

Incoming emails for all of the Discourse forums on the server are received by Postfix through the standard port 25, where the main Postfix configuration file uses a “transport map” to “relay” each email to its intended Discourse forum by parsing out the domain name in the email’s “To:” address.

So, in addition to the instructions of this Topic, I…

1. modified the existing postfix configuration file at: /etc/postfix/main.cf
   

   

2. then, I added the corresponding postfix transport map file at: /etc/postfix/transport
   

   

3. lastly, I added the corresponding files to create the email container for each of the forums:
   /var/discourse/containers/mail-receiver-domain1.yml
/var/discourse/containers/mail-receiver-domain2.yml
/var/discourse/containers/mail-receiver-domain3.yml
/var/discourse/containers/mail-receiver-domain4.yml
/var/discourse/containers/mail-receiver-domain5.yml
   
   

[Canapin (Coin-coin le Canapin)]

@Discourse

open containers/mail-receiver.yml in your preferred text editor and change the MAIL_DOMAIN, DISCOURSE_MAIL_ENDPOINT, and DISCOURSE_API_KEY variables to suit your site. (If you are an advanced user and know that you are using nginx outside your container, see below for additional configuration for external nginx.)

If you use the default mail endpoint (/admin/email/handle_mail), we suggest using the receive_email API key scope to provide an extra layer of security.

There is no DISCOURSE_MAIL_ENDPOINT in mail-receiver.yml, and there’s also DISCOURSE_BASE_URL to change.

[LotusJeff (Jeff Cocking)]

I use a mail forwarding service that supports forwarding emails in a JSON format to a webhook.

Is this an option for direct delivery of email?