Configure direct-delivery incoming email for self-hosted sites with Mail-Receiver

Discourse · August 31, 2016, 10:58am

Discourse is all about enabling civilized discussion. While plenty of people like a web interface, e-mail is still the “hub” of many people’s online lives. That’s why sending e-mail is so important, and when you’re sending e-mail, you really want to be able to receive it, too. There are several reasons why:

If e-mails “bounce” (they can’t be delivered for some reason), you need to know about that. Repeatedly sending e-mails that bounce will get your e-mails flagged as spam. Receiving e-mail bounces allows you to disable sending to non-existent addresses.
Allowing people to reply to posts via e-mail can significantly improve engagement, as people can reply straight away from their mail client, even if they’re not able to visit the forum at that moment.
Letting people post new topics, or send PMs, via e-mail has similar benefits to engagement. In addition, you can use Discourse to handle e-mail for a group, such as an e-mail-based support channel (which is how Discourse’ own e-mail support is handled).

Delivering e-mail directly into your Discourse forum, rather than setting up POP3 polling, has a number of benefits:

No need to deal with gmail or another provider’s foibles;
You have more control over the e-mail addresses that people use to send posts; and
There are no delays in delivery – no more waiting for the next polling run to see new posts appear!

This howto is all about getting that hawtness into your forum.

Overview

This procedure creates a new container on your Discourse server, alongside the typical app container, which receives e-mail and forwards it into Discourse for processing. It supports all e-mail processes: handling bounces, replies, and new topic creation. Any self-hosted Discourse forum using our supported installation process can make use of this procedure to get easy, smooth-flowing incoming e-mail.

Container Setup

We’re going to get the mail-receiver container up and running on the server that’s already running your Discourse instance. There’s no need for a separate droplet just to handle mail – the whole container only takes about 5MB of memory!

So, start off by logging into your Discourse server, and becoming root via sudo:

ssh ubuntu@192.0.2.42
sudo -i

Now, go to your /var/discourse directory and create a new mail-receiver.yml container definition from the sample conveniently provided:

cd /var/discourse
git pull
cp samples/mail-receiver.yml containers/

Since every site is unique, open containers/mail-receiver.yml in your preferred text editor and change the MAIL_DOMAIN, DISCOURSE_MAIL_ENDPOINT, and DISCOURSE_API_KEY variables to suit your site. (If you are an advanced user and know that you are using nginx outside your container, see below for additional configuration for external nginx.)

If you use the default mail endpoint (/admin/email/handle_mail), we suggest using the receive_email API key scope to provide an extra layer of security.

If you’re not sure what your favourite text editor is, try nano:

nano containers/mail-receiver.yml

Use Ctrl-X to exit (say “Yes” to “Do you want to save changes?”, or all your work will be for nothing).

Now, do an initial build of the container, and fire it up!

./launcher bootstrap mail-receiver
./launcher start mail-receiver

To check everything’s OK, take a peek in the logs:

./launcher logs mail-receiver

The last line printed should look rather a lot like this:

<22>Aug 31 04:14:31 postfix/master[1]: daemon started -- version 3.1.1, configuration /etc/postfix

If so, all is well, and you can go on to then next step.

DNS Setup

In order for everyone else on the Internet to know where to deliver mail, you must create an MX record for your forum. The exact details of how to do this vary by DNS provider, but in general, the procedure should be very similar to how you setup the DNS records for your forum in the first place, except that instead of creating an A (or “Address”) record, you’re creating an MX (or “Mail eXchange”) record. If your forum is at forum.example.com, and you set MAIL_DOMAIN to forum.example.com in the mail-receiver.yml, then the DNS record should look like this:

DNS Name: forum.example.com (this is the MAIL_DOMAIN)
Type: MX
Priority: 10
Value: forum.example.com (this is the domain of your forum)

To make sure the DNS is setup correctly, use a testing site such as http://mxtoolbox.com/ to look up the MAIL_DOMAIN you configured, and make sure it’s pointing to where you expect.

Note: outbound email providers like mailgun may ask you to add MX records pointing to their servers. You want to remove these so the MX records for your forum only point to your forum’s domain name. SPF and DKIM records must still point to your outbound email provider servers so you can send email.

Discourse Configuration

Now e-mail is being fed into Discourse, it’s time to explain to Discourse what to do with the e-mail it receives.

Log into your Discourse forum as Admin and navigate to the Admin panel’s Site Settings, then click the Email tab. (forum.example.com/admin/site_settings/category/email)
Change the following settings:
- Enable the reply by email setting
- In the reply_by_email_address field, enter replies+%{reply_key}@forum.example.com
- Enable the manual polling setting

You can automatically, without any further setup, use any address @forum.example.com as an address for category or group e-mails.

Troubleshooting

Nothing ever goes according to plan. Here’s how to figure out what went wrong.

OCI runtime create failed error running ./launcher start mail-receiver? Your hostname might be too long. Rename it using these instructions and choose a shorter name, then rebuild.
Did the e-mail even make it to mail-receiver? Run ./launcher logs mail-receiver, and look for log entries that mentions the address that the e-mail was sent from and to. If there’s none of those, then the message never even made it, and the problem is upstream. Check MX records, sending mail server logs, and firewall permissions (SMTP port 25).
Is the message stuck in the queue? Run ./launcher enter mail-receiver, then run mailq. It should report, “Mail queue is empty”. If there’s any messages in there, you’ll get the to/from addresses listed. Messages only sit around in the queue if there’s a problem delivering to Discourse itself, so exit out of the container and then check…
Did mail-receiver error out somehow? Run ./launcher logs mail-receiver | grep receive-mail and look for anything that looks like a stack trace, or basically anything other than “Recipient: <something>@forum.example.com”. Those error messages, whilst not necessarily self-explanatory, should go an awfully long way to explaining what went wrong. Look for typos in your yml file. In particular, check that DISCOURSE_MAIL_ENDPOINT URL matches your site URL, usually starting with https://.

Integrating with External nginx

If you are an advanced user and have configured external nginx such as for Add an offline page to display when Discourse is rebuilding or starting up you will find that the combination of mail-receiver and HTTPS being handled in external nginx requires slightly different handling to enable SSL for email over TLS. Here are example containers/mail-receiver.yml snippets that work with the recommended configuration for external nginx with letsencrypt certificates:

  POSTCONF_smtpd_tls_key_file:  /letsencrypt/live/=DOMAIN=/privkey.pem
  POSTCONF_smtpd_tls_cert_file:  /letsencrypt/live/=DOMAIN=/fullchain.pem

…

volumes:
  - volume:
      host: /var/discourse/shared/mail-receiver/postfix-spool
      guest: /var/spool/postfix
# uncomment to support TLS
  - volume:
      host: /etc/letsencrypt/
      guest: /letsencrypt

Note that you can’t export as a volume only /etc/letsencrypt/live because the actual files are symlinks into ../../archive/... and those won’t resolve if you are more specific in the volume specification.

Prevent outgoing host email from interfering (Postfix)

If you have (or want) automated messages from your host server (via Postfix), the mail-receiver will conflict because it needs port 25 to operate. One solution is to disable the host Postfix from listening on port 25:

nano /etc/postfix/master.cf

and comment out the line that looks like this:

smtp inet n - y - - smtpd

Then service postfix reload. You may also need to restart the mail-receiver container.

With both the host Postfix and the mail-receiver running, do netstat -tulpn | grep :25 to confirm that docker-proxy is using port 25.

Block unwanted domains from sending to you

To stop email from unwanted domains from even reaching your Discourse, your mail-receiver.yml should look something like this:

  DISCOURSE_API_USERNAME: system

  POSTCONF_smtpd_sender_restrictions: 'texthash:/etc/postfix/shared/sender_access'

volumes:
  - volume:
      host: /var/discourse/shared/mail-receiver/postfix-spool
      guest: /var/spool/postfix
  - volume:
      host: /var/discourse/shared/mail-receiver/etc
      guest: /etc/postfix/shared
# uncomment to support TLS
#  - volume:
#      host: /var/discourse/shared/standalone/letsencrypt
#      guest: /letsencrypt

Then create /var/discourse/shared/mail-receiver/etc path, and within it create a sender_access file containing the domains to reject, like this:

qq.com REJECT
163.com REJECT

Rebuild and you’re golden!

DMARC support

DMARC support has been enabled by default in the discourse/mail-receiver:release image to more strongly validate incoming email. This is enabled since the timestamped image discourse/mail-receiver:20240720054629.

This functionality can be toggled via the INCLUDE_DMARC docker environment variable. If a more permissive incoming mail server configuration is preferred, set that environment variable to false and rebuild the image.

The last version without DMARC support is discourse/mail-receiver:20211208001915.

Be sure to secure the new container configuration file with: chmod o-rwx containers/mail-receiver.yml. If you don’t you’ll get prompted to do this when you bootstrap the container.
When creating the API key, I selected “All Users” and “Global” scope. I don’t know if a more restricted key would work.
The sample mail-receiver.yml file has quite different TLS settings so you’ll want to use the instructions here rather than tying to edit the sample.
It also includes a smtpd_tls_security_level setting that I uncommented. I didn’t do any research to figure out if it’s needed or if I’d be better off with a different setting then “may”.
If you want to set up an email for a particular category, that can be done in /c/{category-name}/edit/settings. (This is useful if you want to make a sorta mailing list category.) For a group, you can set up an email address in /g/{group-name}/manage/interaction.

I dunno if any of this will help others, but it would have helped me.

md-misko · January 9, 2024, 7:10am

You really should use a least privilege api key, these settings definitely work:

User Level: All users
Scope: Granular
email—receive emails

Carleas · March 18, 2024, 6:16pm

As far as I can tell my set up is right, but there’s no record of any of the emails in Discourse.

The emails show up in the log like this:

Mar 18 17:20:41 [myserver]-mail-receiver postfix/smtpd[122]: NOQUEUE: reject: RCPT from [XXX].google.com[XXX.XX.XXX.XXX]: 554 5.7.1 <test004@www.[mysite].com>: Recipient address rejected: Mail to this address is not accepted. Check the address and try to send again?; from=<[sender]@gmail.com> to=<test004@www.[mysite].com> proto=ESMTP helo=<[XXX].google.com>
Mar 18 17:20:42 [myserver]-mail-receiver postfix/smtpd[122]: disconnect from [XXX].google.com[XXX.XX.XXX.XXX] ehlo=2 starttls=1 mail=1 rcpt=0/1 bdat=0/1 quit=1 commands=5/7

And I get a rejection notification at the sending address as well. Nothing remains in the queue and there aren’t any trace errors in the logs. I triple checked that the URLs all match, and the API settings page shows that the key is being used. But the rejected email list in the admin panel stays empty.

Any suggestions?

mpalmer · March 19, 2024, 5:14am

The error suggests that MAIL_DOMAIN is not set to www.[mysite].com, or else there’s no category or group that is configured to receive email sent to test004@www.[mysite].com.

Carleas · March 19, 2024, 1:33pm

Thanks for your reply. I’ve checked MAIL_DOMAIN every way I can imagine, I’ve tried every combination of MAIL_DOMAIN value and destination email address. What value in the Discourse setup is it checked against e.g. DISCOURSE_HOSTNAME, DISCOURSE_SMTP_DOMAIN, something else?

I’m a bit confused by your second suggestion given this line in the OP:

Shouldn’t rejections show up even before Discourse is set up to do anything with them? Bounces aren’t showing up either, I’ve tested using the method recommended here: Configure VERP to handle bouncing e-mails. No trace of anything on admin/email.

Is there a log in either container that shows (or could be configured to show) more information about the interaction between mail-receiver and app are interacting?

Simon_Manning · March 19, 2024, 3:53pm

There are two major types of rejections, ones that happens early and decide whether or not the email should be passed to Discourse’s EmailReceiver and ones during Discourse’s email processing.

In my experience the former does not appear in the Discourse logs, meaning most (all?) email related rejections (failed DMARC, wrong address, etc.) don’t appear there. The ones that do appear are things like email too short, user not allowed to post, etc.

I’m not sure if that’s something that has changed since that paragraph was written but the above has been my experience since ~2.5 years ago when I set it up.

If I send an email to test-reject@[my-instance], I get a generic bounce notice from my email provider (not mail-receiver / Discourse) telling me that the recipient address was rejected. This is because mail-receiver is rejecting it during the SMTP interaction.

Bounces and VERP are related to emails your Discourse instance is sending rather than receiving, e.g. to automatically stop sending notification emails to an address that is consistently bouncing. They don’t relate to mail-receiver.

My suspicion is that your quote from the guide has thrown you off and actually everything is probably working correctly. Sending to some-random-address@MAIL_DOMAIN will not be accepted and not show up in the rejections so that’s not a very useful test on its own (besides making sure mail-receiver is receiving emails, which you have seen it is).

Navigate to an existing category or create a new one, open its configuration and go to the settings tab. Near the bottom you will find custom incoming email address. Set this to something@MAIL_DOMAIN, e.g. the test004 address you tried previously, save and then try sending to that address.

That should get past mail-receiver so you should then either see a new post created in the category or a rejection in Discourse.

Carleas · March 19, 2024, 4:51pm

Thank you, this is very helpful clarification, I’ll set it up and test it to see.

For bounces, I was again confused by the OP, since bounces are the first point in the list of reasons why you might want to follow this guide.

So even with this set up, and even if I’ve taken down my Mailgun MX records, I still need to configure VERP on that side to catch bounces etc.? Well shoot, I thought direct-delivery was a workaround to my issues with Mailgun webhooks, looks like I’ll need to start troubleshooting that again.

Simon_Manning · March 19, 2024, 6:12pm

Oh apologies, you’re right that it says you can use mail-receiver for bounces, I’m not very familiar with how that works.

My mail-receiver does not receive bounces but I am using Mailgun webhooks, perhaps Mailgun is changing the envelope sender so it receives the bounces if webhooks are enabled. (I.e. if webhooks were disabled, perhaps my mail-receiver would be receiving the bounces instead.)

mpalmer · March 19, 2024, 11:20pm

Yeah, I’m pretty sure that’s now inaccurate, since fast-rejection got implemented in… (checks git log) May 2017.

Without seeing your actual config, including the Discourse group/category config, it’s really hard to say what’s going wrong. At least 80% of the time it’s a typo somewhere, though; ask a colleague (doesn’t need to be someone deeply technical) to look at it, and they’ll probably spot where you put an l instead of an i in about five seconds. My wife does that for me on the regular.

It is. With direct delivery, your outgoing mail provider doesn’t need get involved at all for incoming mail. Everything, be it a new topic, a reply, or a bounce, should all go straight to mail-receiver (and thence to Discourse for processing).

pfaffman · March 19, 2024, 11:24pm

I’m pretty sure that’s what happened to me with this exact issue last week. I finally copied another YML file from somewhere else and it worked.

It did seem strange, though, Matt. I looked in the postfix files and they too looked right, but it said the hostname didn’t match. I swear I copy/pasted it, but, maybe I made the mistake of thinking that I could type.

mpalmer · March 19, 2024, 11:38pm

It’s good that AI voice recognition will fix all that for us any day now.

Carleas · March 20, 2024, 2:17pm

You were correct, setting up an email for a category and sending email there worked as expected, so I was just beating my head against wall because the rejections were silent.

I’m glad I know now, and I hope the guide is updated, though personally I’d prefer if it worked as the guide describes. For example, if users are trying to send email to some address and it’s failing, might help me either let them know or realize there’s demand for communicating with a category or group by email. Seems like without that, there’s no easy way to see those emails.

This still isn’t working as expected. I got the webhooks working, so I can see several bounces, but I know they are from Mailgun webhooks because they have the issue described here: “Discourse::NotFound” error when click “Email Type” field on admin/email/bounced

I don’t really understand how Mailgun is getting the bounces in the first place, since I don’t have any MX records pointing at their servers, I assume they’re setting a return path as they send the outgoing email?

And I see the bounces in mail-receiver logs, but they aren’t getting to app. It looks like they’re getting silently rejected. Here’s a line in the logs that I can connect with a bounce received through the webhooks:

NOQUEUE: reject: RCPT from mail-[id1].outbound.protection.outlook.com[XX.XX.XX.XX]: 450 4.7.1 <bounce+[id2]-[email]=[address].com@www.[mydomain].com>: Recipient address rejected: Internal error, API request failed; from=<> to=<bounce+[id#]-[email]=[address].com@www.[mydomain].com> proto=ESMTP helo=<[id3].outbound.protection.outlook.com>

Do I need to add bounce+{%something}@www.mydomain.com as a whitelisted address somewhere so that they get through?

mpalmer · March 20, 2024, 10:09pm

Yes, they’re probably rewriting the return path (aka “envelope from”) when the outgoing email passes through their servers. There’s probably a setting somewhere to turn that off, but I’ve not used mailgun, so I can’t say for sure (or where such a setting would be).

OK, that is an error between mail-receiver and Discourse. There should be a line in the logs soon before that that starts “Failed to GET smtp_should_reject answer” that will tell you more about what failed and why, and that should correlate with an error message of some sort in the Discourse logs.

Carleas · March 22, 2024, 3:11pm

Mar 21 17:02:21 discourse-smtp-fast-rejection[1149]: Failed to GET smtp_should_reject answer from https://www.mydomain.com/admin/email/smtp_should_reject.json: 400

Could it be related to the null sender, from=<>? I don’t see anything in the logs about it. Is the 400 saying smtp_should_reject.json doesn’t exist?

mpalmer · March 24, 2024, 7:16am

If that HTTP resource didn’t exist, it’d be a 404, not a 400. I don’t think a null sender should be a problem, because that’s how all bounces will be delivered. An incorrect API key should (I think) return a 403, but I can’t say that for absolute certain off the top of my head, so that’s probably at least worth checking, just in case. If the Discourse logs are giving no indications as to why the request was bad, I’m afraid you’re probably in for a session of painful debugging – I don’t have a mail-receiver-enabled system to easily play with at the current time. It’d be a consulting job for me to get to the bottom of what’s going on and fix it for you, I’m afraid.

Carleas · March 24, 2024, 8:53pm

For now, it doesn’t seem to be breaking anything, I got bounces working with webhooks and most bounces don’t generate an email (another topic mentioned this stackoverflow answer, which matches what I’m seeing). And reply-by-email is working as expected too. Whatever the failure is, it’s rare and not breaking normal function.

I’ll keep an eye on it just in case, and report back if I figure anything out that might be useful for others. Thanks again for your help!

pfaffman · April 3, 2024, 5:25pm

@JammyDodger Could this be renamed something that will allow searching for “mail-receiver” to find it? I’ve mostly not been able to find this topic without making several retries since three years ago when “straightforward” was removed from the title.

Topic		Replies	Views
Mail-receiver won’t deliver mail to Discourse Installation mail-receiver	16	357	January 26, 2023
Email receive issue Bug mail-receiver	24	240	July 19, 2025
Set up mail-receiver, but now site won't send any emails? Installation email , mail-receiver	11	772	September 30, 2023
Mail-receiver relay access denied Installation mail-receiver	21	1735	January 26, 2025
Customize direct-delivery Postfix configuration Sysadmins how-to , advanced-setup	18	4869	November 5, 2024