Configure direct-delivery incoming email for self-hosted sites with Mail-Receiver

Documentation Self-Hosting
, , ,
1

Discourse is all about enabling civilized discussion. While plenty of people like a web interface, e-mail is still the “hub” of many people’s online lives. That’s why sending e-mail is so important, and when you’re sending e-mail, you really want to be able to receive it, too. There are several reasons why:

  • If e-mails “bounce” (they can’t be delivered for some reason), you need to know about that. Repeatedly sending e-mails that bounce will get your e-mails flagged as spam. Receiving e-mail bounces allows you to disable sending to non-existent addresses.
  • Allowing people to reply to posts via e-mail can significantly improve engagement, as people can reply straight away from their mail client, even if they’re not able to visit the forum at that moment.
  • Letting people post new topics, or send PMs, via e-mail has similar benefits to engagement. In addition, you can use Discourse to handle e-mail for a group, such as an e-mail-based support channel (which is how Discourse’ own e-mail support is handled).

Delivering e-mail directly into your Discourse forum, rather than setting up POP3 polling, has a number of benefits:

  • No need to deal with gmail or another provider’s foibles;
  • You have more control over the e-mail addresses that people use to send posts; and
  • There are no delays in delivery – no more waiting for the next polling run to see new posts appear!

This howto is all about getting that hawtness into your forum.

Overview

This procedure creates a new container on your Discourse server, alongside the typical app container, which receives e-mail and forwards it into Discourse for processing. It supports all e-mail processes: handling bounces, replies, and new topic creation. Any self-hosted Discourse forum using our supported installation process can make use of this procedure to get easy, smooth-flowing incoming e-mail.

Container Setup

We’re going to get the mail-receiver container up and running on the server that’s already running your Discourse instance. There’s no need for a separate droplet just to handle mail – the whole container only takes about 5MB of memory!

So, start off by logging into your Discourse server, and becoming root via sudo:

ssh ubuntu@192.0.2.42
sudo -i

Now, go to your /var/discourse directory and create a new mail-receiver.yml container definition from the sample conveniently provided:

cd /var/discourse
git pull
cp samples/mail-receiver.yml containers/

Since every site is unique, open containers/mail-receiver.yml in your preferred text editor and change the MAIL_DOMAIN, DISCOURSE_MAIL_ENDPOINT, and DISCOURSE_API_KEY variables to suit your site. (If you are an advanced user and know that you are using nginx outside your container, see below for additional configuration for external nginx.)

:bulb: If you use the default mail endpoint (/admin/email/handle_mail), we suggest using the receive_email API key scope to provide an extra layer of security.

If you’re not sure what your favourite text editor is, try nano:

nano containers/mail-receiver.yml

Use Ctrl-X to exit (say “Yes” to “Do you want to save changes?”, or all your work will be for nothing).

Now, do an initial build of the container, and fire it up!

./launcher bootstrap mail-receiver
./launcher start mail-receiver

To check everything’s OK, take a peek in the logs:

./launcher logs mail-receiver

The last line printed should look rather a lot like this:

<22>Aug 31 04:14:31 postfix/master[1]: daemon started -- version 3.1.1, configuration /etc/postfix

If so, all is well, and you can go on to then next step.

DNS Setup

In order for everyone else on the Internet to know where to deliver mail, you must create an MX record for your forum. The exact details of how to do this vary by DNS provider, but in general, the procedure should be very similar to how you setup the DNS records for your forum in the first place, except that instead of creating an A (or “Address”) record, you’re creating an MX (or “Mail eXchange”) record. If your forum is at forum.example.com, and you set MAIL_DOMAIN to forum.example.com in the mail-receiver.yml, then the DNS record should look like this:

  • DNS Name: forum.example.com (this is the MAIL_DOMAIN)
  • Type: MX
  • Priority: 10
  • Value: forum.example.com (this is the domain of your forum)

To make sure the DNS is setup correctly, use a testing site such as http://mxtoolbox.com/ to look up the MAIL_DOMAIN you configured, and make sure it’s pointing to where you expect.

Note: outbound email providers like mailgun may ask you to add MX records pointing to their servers. You want to remove these so the MX records for your forum only point to your forum’s domain name. SPF and DKIM records must still point to your outbound email provider servers so you can send email.

Discourse Configuration

Now e-mail is being fed into Discourse, it’s time to explain to Discourse what to do with the e-mail it receives.

  • Log into your Discourse forum as Admin and navigate to the Admin panel’s Site Settings, then click the Email tab. (forum.example.com/admin/site_settings/category/email)
  • Change the following settings:
    • Enable the reply by email setting
    • In the reply_by_email_address field, enter replies+%{reply_key}@forum.example.com
    • Enable the manual polling setting

You can automatically, without any further setup, use any address @forum.example.com as an address for category or group e-mails.

Troubleshooting

Nothing ever goes according to plan. Here’s how to figure out what went wrong.

  1. OCI runtime create failed error running ./launcher start mail-receiver? Your hostname might be too long. Rename it using these instructions and choose a shorter name, then rebuild.
  2. Did the e-mail even make it to mail-receiver? Run ./launcher logs mail-receiver, and look for log entries that mentions the address that the e-mail was sent from and to. If there’s none of those, then the message never even made it, and the problem is upstream. Check MX records, sending mail server logs, and firewall permissions (SMTP port 25).
  3. Is the message stuck in the queue? Run ./launcher enter mail-receiver, then run mailq. It should report, “Mail queue is empty”. If there’s any messages in there, you’ll get the to/from addresses listed. Messages only sit around in the queue if there’s a problem delivering to Discourse itself, so exit out of the container and then check…
  4. Did mail-receiver error out somehow? Run ./launcher logs mail-receiver | grep receive-mail and look for anything that looks like a stack trace, or basically anything other than “Recipient: <something>@forum.example.com”. Those error messages, whilst not necessarily self-explanatory, should go an awfully long way to explaining what went wrong. Look for typos in your yml file. In particular, check that DISCOURSE_MAIL_ENDPOINT URL matches your site URL, usually starting with https://.

Integrating with External nginx

If you are an advanced user and have configured external nginx such as for Add an offline page to display when Discourse is rebuilding or starting up you will find that the combination of mail-receiver and HTTPS being handled in external nginx requires slightly different handling to enable SSL for email over TLS. Here are example containers/mail-receiver.yml snippets that work with the recommended configuration for external nginx with letsencrypt certificates:

  POSTCONF_smtpd_tls_key_file:  /letsencrypt/live/=DOMAIN=/privkey.pem
  POSTCONF_smtpd_tls_cert_file:  /letsencrypt/live/=DOMAIN=/fullchain.pem

…

volumes:
  - volume:
      host: /var/discourse/shared/mail-receiver/postfix-spool
      guest: /var/spool/postfix
# uncomment to support TLS
  - volume:
      host: /etc/letsencrypt/
      guest: /letsencrypt

Note that you can’t export as a volume only /etc/letsencrypt/live because the actual files are symlinks into ../../archive/... and those won’t resolve if you are more specific in the volume specification.

Prevent outgoing host email from interfering (Postfix)

If you have (or want) automated messages from your host server (via Postfix), the mail-receiver will conflict because it needs port 25 to operate. One solution is to disable the host Postfix from listening on port 25:

nano /etc/postfix/master.cf

and comment out the line that looks like this:

smtp inet n - y - - smtpd

Then service postfix reload. You may also need to restart the mail-receiver container.

With both the host Postfix and the mail-receiver running, do netstat -tulpn | grep :25 to confirm that docker-proxy is using port 25.

Block unwanted domains from sending to you

To stop email from unwanted domains from even reaching your Discourse, your mail-receiver.yml should look something like this:

  DISCOURSE_API_USERNAME: system

  POSTCONF_smtpd_sender_restrictions: 'texthash:/etc/postfix/shared/sender_access'

volumes:
  - volume:
      host: /var/discourse/shared/mail-receiver/postfix-spool
      guest: /var/spool/postfix
  - volume:
      host: /var/discourse/shared/mail-receiver/etc
      guest: /etc/postfix/shared
# uncomment to support TLS
#  - volume:
#      host: /var/discourse/shared/standalone/letsencrypt
#      guest: /letsencrypt

Then create /var/discourse/shared/mail-receiver/etc path, and within it create a sender_access file containing the domains to reject, like this:

qq.com REJECT
163.com REJECT

Rebuild and you’re golden!

DMARC support

DMARC support has been enabled by default in the discourse/mail-receiver:release image to more strongly validate incoming email. This is enabled since the timestamped image discourse/mail-receiver:20240720054629.

This functionality can be toggled via the INCLUDE_DMARC docker environment variable. If a more permissive incoming mail server configuration is preferred, set that environment variable to false and rebuild the image.

The last version without DMARC support is discourse/mail-receiver:20211208001915.

Further Reading

Last edited by @kelv 2024-07-22T03:53:51Z

Check documentPerform check on document:
90 Likes