Configure direct-delivery incoming email for self-hosted sites

Discourse is all about enabling civilized discussion. While plenty of people like a web interface, e-mail is still the “hub” of many people’s online lives. That’s why sending e-mail is so important, and when you’re sending e-mail, you really want to be able to receive it, too. There are several reasons why:

  • If e-mails “bounce” (they can’t be delivered for some reason), you need to know about that. Repeatedly sending e-mails that bounce will get your e-mails flagged as spam. Receiving e-mail bounces allows you to disable sending to non-existent addresses.
  • Allowing people to reply to posts via e-mail can significantly improve engagement, as people can reply straight away from their mail client, even if they’re not able to visit the forum at that moment.
  • Letting people post new topics, or send PMs, via e-mail has similar benefits to engagement. In addition, you can use Discourse to handle e-mail for a group, such as an e-mail-based support channel (which is how Discourse’ own e-mail support is handled).

Delivering e-mail directly into your Discourse forum, rather than setting up POP3 polling, has a number of benefits:

  • No need to deal with gmail or another provider’s foibles;
  • You have more control over the e-mail addresses that people use to send posts; and
  • There are no delays in delivery – no more waiting for the next polling run to see new posts appear!

This howto is all about getting that hawtness into your forum.

Overview

This procedure creates a new container on your Discourse server, alongside the typical app container, which receives e-mail and forwards it into Discourse for processing. It supports all e-mail processes: handling bounces, replies, and new topic creation. Any self-hosted Discourse forum using our supported installation process can make use of this procedure to get easy, smooth-flowing incoming e-mail.

Container Setup

We’re going to get the mail-receiver container up and running on the server that’s already running your Discourse instance. There’s no need for a separate droplet just to handle mail – the whole container only takes about 5MB of memory!

So, start off by logging into your Discourse server, and becoming root via sudo:

ssh ubuntu@192.0.2.42
sudo -i

Now, go to your /var/discourse directory and create a new mail-receiver.yml container definition from the sample conveniently provided:

cd /var/discourse
git pull
cp samples/mail-receiver.yml containers/

Since every site is unique, open containers/mail-receiver.yml in your preferred text editor and change the MAIL_DOMAIN, DISCOURSE_MAIL_ENDPOINT, and DISCOURSE_API_KEY variables to suit your site. (If you are an advanced user and know that you are using nginx outside your container, see below for additional configuration for external nginx.)

:bulb: If you use the default mail endpoint (/admin/email/handle_mail), we suggest using the receive_email API key scope to provide an extra layer of security.

If you’re not sure what your favourite text editor is, try nano:

nano containers/mail-receiver.yml

Use Ctrl-X to exit (say “Yes” to “Do you want to save changes?”, or all your work will be for nothing).

Now, do an initial build of the container, and fire it up!

./launcher bootstrap mail-receiver
./launcher start mail-receiver

To check everything’s OK, take a peek in the logs:

./launcher logs mail-receiver

The last line printed should look rather a lot like this:

<22>Aug 31 04:14:31 postfix/master[1]: daemon started -- version 3.1.1, configuration /etc/postfix

If so, all is well, and you can go on to then next step.

DNS Setup

In order for everyone else on the Internet to know where to deliver mail, you must create an MX record for your forum. The exact details of how to do this vary by DNS provider, but in general, the procedure should be very similar to how you setup the DNS records for your forum in the first place, except that instead of creating an A (or “Address”) record, you’re creating an MX (or “Mail eXchange”) record. If your forum is at forum.example.com, and you set MAIL_DOMAIN to forum.example.com in the mail-receiver.yml, then the DNS record should look like this:

  • DNS Name: forum.example.com (this is the MAIL_DOMAIN)
  • Type: MX
  • Priority: 10
  • Value: forum.example.com (this is the domain of your forum)

To make sure the DNS is setup correctly, use a testing site such as http://mxtoolbox.com/ to look up the MAIL_DOMAIN you configured, and make sure it’s pointing to where you expect.

You can also now try sending an e-mail to nobody@forum.example.com. While Discourse won’t do anything useful with it yet, the e-mail you sent should show up in your admin panel under “Emails”, “Rejected” in a matter of seconds. If that happens, you’re definitely ready to proceed.

Note: outbound email providers like mailgun may ask you to add MX records pointing to their servers. You want to remove these so the MX records for your forum only point to your forum’s domain name. SPF and DKIM records must still point to your outbound email provider servers so you can send email.

Discourse Configuration

Now e-mail is being fed into Discourse, it’s time to explain to Discourse what to do with the e-mail it receives. For the most part, this is identical to setting up incoming e-mail via POP3 polling, with a few minor exceptions:

  • Enable the “manual polling” setting, rather than “pop3 polling”; and
  • You can automatically, without any further setup, use any address @forum.example.com as an address for category or group e-mails.

Troubleshooting

Nothing ever goes according to plan. Here’s how to figure out what went wrong.

  1. OCI runtime create failed error running ./launcher start mail-receiver? Your hostname might be too long. Rename it using these instructions and choose a shorter name, then rebuild.
  2. Did the e-mail even make it to mail-receiver? Run ./launcher logs mail-receiver, and look for log entries that mentions the address that the e-mail was sent from and to. If there’s none of those, then the message never even made it, and the problem is upstream. Check MX records, sending mail server logs, and firewall permissions (SMTP port 25).
  3. Is the message stuck in the queue? Run ./launcher enter mail-receiver, then run mailq. It should report, “Mail queue is empty”. If there’s any messages in there, you’ll get the to/from addresses listed. Messages only sit around in the queue if there’s a problem delivering to Discourse itself, so exit out of the container and then check…
  4. Did mail-receiver error out somehow? Run ./launcher logs mail-receiver | grep receive-mail and look for anything that looks like a stack trace, or basically anything other than “Recipient: <something>@forum.example.com”. Those error messages, whilst not necessarily self-explanatory, should go an awfully long way to explaining what went wrong. Look for typos in your yml file. In particular, check that DISCOURSE_MAIL_ENDPOINT URL matches your site URL, usually starting with https://.

Integrating with External nginx

If you are an advanced user and have configured external nginx such as for Add an offline page to display when Discourse is rebuilding or starting up you will find that the combination of mail-receiver and HTTPS being handled in external nginx requires slightly different handling to enable SSL for email over TLS. Here are example containers/mail-receiver.yml snippets that work with the recommended configuration for external nginx with letsencrypt certificates:

  POSTCONF_smtpd_tls_key_file:  /letsencrypt/live/=DOMAIN=/privkey.pem
  POSTCONF_smtpd_tls_cert_file:  /letsencrypt/live/=DOMAIN=/fullchain.pem

volumes:
  - volume:
      host: /var/discourse/shared/mail-receiver/postfix-spool
      guest: /var/spool/postfix
# uncomment to support TLS
  - volume:
      host: /etc/letsencrypt/
      guest: /letsencrypt

Note that you can’t export as a volume only /etc/letsencrypt/live because the actual files are symlinks into ../../archive/... and those won’t resolve if you are more specific in the volume specification.

Prevent outgoing host email from interfering (Postfix)

If you have (or want) automated messages from your host server (via Postfix), the mail-receiver will conflict because it needs port 25 to operate. One solution is to disable the host Postfix from listening on port 25:

nano /etc/postfix/master.cf

and comment out the line that looks like this:

smtp inet n - y - - smtpd

Then service postfix reload. You may also need to restart the mail-receiver container.

With both the host Postfix and the mail-receiver running, do netstat -tulpn | grep :25 to confirm that docker-proxy is using port 25.

Block unwanted domains from sending to you

To stop email from unwanted domains from even reaching your Discourse, your mail-receiver.yml should look something like this:

  DISCOURSE_API_USERNAME: system

  POSTCONF_smtpd_sender_restrictions: 'texthash:/etc/postfix/shared/sender_access'

volumes:
  - volume:
      host: /var/discourse/shared/mail-receiver/postfix-spool
      guest: /var/spool/postfix
  - volume:
      host: /var/discourse/shared/mail-receiver/etc
      guest: /etc/postfix/shared
# uncomment to support TLS
#  - volume:
#      host: /var/discourse/shared/standalone/letsencrypt
#      guest: /letsencrypt

Then create /var/discourse/shared/mail-receiver/etc path, and within it create a sender_access file containing the domains to reject, like this:

qq.com REJECT
163.com REJECT

Rebuild and you’re golden!

Further Reading

83 Likes

Hi, Thank you for your work to make direct-delivery so much easier!

When building the container, I get one error message that it can’t find ruby and one warning on locales
Full output:

/var/discourse$ sudo ./launcher bootstrap mail-receiver
x86_64 arch detected.
release: Pulling from discourse/mail-receiver
Digest: sha256:25abf0d6f1e4324fb0f6c6ec5839b20b6d6c685bb55eae63146065991ae5c9f4
Status: Image is up to date for discourse/mail-receiver:release
docker.io/discourse/mail-receiver:release
ERROR:  Can't find Ruby library file or shared library pups
cd /pups && /pups/bin/pups --stdin
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
sha256:a97b69f89da4f9cb1441aca4f19c1cca8f717a54fdc75af54f4554f2c08da01e
516dc1c4fb59804256373f8a29aaf19d937e4d41b068dfa45d499c8d6b41aee5
Successfully bootstrapped, to startup use ./launcher start mail-receiver

Is this safe to ignore or a bug?
Using latest commit 8c40388ae4dabf1c0052ddd8eab4c72dbf7e0ca3 on branch main.

I suspect it’s OK. Start it up and see if it works.

1 Like

4 posts were split to a new topic: Incoming mail server has certificate error

I’ve 2 containers setup for my self-hosted website, nginx is setup outside the container.
And 3rd container for this mail setup is created now.
But, can you pls tell from where I can copy the above said variables of my site to be entered into new mail.yml?

You need a third container as descibed below this:

1 Like

Container is already created (it only needed one command). But then as described in above/initial post(s), I’ve to fill 2-3 values/variables in my mail.yml. I was in doubt from where do I copy the value for the same?

I think you’re asking how to find what values you should be setting those variables to. This is mostly explained in the comments around the variables in the file itself and besides the API key, probably you just need to replace discourse.example.com with your own forum’s domain name.

Here is the sample file in GitHub in case that makes it easier to read everything: discourse_docker/mail-receiver.yml at main · discourse/discourse_docker · GitHub

The API key comment directs you to the API section of your forum’s admin interface but it doesn’t talk about what kind should be created. Assuming you haven’t renamed your system user, it should look like this when creating the key: (enter your own description)

Then below that, find and tick receive emails:
image

After saving that, you will be given the API key needed for DISCOURSE_API_KEY.

4 Likes

My app.yml:

  ## Where e-mail to your forum should be sent.  In general, it's perfectly fine
  ## to use the same domain as the forum itself here.
  MAIL_DOMAIN: bathindahelper.com

# uncomment these (and the volume below!) to support TLS 
#  POSTCONF_smtpd_tls_key_file:  /letsencrypt/discourse.example.com/discourse.example.com.key
#  POSTCONF_smtpd_tls_cert_file:  /letsencrypt/discourse.example.com/fullchain.cer
#  POSTCONF_smtpd_tls_security_level: may


  ## The URL of the mail processing endpoint of your Discourse forum.
  ## This is simply your forum's base URL, with `/admin/email/handle_mail`
  ## appended.  Be careful if you're running a subfolder setup -- in that case,
  ## the URL needs to have the subfolder included!
  DISCOURSE_MAIL_ENDPOINT: 'https://bathindahelper.com/admin/email/handle_mail'

  ## The master API key of your Discourse forum.  You can get this from
  ## the "API" tab of your admin panel.
  DISCOURSE_API_KEY: abcdefghijklmnop

I changed 2 values: mail domain (doubt: don’t I prefix my domain name with some word, like ‘mail’ or only ‘m’)? And mail endpoint (doubt: would including only my actual domain name and leaving all the subfolder path up to ‘handle_mail’ as they’re in the sample file, would be ok?)
I am afraid that some more needs to be done here.

And then, I couldn’t find and copy my own site’s api keys (if I click on ‘All Api Keys’, there is only this one api key to be found). Giving screen shot:


And last, I’m in doubt that while setting my dns settings mx record here:
image

can I give 2 diff values to 'Dns Name and to ‘value’?
E.g.

Dns Name: bathindahelper.com
Value: mail.bathindahelper.com

This is one reason that you wanted your site to be at www.bathindahelper.com.

You appear to be using bathindahelper.com to receive mail already, so you need to do something now complicated.

Set the

You’ll need to configure the MAIL_DOMAIN in your mail_receiver.yml to mail.bathindahelper.com. Yes, you can have different values for the A record and the mx record. You already have two mx records for bathindahelper.com you don’t want to change those unless you don’t want to use that mailbox any more.

Add a dns A record for mail.bathindahelper.com with 5.161.135.164.

And an mx record for mail.bathindahelper.com pointing to mail.bathindahelper.com with priority 10.

Generate a new api key and paste it into the file.

./launcher rebuild mail_receiver.

I think that should do it, though it’s the middle of the night and I’m typing on my phone.

2 Likes

I made this mistake, and left discourse.example.com in the mail-receiver.yml file.

I have fixed this now, but mail-receiver doesn’t seem to be ‘getting’ the new details.
How do I ‘reset’ mail-receiver (e.g. what is the equivalent command to ./launcher rebuild app?).

Edit: I didn’t read the previous post closely enough, the command is ./launcher rebuild mail_receiver.

1 Like

I’m running into a further issue now where mail-receiver won’t deliver mail to Discourse — I’ve tried searching for help, but no luck.

Logs:

Starting Postfix
Dec 14 03:12:32 forum-mail-receiver postfix/master[1]: daemon started -- version 3.5.6, configuration /etc/postfix
Dec 14 03:15:47 forum-mail-receiver postfix/smtpd[113]: connect from mail-pl1-f169.google.com[209.85.214.169]
Dec 14 03:15:47 forum-mail-receiver postfix/smtpd[113]: 821CB37A659: client=mail-pl1-f169.google.com[209.85.214.169]
Dec 14 03:15:47 forum-mail-receiver postfix/cleanup[120]: 821CB37A659: message-id=<602f2194be912e92b969eacf5eac26e2@frontapp.com>
Dec 14 03:15:47 forum-mail-receiver postfix/qmgr[98]: 821CB37A659: from=<[my personal email address]>, size=4086, nrcpt=1 (queue active)
<23>Dec 14 03:15:47 receive-mail[122]: Recipient: nobody@[my forum URL]Dec 14 03:16:20 forum-mail-receiver postfix/smtpd[113]: disconnect from mail-pl1-f169.google.com[209.85.214.169] ehlo=1 mail=1 rcpt=1 bdat=1 quit=1 commands=5
<19>Dec 14 03:16:47 receive-mail[122]: Failed to POST the e-mail to [my forum URL]/admin/email/handle_mail: execution expired (Net::OpenTimeout)<19>Dec 14 03:16:47 receive-mail[122]:   /usr/lib/ruby/2.7.0/net/http.rb:960:in `initialize'
  /usr/lib/ruby/2.7.0/net/http.rb:960:in `open'
  /usr/lib/ruby/2.7.0/net/http.rb:960:in `block in connect'
  /usr/lib/ruby/2.7.0/timeout.rb:105:in `timeout'
  /usr/lib/ruby/2.7.0/net/http.rb:958:in `connect'
  /usr/lib/ruby/2.7.0/net/http.rb:943:in `do_start'
  /usr/lib/ruby/2.7.0/net/http.rb:932:in `start'
  /usr/lib/ruby/2.7.0/net/http.rb:1483:in `request'
  /usr/local/lib/site_ruby/mail_receiver/discourse_mail_receiver.rb:43:in `process'
  /usr/local/bin/receive-mail:13:in `<main>'Dec 14 03:16:47 forum-mail-receiver postfix/pipe[121]: 821CB37A659: to=<nobody@[my forum URL]>, relay=discourse, delay=60, delays=0.17/0.01/0/60, dsn=4.3.0, status=deferred (temporary failure)
Dec 14 03:17:32 forum-mail-receiver postfix/qmgr[98]: 7C67437A663: from=<[my personal email address]>, size=4093, nrcpt=1 (queue active)

Any idea what could be causing this?

The mail-receiver.yml file is valid, and I’ve checked for typos:
image

This is my API key scope:

The mail is making it into mail-receiver, but it is just sitting in mailq:

Alternately, is there any way to delete the mail-receiver container entirely and start over?

The issue may be that you don’t have the API key set

Thanks for the response @pfaffman… it is definitely set in my mail-receiver.yml config. Should it be in quotes?

 (Net::OpenTimeout)

That’s your issue. The mail receiver can’t access your forum URL. So either you have the wrong one somehow or there’s some network issue in docker between the mail receiver and your forum., I think.

How can I troubleshoot further?

ping forum.[mydomain].co.nz

from inside mailq shows:

64 bytes from [ip].vultrusercontent.com ([ip]): icmp_seq=1 ttl=64 time=0.113 ms
64 bytes from [ip].vultrusercontent.com ([ip]): icmp_seq=2 ttl=64 time=0.074 ms
64 bytes from [ip].vultrusercontent.com ([ip]): icmp_seq=3 ttl=64 time=0.069 ms

and so on, showing that a connection is successful.
forum.[mydomain].co.nz is where the forum is hosted, and this same URL is used in MAIL_DOMAIN and DISCOURSE_MAIL_ENDPOINT.

Looking at the mail-receiver.yml settings closer, am I missing quotes or https:// anywhere that should have it?

## this is the incoming mail receiver container template
##
## After making changes to this file, you MUST rebuild
## /var/discourse/launcher rebuild mail-receiver
##
## BE *VERY* CAREFUL WHEN EDITING!
## YAML FILES ARE SUPER SUPER SENSITIVE TO MISTAKES IN WHITESPACE OR ALIGNMENT!
## visit http://www.yamllint.com/ to validate this file as needed

base_image: discourse/mail-receiver:release
update_pups: false

expose:
  - "25:25"   # SMTP

env:
  LC_ALL: en_US.UTF-8
  LANG: en_US.UTF-8
  LANGUAGE: en_US.UTF-8

  ## Where e-mail to your forum should be sent.  In general, it's perfectly fine
  ## to use the same domain as the forum itself here.
  MAIL_DOMAIN: forum.[domain].co.nz
# uncomment these (and the volume below!) to support TLS
#  POSTCONF_smtpd_tls_key_file:  /letsencrypt/discourse.example.com/discourse.example.com.key
#  POSTCONF_smtpd_tls_cert_file:  /letsencrypt/discourse.example.com/fullchain.cer
#  POSTCONF_smtpd_tls_security_level: may


  ## The URL of the mail processing endpoint of your Discourse forum.
  ## This is simply your forum's base URL, with `/admin/email/handle_mail`
  ## appended.  Be careful if you're running a subfolder setup -- in that case,
  ## the URL needs to have the subfolder included!
  DISCOURSE_MAIL_ENDPOINT: 'https://forum.[domain].co.nz/admin/email/handle_mail'

  ## The master API key of your Discourse forum.  You can get this from
  ## the "API" tab of your admin panel.
  DISCOURSE_API_KEY: 639[rest of API key]884ef

  ## The username to use for processing incoming e-mail.  Unless you have
  ## renamed the `system` user, you should leave this as-is.
  DISCOURSE_API_USERNAME: system

volumes:
  - volume:
      host: /var/discourse/shared/mail-receiver/postfix-spool
      guest: /var/spool/postfix
# uncomment to support TLS
#  - volume:
#      host: /var/discourse/shared/standalone/letsencrypt
#      guest: /letsencrypt

Are you running the ping inside the container, i.e. after first running ./launcher enter mail-receiver?

It’s also worth noting that ping (typically ICMP) is different from connecting to http/https (TCP) and may behave differently depending on lots of factors in network configuration.

I would try using curl after entering the container to see if it can connect to your forum over https, e.g.

cd /var/discourse
./launcher enter mail-receiver
curl -v https://forum.[domain].co.nz

If it’s working, it’ll print a bunch of HTML. If it’s not, it’ll show some error and -v will make it print lots of information along the way which might help reveal why it failed.

If it does fail, it is also worth trying to run the same curl command outside of the container to identify whether it’s specific to the container or to the host system in general.

3 Likes

Thanks @Simon_Manning, your help is very much appreciated! I didn’t know that connections through ping aren’t necessarily the same as connections through curl.

I was running ping inside the container, and it succeeded.

I followed your instructions and ran curl inside the container, and it failed:

root@forum:/var/discourse# ./launcher enter mail-receiver
x86_64 arch detected.
WARNING: containers/mail-receiver.yml file is world-readable. You can secure this file by running: chmod o-rwx containers/mail-receiver.yml
bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
root@forum-mail-receiver:/# curl -v https://forum.[domain].co.nz
*   Trying [IPv4 address]:443...
*   Trying [IPv6 address]:443...
* Immediate connect fail for [IPv6 address]: Cannot assign requested address
* connect to [IPv4 address] port 443 failed: Connection timed out
* Failed to connect to forum.[domain].co.nz port 443: Connection timed out
* Closing connection 0
curl: (28) Failed to connect to forum.[domain].co.nz port 443: Connection timed out

Then I ran exit and then curl again, and got:

root@forum:/var/discourse# curl -v https://forum.[domain].co.nz
*   Trying 127.0.1.1:443...
* Connected to forum.[domain].co.nz (127.0.1.1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
...and so on.

It looks like it’s specific to the container, and not to the host system — any ideas?

I have also opened a support ticket with Vultr (VPS provider for this instance) to see if it is a problem on their end!

Docker creates virtual networks for the containers and in the absence of specifying one, containers will use the default network. This default network does not allow container-to-container communication.

Usually this is fine for mail-receiver because your Discourse container will have port 443 exposed outside of that network and when mail-receiver tries to connect to 1.2.3.4, it will leave the Docker network. The host system (or some networking further out) will realise it just needs to come back again and it will end up entering the Discourse container from outside.

Two possibilities come to mind. One is that mail-receiver is somehow aware of the Discourse container IP when looking up the domain name and therefore an intra-container connection is being blocked. I think this is unlikely.

The other is that a firewall on the host system is blocking connections from leaving a container and entering a different one. Vultr may use default firewall rules that cause this or I also vaguely remember that Docker installs some rules in UFW by default, so that could be related if that’s used.

2 Likes