Straightforward direct-delivery incoming mail

Discourse is all about enabling civilized discussion. While plenty of people like a web interface, e-mail is still the “hub” of many people’s online lives. That’s why sending e-mail is so important, and when you’re sending e-mail, you really want to be able to receive it, too. There are several reasons why:

  • If e-mails “bounce” (they can’t be delivered for some reason), you need to know about that. Repeatedly sending e-mails that bounce will get your e-mails flagged as spam. Receiving e-mail bounces allows you to disable sending to non-existent addresses.
  • Allowing people to reply to posts via e-mail can significantly improve engagement, as people can reply straight away from their mail client, even if they’re not able to visit the forum at that moment.
  • Letting people post new topics, or send PMs, via e-mail has similar benefits to engagement. In addition, you can use Discourse to handle e-mail for a group, such as an e-mail-based support channel (which is how Discourse’ own e-mail support is handled).

Delivering e-mail directly into your Discourse forum, rather than setting up POP3 polling, has a number of benefits:

  • No need to deal with gmail or another provider’s foibles;
  • You have more control over the e-mail addresses that people use to send posts; and
  • There are no delays in delivery – no more waiting for the next polling run to see new posts appear!

This howto is all about getting that hawtness into your forum.

Overview

This procedure creates a new container on your Discourse server, alongside the typical app container, which receives e-mail and forwards it into Discourse for processing. It supports all e-mail processes: handling bounces, replies, and new topic creation. Any self-hosted Discourse forum using our supported installation process can make use of this procedure to get easy, smooth-flowing incoming e-mail.

Container Setup

We’re going to get the mail-receiver container up and running on the server that’s already running your Discourse instance. There’s no need for a separate droplet just to handle mail – the whole container only takes about 5MB of memory!

So, start off by logging into your Discourse server, and becoming root via sudo:

ssh ubuntu@192.0.2.42
sudo -i

Now, go to your /var/discourse directory and create a new mail-receiver.yml container definition from the sample conveniently provided:

cd /var/discourse
git pull
cp samples/mail-receiver.yml containers/

Since every site is unique, open containers/mail-receiver.yml in your preferred text editor and change the MAIL_DOMAIN, DISCOURSE_MAIL_ENDPOINT, and DISCOURSE_API_KEY variables to suit your site. (If you are an advanced user and know that you are using nginx outside your container, see below for additional configuration for external nginx.)

:bulb: If you use the default mail endpoint (/admin/email/handle_mail), we suggest using the receive_email API key scope to provide an extra layer of security.

If you’re not sure what your favourite text editor is, try nano:

nano containers/mail-receiver.yml

Use Ctrl-X to exit (say “Yes” to “Do you want to save changes?”, or all your work will be for nothing).

Now, do an initial build of the container, and fire it up!

./launcher bootstrap mail-receiver
./launcher start mail-receiver

To check everything’s OK, take a peek in the logs:

./launcher logs mail-receiver

The last line printed should look rather a lot like this:

<22>Aug 31 04:14:31 postfix/master[1]: daemon started -- version 3.1.1, configuration /etc/postfix

If so, all is well, and you can go on to then next step.

DNS Setup

In order for everyone else on the Internet to know where to deliver mail, you must create an MX record for your forum. The exact details of how to do this vary by DNS provider, but in general, the procedure should be very similar to how you setup the DNS records for your forum in the first place, except that instead of creating an A (or “Address”) record, you’re creating an MX (or “Mail eXchange”) record. If your forum is at forum.example.com, and you set MAIL_DOMAIN to forum.example.com in the mail-receiver.yml, then the DNS record should look like this:

  • DNS Name: forum.example.com (this is the MAIL_DOMAIN)
  • Type: MX
  • Priority: 10
  • Value: forum.example.com (this is the domain of your forum)

To make sure the DNS is setup correctly, use a testing site such as http://mxtoolbox.com/ to look up the MAIL_DOMAIN you configured, and make sure it’s pointing to where you expect.

You can also now try sending an e-mail to nobody@forum.example.com. While Discourse won’t do anything useful with it yet, the e-mail you sent should show up in your admin panel under “Emails”, “Rejected” in a matter of seconds. If that happens, you’re definitely ready to proceed.

Note: outbound email providers like mailgun may ask you to add MX records pointing to their servers. You want to remove these so the MX records for your forum only point to your forum’s domain name. SPF and DKIM records must still point to your outbound email provider servers so you can send email.

Discourse Configuration

Now e-mail is being fed into Discourse, it’s time to explain to Discourse what to do with the e-mail it receives. For the most part, this is identical to setting up incoming e-mail via POP3 polling, with a few minor exceptions:

  • Enable the “manual polling” setting, rather than “pop3 polling”; and
  • You can automatically, without any further setup, use any address @forum.example.com as an address for category or group e-mails.

Troubleshooting

Nothing ever goes according to plan. Here’s how to figure out what went wrong.

  1. OCI runtime create failed error running ./launcher start mail-receiver? Your hostname might be too long. Rename it using these instructions and choose a shorter name, then rebuild.
  2. Did the e-mail even make it to mail-receiver? Run ./launcher logs mail-receiver, and look for log entries that mentions the address that the e-mail was sent from and to. If there’s none of those, then the message never even made it, and the problem is upstream. Check MX records and sending mail server logs.
  3. Is the message stuck in the queue? Run ./launcher enter mail-receiver, then run mailq. It should report, “Mail queue is empty”. If there’s any messages in there, you’ll get the to/from addresses listed. Messages only sit around in the queue if there’s a problem delivering to Discourse itself, so exit out of the container and then check…
  4. Did mail-receiver error out somehow? Run ./launcher logs mail-receiver | grep receive-mail and look for anything that looks like a stack trace, or basically anything other than “Recipient: <something>@forum.example.com”. Those error messages, whilst not necessarily self-explanatory, should go an awfully long way to explaining what went wrong. Look for typos in your yml file. In particular, check that DISCOURSE_MAIL_ENDPOINT URL matches your site URL, usually starting with https://.

Integrating with External nginx

If you are an advanced user and have configured external nginx such as for Adding an offline page when rebuilding you will find that the combination of mail-receiver and HTTPS being handled in external nginx requires slightly different handling to enable SSL for email over TLS. Here are example containers/mail-receiver.yml snippets that work with the recommended configuration for external nginx with letsencrypt certificates:

  POSTCONF_smtpd_tls_key_file:  /letsencrypt/live/=DOMAIN=/privkey.pem
  POSTCONF_smtpd_tls_cert_file:  /letsencrypt/live/=DOMAIN=/fullchain.pem

volumes:
  - volume:
      host: /var/discourse/shared/mail-receiver/postfix-spool
      guest: /var/spool/postfix
# uncomment to support TLS
  - volume:
      host: /etc/letsencrypt/
      guest: /letsencrypt

Note that you can’t export as a volume only /etc/letsencrypt/live because the actual files are symlinks into ../../archive/... and those won’t resolve if you are more specific in the volume specification.

Prevent outgoing host email from interfering (Postfix)

If you have (or want) automated messages from your host server (via Postfix), the mail-receiver will conflict because it needs port 25 to operate. One solution is to disable the host Postfix from listening on port 25:

nano /etc/postfix/master.cf

and comment out the line that looks like this:

smtp inet n - y - - smtpd

Then service postfix reload. You may also need to restart the mail-receiver container.

With both the host Postfix and the mail-receiver running, do netstat -tulpn | grep :25 to confirm that docker-proxy is using port 25.

Block unwanted domains from sending to you

To stop email from unwanted domains from even reaching your Discourse, your mail-receiver.yml should look something like this:

  DISCOURSE_API_USERNAME: system

  POSTCONF_smtpd_sender_restrictions: 'texthash:/etc/postfix/shared/sender_access'

volumes:
  - volume:
      host: /var/discourse/shared/mail-receiver/postfix-spool
      guest: /var/spool/postfix
  - volume:
      host: /var/discourse/shared/mail-receiver/etc
      guest: /etc/postfix/shared
# uncomment to support TLS
#  - volume:
#      host: /var/discourse/shared/standalone/letsencrypt
#      guest: /letsencrypt

Then create /var/discourse/shared/mail-receiver/etc path, and within it create a sender_access file containing the domains to reject, like this:

qq.com REJECT
163.com REJECT

Rebuild and you’re golden!

Further Reading

70 Likes
Customising direct-delivery Postfix configuration
Set up Reply via Email Support :e-mail:
Filtering known-bad sender domains from your mail-receiver
Can Discourse ship frequent Docker images that do not need to be bootstrapped?
Set up Reply via Email Support :e-mail:
POP3 Authentication with CRAM-MD5
E-mail polling on GMAIL not working
Troubleshooting Gmail 'Poll via POP3 for email replies'
Set up Reply via Email Support :e-mail:
Protonmail bridged SMTP in Discourse
Configuring Reply via Email :e-mail:
Bad CSRF with mailgun replies
Whether it is possible to set the possibility of answering the noreply@mydomain.com adress?
Reply by Email, Gsuite, Limited users, Aliases?
Can I start a new topic by sending an email message?
Set up Reply via Email Support :e-mail:
NOQUEUE: reject using Direct Email Delivery
Does Discourse Need Incoming Mail?
How to install Discourse on an isolated CentOS 7 server
Started getting 'Unknown To: Address" failures
Set up Reply via Email Support :e-mail:
Can I run wordpress in the same VPS with a second IP?
Configuring Reply via Email :e-mail:
Getting 'address not found' error trying to set up reply via email
Trying to set up reply by email: ActionController::RoutingError (No route matches [POST] "/admin/email/handle_mail")
Joe's Personal Discourse Theme
Additional email address per user account support
How %{reply_key} create?
Allow inbound email delivery to forum contact email for inactive users
Handling bouncing e-mails
How %{reply_key} create?
Set up Reply via Email Support :e-mail:
Code for receiving emails via Discourse API
Set up Reply via Email Support :e-mail:
Reply via email failed
Is there any way to deliver all incoming email to discourse instead of rejecting it?
'hostname "mail.domain.tld" does not match the server certificate' :: SNI support? & how to query cert from Discourse container?
'hostname "mail.domain.tld" does not match the server certificate' :: SNI support? & how to query cert from Discourse container?
Will g-suite POP3 stop working on 2/15/2021
How to migrate from Yahoo groups to Discourse
Handling bouncing e-mails
Handling bouncing e-mails
Setting Up Mail
How to reset or recreate direct-delivery's mail-receiver container?
Handling bouncing e-mails
How to update mail-receiver to the release version
No received mails in new Discourse installation
Can I run wordpress in the same VPS with a second IP?
How might we better structure #howto?
Set up Reply via Email Support :e-mail:
Create new email address on forum domain
Emailed topics arrive only once per hour as opposed to instantly
Reply via email setup with Exchange 2016
Set up Microsoft Teams notifications using the discourse-chat-integration plugin
Disable reply via email?
Need help with reply by email
How hard is it to handle Discourse after installation
Installation by the book
Learnings from setting up neighborhood community on Digital Ocean
Question about the email address domain used by Discourse
Configuring both direct delivery email and a forwarding rule
Configuring both direct delivery email and a forwarding rule
Able to receive e-mail in mail-receiver but NOT in Discourse?
Able to receive e-mail in mail-receiver but NOT in Discourse?
Email in main domain or sub-domain for the forum?
Mail receiver app
Configuring Reply via Email :e-mail:
Custom domain email with discourse DO setup
Gmail has just started preventing Discourse from logging in to check for reply-by email notices
Admin Interface and capabilities
Whether it is possible to set the possibility of answering the noreply@mydomain.com adress?
Handling bouncing e-mails
Generate LetsEnrypt certificates so mail-recevier can use them
Allow for email re-writing to solve the additional email address?
Webhook to forward posts to a mailinglist: best practice?
Email from Google: Review blocked sign-in attempt
"Discourse::NotFound" error when click "Email Type" field on admin/email/bounced
Looking for email hosting service recommendations
Creating a topic via email without write access to the category?
Bad CSRF with mailgun replies
Bad CSRF with mailgun replies
How to use aliases in iRedMail for categories with incoming email feature enabled
[Paid] Discourse configuration changes
Allowing topic creation through direct delivery email in makes my forum vulnerable to spammers
Failed to POST the e-mail (301 error)

Discourse can accept email from unknown users, and stage an account.

As soon as a user becomes known in any context their access is enumerated against their account status. Their posting access is determined by their permissions, or lack thereof.

AFAIK is has always worked that way, and is by design.

5 Likes

You don’t need the Mailgun MX records. All you need to configure in mailgun is spf and DKIM as they are sending email on your behalf.

If you’re using the mail-receiver the mx record should be pointing back at your discourse hostname, otherwise emails are routed to mailgun rather than the mail-receiver container.

6 Likes

I haven’t seen mention of this before, but the combination of mail-receiver and HTTPS being handled in external nginx requires slightly different handling if you want to enable SSL. Here are example container configuration snippets that I think are right for the standard external nginx with letsencrypt certificates:

  POSTCONF_smtpd_tls_key_file:  /letsencrypt/live/forum.example.com/privkey.pem
  POSTCONF_smtpd_tls_cert_file:  /letsencrypt/live/forum.example.com/fullchain.pem
volumes:
  - volume:
      host: /var/discourse/shared/mail-receiver/postfix-spool
      guest: /var/spool/postfix
# uncomment to support TLS
  - volume:
      host: /etc/letsencrypt/
      guest: /letsencrypt

Note that you can’t export as a volume only /etc/letsencrypt/live because the actual files are symlinks into ../../archive/... and those won’t resolve if you are more specific in the volume specification.

5 Likes

Unfortunately, core Discourse currently doesn’t support any of those scopes, but I think it makes sense to add one for the mail-receiver endpoint. It won’t be possible to limit incoming mails to a specific group using though. The receiver address is included inside the email, which is a big chunk of text.

Global API keys are just keys without scopes, this was how keys worked before we added the feature. I recommend selecting the system user for this case instead, “All users” lets you impersonate every user on your site, which is something you don’t need. Regarding the harm it can cause, the key will have admin privileges, so it can do a lot of things. Keep it somewhere safe.

5 Likes

Oh yeah, good point re spam. Spam handling is much better with gmail and is definitely one reason direct delivery is less straightforward than just setting up POP3 with gmail.

That said, it does depend on how you want to use your site. If you are using it primarily to allow posting to categories, by existing users, then you can rely on built in spam handling for discourse which is quite good, esp with akismet enabled and default trust level permissions. You’re just not going to see much spam.

However, if you want to enable posting by non-users, and allow emailing in to groups by non-users, then you are in for a world of pain unless you know how to set up postfix to prevent email from reaching your discourse. There are no bulk actions for messages and users to e.g. select lots of messages at once and mark as spam and delete spammer, or on the user list to delete a selection of users at once. You end up with lots of spam topics and spam users that you have to delete one by one.

I did this for a while and got hammered by emails from qq.com and 163.com. Setting up postfix to block unwanted domains from sending to you works - see the instructions in the OP. But the trouble is that you are then playing whack a mole as spammers move from domain to domain.

5 Likes

I am getting to the point of starting the mail-receiver and getting this error:

Unable to find image 'local_discourse/mail-receiver:latest' locally
docker: Error response from daemon: pull access denied for local_discourse/mail-receiver, repository does not exist or may require 'dock
er login': denied: requested access to the resource is denied.
See 'docker run --help'.

I updated Docker also and tried again. Same error.

1 Like

Can you confirm you did this part?

1 Like

Yes the prior part went off no issues.

This was the line for the error: ./launcher start mail-receiver

1 Like

I have seen that error when docker has its permissions broken such that it can’t do a pull.

What does

docker run hello-world

produce?

3 Likes

Hmmm… issue? :slight_smile:

New release '20.04.2 LTS' available.
Run 'do-release-upgrade' to upgrade to it.


*** System restart required ***
Last login: Fri Feb 12 21:23:13 2021 from 52.94.38.64
ubuntu@ip-172-26-10-129:~$ sudo su -
root@ip-172-26-10-129:~# docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
0e03bdcc26d7: Pull complete 
Digest: sha256:95ddb6c31407e84e91a986b004aee40975cb0bda14b5949f6faac5d2deadb4b9
Status: Downloaded newer image for hello-world:latest
1 Like

I think that’s your issue. The old “did you turn it off and on again?”

2 Likes

I was in the middle of a Discourse update when I did that. Now re-running it, it seems to be fine.

# docker run hello-world

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/
2 Likes

Still getting the same error though on

./launcher start mail-receiver

1 Like

try

./launcher rebuild mail-receiver

2 Likes

Success! Thanks for the help!

3 Likes

Well I was able to get my AWS Lightsail working with the API version. The mail is coming in but I am still getting this error on all messages.

Email::Receiver::BadDestinationAddress

My settings are set to:

discourse-replies+%{reply_key}@forums.kitmaker.net

I thought it may have been due to my sending email not coming from a recognized account in Discourse, but I added that email as a secondary to my account. Also the error is “BadDestinationAddress” so I assume that is more the issue. I don’t get it though it looks like everything is set properly and mail is being received. :frowning:

1 Like

I think a better test is to reply to a notification from the same address to which it was sent. My test for whether incoming mail is working is to send a message to the address and if it is reported as (I think) bad destination address, everything is working as expected.

1 Like

Sorry false alarm. Although odd… I added the secondary email and then did the confirmation link in the email. However maybe since I have 2-factor activated it sent a SECOND email that I didn’t see right away and that had to be confirmed as well (with authenticator code).

Now it’s working. :wink:

2 Likes