Straightforward direct-delivery incoming mail

Discourse is all about enabling civilized discussion. While plenty of people like a web interface, e-mail is still the “hub” of many people’s online lives. That’s why sending e-mail is so important, and when you’re sending e-mail, you really want to be able to receive it, too. There are several reasons why:

  • If e-mails “bounce” (they can’t be delivered for some reason), you need to know about that. Repeatedly sending e-mails that bounce will get your e-mails flagged as spam. Receiving e-mail bounces allows you to disable sending to non-existent addresses.
  • Allowing people to reply to posts via e-mail can significantly improve engagement, as people can reply straight away from their mail client, even if they’re not able to visit the forum at that moment.
  • Letting people post new topics, or send PMs, via e-mail has similar benefits to engagement. In addition, you can use Discourse to handle e-mail for a group, such as an e-mail-based support channel (which is how Discourse’ own e-mail support is handled).

Delivering e-mail directly into your Discourse forum, rather than setting up POP3 polling, has a number of benefits:

  • No need to deal with gmail or another provider’s foibles;
  • You have more control over the e-mail addresses that people use to send posts; and
  • There are no delays in delivery – no more waiting for the next polling run to see new posts appear!

This howto is all about getting that hawtness into your forum.

Overview

This procedure creates a new container on your Discourse server, alongside the typical app container, which receives e-mail and forwards it into Discourse for processing. It supports all e-mail processes: handling bounces, replies, and new topic creation. Any self-hosted Discourse forum using our supported installation process can make use of this procedure to get easy, smooth-flowing incoming e-mail.

Container Setup

We’re going to get the mail-receiver container up and running on the server that’s already running your Discourse instance. There’s no need for a separate droplet just to handle mail – the whole container only takes about 5MB of memory!

So, start off by logging into your Discourse server, and becoming root via sudo:

ssh ubuntu@192.0.2.42
sudo -i

Now, go to your /var/discourse directory and create a new mail-receiver.yml container definition from the sample conveniently provided:

cd /var/discourse
git pull
cp samples/mail-receiver.yml containers/

Since every site is unique, open containers/mail-receiver.yml in your preferred text editor and change the MAIL_DOMAIN, DISCOURSE_MAIL_ENDPOINT, and DISCOURSE_API_KEY variables to suit your site. If you’re not sure what your favourite text editor is, try nano:

nano containers/mail-receiver.yml

Use Ctrl-X to exit (say “Yes” to “Do you want to save changes?”, or all your work will be for nothing).

Now, do an initial build of the container, and fire it up!

./launcher bootstrap mail-receiver
./launcher start mail-receiver

To check everything’s OK, take a peek in the logs:

./launcher logs mail-receiver

The last line printed should look rather a lot like this:

<22>Aug 31 04:14:31 postfix/master[1]: daemon started -- version 3.1.1, configuration /etc/postfix

If so, all is well, and you can go on to then next step.

DNS Setup

In order for everyone else on the Internet to know where to deliver mail, you must create an MX record for your forum. The exact details of how to do this vary by DNS provider, but in general, the procedure should be very similar to how you setup the DNS records for your forum in the first place, except that instead of creating an A (or “Address”) record, you’re creating an MX (or “Mail eXchange”) record. If your forum is at forum.example.com, and you set MAIL_DOMAIN to forum.example.com in the mail-receiver.yml, then the DNS record should look like this:

  • DNS Name: forum.example.com (this is the MAIL_DOMAIN)
  • Type: MX
  • Priority: 10
  • Value: forum.example.com (this is the domain of your forum)

To make sure the DNS is setup correctly, use a testing site such as http://mxtoolbox.com/ to look up the MAIL_DOMAIN you configured, and make sure it’s pointing to where you expect.

You can also now try sending an e-mail to nobody@forum.example.com. While Discourse won’t do anything useful with it yet, the e-mail you sent should show up in your admin panel under “Emails”, “Rejected” in a matter of seconds. If that happens, you’re definitely ready to proceed.

Discourse Configuration

Now e-mail is being fed into Discourse, it’s time to explain to Discourse what to do with the e-mail it receives. For the most part, this is identical to setting up incoming e-mail via POP3 polling, with a few minor exceptions:

  • Enable the “manual polling” setting, rather than “pop3 polling”; and
  • You can automatically, without any further setup, use any address @forum.example.com as an address for category or group e-mails.

Troubleshooting

Nothing ever goes according to plan. Here’s how to figure out what went wrong.

  1. Did the e-mail even make it to mail-receiver? Run ./launcher logs mail-receiver, and look for log entries that mentions the address that the e-mail was sent from and to. If there’s none of those, then the message never even made it, and the problem is upstream. Check MX records and sending mail server logs.
  2. Is the message stuck in the queue? Run ./launcher enter mail-receiver, then run mailq. It should report, “Mail queue is empty”. If there’s any messages in there, you’ll get the to/from addresses listed. Messages only sit around in the queue if there’s a problem delivering to Discourse itself, so exit out of the container and then check…
  3. Did mail-receiver error out somehow? Run ./launcher logs mail-receiver | grep receive-mail and look for anything that looks like a stack trace, or basically anything other than “Recipient: <something>@forum.example.com”. Those error messages, whilst not necessarily self-explanatory, should go an awfully long way to explaining what went wrong.

Further Reading

64 Likes
Customising direct-delivery Postfix configuration
Set up Reply via Email Support :e-mail:
Filtering known-bad sender domains from your mail-receiver
Configuring Reply via Email :e-mail:
Can Discourse ship frequent Docker images that do not need to be bootstrapped?
Set up Reply via Email Support :e-mail:
POP3 Authentication with CRAM-MD5
E-mail polling on GMAIL not working
Set up Reply via Email Support :e-mail:
Bad CSRF with mailgun replies
Troubleshooting Gmail 'Poll via POP3 for email replies'
Whether it is possible to set the possibility of answering the noreply@mydomain.com adress?
[Paid] Discourse configuration changes
Allowing topic creation through direct delivery email in makes my forum vulnerable to spammers
Failed to POST the e-mail (301 error)
How reply email is working with Outlook
Reply by Email, Gsuite, Limited users, Aliases?
Can I start a new topic by sending an email message?
Set up Reply via Email Support :e-mail:
NOQUEUE: reject using Direct Email Delivery
Does Discourse Need Incoming Mail?
How to install Discourse on an isolated CentOS 7 server
Started getting 'Unknown To: Address" failures
Set up Reply via Email Support :e-mail:
Can I run wordpress in the same VPS with a second IP?
Configuring Reply via Email :e-mail:
Configuring Reply via Email :e-mail:
Getting 'address not found' error trying to set up reply via email
Trying to set up reply by email: ActionController::RoutingError (No route matches [POST] "/admin/email/handle_mail")
Joe's Personal Discourse Theme
Using discourse as your email client
Additional email address per user account support
How %{reply_key} create?
Allow inbound email delivery to forum contact email for inactive users
Handling bouncing e-mails
How %{reply_key} create?
Code for receiving emails via Discourse API
Set up Reply via Email Support :e-mail:
Reply via email failed
Is there any way to deliver all incoming email to discourse instead of rejecting it?
'hostname "mail.domain.tld" does not match the server certificate' :: SNI support? & how to query cert from Discourse container?
'hostname "mail.domain.tld" does not match the server certificate' :: SNI support? & how to query cert from Discourse container?
Will g-suite POP3 stop working on 2/15/2021
How to migrate from Yahoo groups to Discourse
Handling bouncing e-mails
Handling bouncing e-mails
Setting Up Mail
Protonmail bridged SMTP in Discourse
How to reset or recreate direct-delivery's mail-receiver container?
Handling bouncing e-mails
How to update mail-receiver to the release version
No received mails in new Discourse installation
Can I run wordpress in the same VPS with a second IP?
Create new email address on forum domain
Emailed topics arrive only once per hour as opposed to instantly
Reply via email setup with Exchange 2016
Issues setting up new topic via email
Custom domain email with discourse DO setup
Gmail has just started preventing Discourse from logging in to check for reply-by email notices
Admin Interface and capabilities
Whether it is possible to set the possibility of answering the noreply@mydomain.com adress?
Handling bouncing e-mails
Allow for email re-writing to solve the additional email address?
Bounce+ email address not verifying
Webhook to forward posts to a mailinglist: best practice?
Discourse::NotFound error when clicking bounced email type
Email from Google: Review blocked sign-in attempt
Looking for email hosting service recommendations
Creating a topic via email without write access to the category?
Bad CSRF with mailgun replies
Bad CSRF with mailgun replies
How to use aliases in iRedMail for categories with incoming email feature enabled

Forgot I hadn’t responded to this, what I meant by multiple domains was actually multisite. AFAIK there’s no way to associate different inbound email domains with separate URLs and API keys.

I’m still really interested in any solutions to use the mail-receiver with multisite.

I know several places where communities are deploying on multiple servers just to be able to have inbound email on each board.

3 Likes

My solution has been to create individual incoming mail servers on a single host with unique MXed (IPv6) IP addresses for separate incoming mail containers. I think it would work fine (if I hadn’t also broken some other stuff by introducing IPv6).

4 Likes

It doesn’t need to match, you can still use a different FQDN such as Discourse.*

So you’re saying that even though Discourse runs on domain.tld and mail.domain.tld operates on a different server, we can make up a new name for the Dockerized mail server such as discourse.domain.tld that I will assign in the yaml config and that will send and receive just fine as long as this discourse.domain.tld is assigned the Docker host that houses both the Discourse domain.tld and Dockerized (Postfix I believe) mail server living alongside of Discourse is set?

I have configured the incoming email according to the instructions, assigned support@discourse.forum.tld to a group, and set this address as forum contact email.

Everything is working as expected except one scenario:

  • new user registers to the forum and receives confirmation email
  • account is created, email is not yet verified
  • user is (for some reason) unable to use the confirmation link
  • they send email to the forum contact email, which was set according to the instructions in this howto
  • the email is rejected with the reason Email::Receiver::InactiveUserError

I have caught this purely by chance, by reviewing the logs.

Is there any chance to configure inbound email so that it receives emails from inactive users (perhaps limiting it only to forum contact email)? Otherwise there is no way for a user to contact the forum support in such scenario.

1 Like

This is worth testing! I have not seen it. In the meantime, to be sure you are reachable, you can change the contact email in your admin settings to a separate email address not connected with your forum. In my community, I send folks to a form on the wordpress website associated with the forum.

I wonder if this issue also affects the usual incoming email by pop3 polling method.

1 Like

You can test this easily by deactivating one account and then trying to send an email from their associated email address.

All email from either active accounts or from email addresses not associated with any accounts are delivered, so this is not a connectivity issue.

It looks like Discourse is configured not to accept email from deactivated users by default, which is not a problem per se, only in a scenario like the one described above, where user is not activated yet.

1 Like

It does seem to me that any discourse group used to provide support should be reachable by anybody by email, whether they have an account or not and whether their account is active or not. I thought this was the case already.

Anyhoo this is likely a bug report or feature request not specific to this topic which is about setting up straightforward direct-delivery incoming email.

1 Like

Discourse can accept email from unknown users, and stage an account.

As soon as a user becomes known in any context their access is enumerated against their account status. Their posting access is determined by their permissions, or lack thereof.

AFAIK is has always worked that way, and is by design.

4 Likes

If you want people who aren’t members to be able to contact that group you need to enable staged users.

Edit: Stephen beat me to this answer while I are my cereal.

3 Likes

It already works for (I have tested all those scenarios before deployment):

  • emails associated with active users
  • emails not previously associated with any users (staged account is created on first email delivery)
  • emails associated with suspended users (if their emails were not deactivated)
  • emails associated with silenced users (if their emails were not deactivated)

The problems are members who are not activated yet!

2 Likes

Yes, as has already been mentioned, this is by design, and has nothing to do with the method by which the e-mail arrives in Discourse. If you wish to discuss changing this behaviour, you should start a new topic.

4 Likes

Clearly the answer here is switch to IV caffeine :smiley:

4 Likes

Hi there. I have the system working but I see that system_messages.email_reject_bad_destination_address.text_body_template message is sent at every time that mail system recieves an e-mail.

Maybe I forgot to configure something. Any suggestion is appreciated, thanks!

EDIT: I really don’t know how to reply the e-mails that my domain recieves through Discourse. I see some recieved messages but I can’t reply it.

I’m missing someething?

Hello, I’m getting incoming messages stuck in the queue and errors:

<19>May 13 14:01:24 receive-mail[107]: Failed to POST the e-mail to https://forum-url-here/admin/email/handle_mail: execution expired (Net::OpenTimeout)
<19>May 13 14:01:24 receive-mail[107]:   /usr/local/lib/ruby/2.3.0/net/http.rb:880:in `initialize'
  /usr/local/lib/ruby/2.3.0/net/http.rb:880:in `open'
  /usr/local/lib/ruby/2.3.0/net/http.rb:880:in `block in connect'
  /usr/local/lib/ruby/2.3.0/timeout.rb:101:in `timeout'
  /usr/local/lib/ruby/2.3.0/net/http.rb:878:in `connect'
  /usr/local/lib/ruby/2.3.0/net/http.rb:863:in `do_start'
  /usr/local/lib/ruby/2.3.0/net/http.rb:852:in `start'
  /usr/local/lib/ruby/2.3.0/net/http.rb:1384:in `request'
  /usr/local/lib/ruby/site_ruby/mail_receiver/discourse_mail_receiver.rb:42:in `process'
  /usr/local/bin/receive-mail:12:in `<main>'

How could I further troubleshoot the failed connection?
Thanks.

1 Like

Looks like a firewall problem. Start by looking in your iptables configuration.

3 Likes

i am having issues due to the SSL

<19>Jul 30 12:49:08 receive-mail[249]: Failed to POST the e-mail to https://MYDOMAIN.com/admin/email/handle_mail: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (OpenSSL::SSL::SSLError)

Is there a way to add a trusted CA to the mail-reciever container ?

1 Like

What exactly does activivating TLS support do in mail-receiver.yml?

# uncomment to support TLS
#  - volume:
#      host: /var/discourse/shared/standalone/letsencrypt
#      guest: /letsencrypt
1 Like

It makes it possible to add stuff to mail-receiver.yml for it to use the Let’s Encrypt certs for secure SMTP.

See also:

# uncomment these (and the volume below!) to support TLS 
#  POSTCONF_smtpd_tls_key_file:  /letsencrypt/discourse.example.com/prop.ltcmp.net.key
#  POSTCONF_smtpd_tls_cert_file:  /letsencrypt/discourse.example.com/fullchain.cer
#  POSTCONF_smtpd_tls_security_level: may

It’s badly commented, though, as you need to know to replace prop.ltcmp.net with your own domain.

But, last I checked, I believed that it would allow you to send encrypted data to the mail receiver.

2 Likes

Thanks a lot for the how-to. I managed to set up the mail-receiver container (I’ve sent an email to nobody@mydomain.tld and can see them in the admin panel under > email > rejected, just as described/expected).

Definitely proud about that part :partying_face: (maybe a bit too much though? :sweat_smile: )

But I’m just totally confused by what comes after that.

So I tried to search for anyone also being stuck at that step, so I followed that given link which describes two options

  1. POP3-polling
  2. Using the discourse-API

Since this how-to says:

I assumed they meant the second (API) option. However, another user there seemed to be stuck at the same step as me (link) and as a reply was told s/he doesn’t need to do anything with the API when using mail-receiver container and to ask here. So now I’m totally confused what next steps I should take.

Hence, I’m asking here, if somebody could help me clear up my confusion what to actually do.

Any help is highly appreciated :pray:

1 Like