Straightforward direct-delivery incoming mail


(Matt Palmer) #1

Discourse is all about enabling civilized discussion. While plenty of people like a web interface, e-mail is still the “hub” of many people’s online lives. That’s why sending e-mail is so important, and when you’re sending e-mail, you really want to be able to receive it, too. There are several reasons why:

  • If e-mails “bounce” (they can’t be delivered for some reason), you need to know about that. Repeatedly sending e-mails that bounce will get your e-mails flagged as spam. Receiving e-mail bounces allows you to disable sending to non-existent addresses.
  • Allowing people to reply to posts via e-mail can significantly improve engagement, as people can reply straight away from their mail client, even if they’re not able to visit the forum at that moment.
  • Letting people post new topics, or send PMs, via e-mail has similar benefits to engagement. In addition, you can use Discourse to handle e-mail for a group, such as an e-mail-based support channel (which is how Discourse’ own e-mail support is handled).

Delivering e-mail directly into your Discourse forum, rather than setting up POP3 polling, has a number of benefits:

  • No need to deal with gmail or another provider’s foibles;
  • You have more control over the e-mail addresses that people use to send posts; and
  • There are no delays in delivery – no more waiting for the next polling run to see new posts appear!

This howto is all about getting that hawtness into your forum.

Overview

This procedure creates a new container on your Discourse server, alongside the typical app container, which receives e-mail and forwards it into Discourse for processing. It supports all e-mail processes: handling bounces, replies, and new topic creation. Any self-hosted Discourse forum using our supported installation process can make use of this procedure to get easy, smooth-flowing incoming e-mail.

Container Setup

We’re going to get the mail-receiver container up and running on the server that’s already running your Discourse instance. There’s no need for a separate droplet just to handle mail – the whole container only takes about 5MB of memory!

So, start off by logging into your Discourse server, and becoming root via sudo:

ssh ubuntu@192.0.2.42
sudo -i

Now, go to your /var/discourse directory and create a new mail-receiver.yml container definition from the sample conveniently provided:

cd /var/discourse
git pull
cp samples/mail-receiver.yml containers/

Since every site is unique, open containers/mail-receiver.yml in your preferred text editor and change the MAIL_DOMAIN, DISCOURSE_MAIL_ENDPOINT, and DISCOURSE_API_KEY variables to suit your site. If you’re not sure what your favourite text editor is, try nano:

nano containers/mail-receiver.yml

Use Ctrl-X to exit (say “Yes” to “Do you want to save changes?”, or all your work will be for nothing).

Now, do an initial build of the container, and fire it up!

./launcher bootstrap mail-receiver
./launcher start mail-receiver

To check everything’s OK, take a peek in the logs:

./launcher logs mail-receiver

The last line printed should look rather a lot like this:

<22>Aug 31 04:14:31 postfix/master[1]: daemon started -- version 3.1.1, configuration /etc/postfix

If so, all is well, and you can go on to then next step.

DNS Setup

In order for everyone else on the Internet to know where to deliver mail, you must create an MX record for your forum. The exact details of how to do this vary by DNS provider, but in general, the procedure should be very similar to how you setup the DNS records for your forum in the first place, except that instead of creating an A (or “Address”) record, you’re creating an MX (or “Mail eXchange”) record. If your forum is at forum.example.com, and you set MAIL_DOMAIN to forum.example.com in the mail-receiver.yml, then the DNS record should look like this:

  • DNS Name: forum.example.com (this is the MAIL_DOMAIN)
  • Type: MX
  • Priority: 10
  • Value: forum.example.com (this is the domain of your forum)

To make sure the DNS is setup correctly, use a testing site such as http://mxtoolbox.com/ to look up the MAIL_DOMAIN you configured, and make sure it’s pointing to where you expect.

You can also now try sending an e-mail to nobody@forum.example.com. While Discourse won’t do anything useful with it yet, the e-mail you sent should show up in your admin panel under “Emails”, “Rejected” in a matter of seconds. If that happens, you’re definitely ready to proceed.

Discourse Configuration

Now e-mail is being fed into Discourse, it’s time to explain to Discourse what to do with the e-mail it receives. For the most part, this is identical to setting up incoming e-mail via POP3 polling, with a few minor exceptions:

  • Enable the “manual polling” setting, rather than “pop3 polling”; and
  • You can automatically, without any further setup, use any address @forum.example.com as an address for category or group e-mails.

Troubleshooting

Nothing ever goes according to plan. Here’s how to figure out what went wrong.

  1. Did the e-mail even make it to mail-receiver? Run ./launcher logs mail-receiver, and look for log entries that mentions the address that the e-mail was sent from and to. If there’s none of those, then the message never even made it, and the problem is upstream. Check MX records and sending mail server logs.
  2. Is the message stuck in the queue? Run ./launcher enter mail-receiver, then run mailq. It should report, “Mail queue is empty”. If there’s any messages in there, you’ll get the to/from addresses listed. Messages only sit around in the queue if there’s a problem delivering to Discourse itself, so exit out of the container and then check…
  3. Did mail-receiver error out somehow? Run ./launcher logs mail-receiver | grep receive-mail and look for anything that looks like a stack trace, or basically anything other than “Recipient: <something>@forum.example.com”. Those error messages, whilst not necessarily self-explanatory, should go an awfully long way to explaining what went wrong.

Set up Reply via Email Support :e-mail:
POP3 Authentication with CRAM-MD5
Set up Reply via Email Support :e-mail:
Configuring Reply via Email :e-mail:
Set up Reply via Email Support :e-mail:
Can Discourse ship frequent Docker images that do not need to be bootstrapped?
Bad CSRF with mailgun replies
Troubleshooting Gmail 'Poll via POP3 for email replies'
Discourse::NotFound error when clicking bounced email type
Email from Google: Review blocked sign-in attempt
Looking for email hosting service recommendations
Creating a topic via email without write access to the category?
Bad CSRF with mailgun replies
Bad CSRF with mailgun replies
Set up Reply via Email Support :e-mail:
Can I start a new topic by sending an email message?
Started getting 'Unknown To: Address" failures
[Paid] Discourse configuration changes
NOQUEUE: reject using Direct Email Delivery
Custom domain email with discourse DO setup
Handling bouncing e-mails
Allow for email re-writing to solve the additional email address?
Respond via Email
Webhook to forward posts to a mailinglist: best practice?
Webmail not working after setting up discourse
#2

This is definitely move in the right direction.

Is there also a plan to add support for inbound email relaying from the most popular providers (sparkpost, sendgrid, etc)? This way we wouldn’t need to run mail server on our machine, but use their service, which we use for SMTP already anyway.


(Matt Palmer) #3

There are no plans to add support for the plethora of formats which different mail providers use for their HTTP relaying (and dealing with the attendant foibles and changes over time) to core. Plugins that people could enable for their specific provider, maintained by the community, would be welcome. It’s quite straightforward to write one in general (and all the hard work is already done), if anyone wants to write one for their mail provider of choice.


#12

Running into a timeout upon delivery, might a firewalling issue though.

<22>Sep  1 03:07:34 postfix/smtpd[135]: connect from unknown[172.17.0.1]
<22>Sep  1 03:07:34 postfix/smtpd[135]: 1DF01102DBC: client=unknown[172.17.0.1]
<22>Sep  1 03:07:34 postfix/cleanup[139]: 1DF01102DBC: message-id=<571a80de-a4ad-65c6-1b73-916c7c4414b0@darmstadt.freifunk.net>
<22>Sep  1 03:07:34 postfix/qmgr[131]: 1DF01102DBC: from=<martin@darmstadt.freifunk.net>, size=2048, nrcpt=1 (queue active)
<22>Sep  1 03:07:34 postfix/smtpd[135]: disconnect from unknown[172.17.0.1] ehlo=1 mail=1 rcpt=1 data=1 quit=1 commands=5
<23>Sep  1 03:07:34 receive-mail[141]: Recipient: [redacted]@forum.darmstadt.freifunk.net
<19>Sep  1 03:08:34 receive-mail[141]: Failed to POST the e-mail to https://forum.darmstadt.freifunk.net/admin/email/handle_mail: execution expired (Net::OpenTimeout)
<19>Sep  1 03:08:34 receive-mail[141]:   /usr/local/lib/ruby/2.3.0/net/http.rb:880:in `initialize'
  /usr/local/lib/ruby/2.3.0/net/http.rb:880:in `open'
  /usr/local/lib/ruby/2.3.0/net/http.rb:880:in `block in connect'
  /usr/local/lib/ruby/2.3.0/timeout.rb:101:in `timeout'
  /usr/local/lib/ruby/2.3.0/net/http.rb:878:in `connect'
  /usr/local/lib/ruby/2.3.0/net/http.rb:863:in `do_start'
  /usr/local/lib/ruby/2.3.0/net/http.rb:852:in `start'
  /usr/local/lib/ruby/2.3.0/net/http.rb:1398:in `request'
  /usr/local/bin/receive-mail:66:in `post_email'
  /usr/local/bin/receive-mail:39:in `main'
  /usr/local/bin/receive-mail:81:in `<main>'
<22>Sep  1 03:08:34 postfix/pipe[140]: 1DF01102DBC: to=<[redcated]@forum.darmstadt.freifunk.net>, relay=discourse, delay=60, delays=0.02/0.01/0/60, dsn=4.3.0, status=deferred (temporary failure)

WORKING! \o/


(Matt Palmer) #13

Awesome to hear you got it working. What was the timeout issue caused by?


#14

Definately firewall related, maybe src address protection, maybe forward drop. Have to look into that tonight.


(Matt Palmer) #15

OK. Let me know if there’s something in the docs that could stand to be improved, to save others the same hassle you’ve caught.


(Joshua Rosenfeld) #16

Hey Matt, attempting this on my Discourse instance, and it is failing when attempting to start the container. Each step was successful until then. Not sure what logs might help, but here is my yml and the last few console lines.

## this is the incoming mail receiver container template
##
## After making changes to this file, you MUST rebuild
## /var/discourse/launcher rebuild mail-receiver
##
## BE *VERY* CAREFUL WHEN EDITING!
## YAML FILES ARE SUPER SUPER SENSITIVE TO MISTAKES IN WHITESPACE OR ALIGNMENT!
## visit http://www.yamllint.com/ to validate this file as needed

base_image: discourse/mail-receiver:1.0.0
update_pups: false

expose:
  - "25:25"   # SMTP

env:
  LANG: en_US.UTF-8

  ## Where e-mail to your forum should be sent.  In general, it's perfectly fine
  ## to use the same domain as the forum itself here.
  MAIL_DOMAIN: leet.arc.rpi.edu

  ## The URL of the mail processing endpoint of your Discourse forum.
  ## This is simply your forum's base URL, with `/admin/email/handle_mail`
  ## appended.  Be careful if you're running a subfolder setup -- in that case,
  ## the URL needs to have the subfolder included!
  DISCOURSE_MAIL_ENDPOINT: 'http://leet.arc.rpi.edu/forum/admin/email/handle_mail'

  ## The master API key of your Discourse forum.  You can get this from
  ## the "API" tab of your admin panel.
  DISCOURSE_API_KEY: redacted_for_security

  ## The username to use for processing incoming e-mail.  Unless you have
  ## renamed the `system` user, you should leave this as-is.
  DISCOURSE_API_USERNAME: system

volumes:
  - volume:
      host: /var/discourse/shared/mail-receiver/postfix-spool
      guest: /var/spool/postfix
root@leet:/var/discourse# ./launcher bootstrap mail-receiver
cd /pups && /pups/bin/pups --stdin
sha256:81a4ab6115dbaf9a7daf8af845f715e60dc1bee2b823399c838f9bb5a0f57932
2155191dee63e002683224b459337f1271550dc9303d6152de78a8b3234c7315
Successfully bootstrapped, to startup use ./launcher start mail-receiver
root@leet:/var/discourse# ./launcher start mail-receiver

starting up existing container
+ /usr/bin/docker start mail-receiver
Error response from daemon: driver failed programming external connectivity on endpoint mail-receiver (43925c38019a63ac9904dcf2ba9c3009f6da63a202e487c277642ce6dd12f493): Error starting userland proxy: listen tcp 0.0.0.0:25: bind: address already in use
Error: failed to start containers: mail-receiver
root@leet:/var/discourse#


#17

You already have an SMTP Server running in the host system.


(Matt Palmer) #18

That’s the money shot right there. You’ve got an existing MTA running on your machine, and that’s stopping Docker from listening to the SMTP port. I’ve just double-checked, and the Digital Ocean Ubuntu droplet (14.04 or 16.04) doesn’t install an MTA by default, so my guess is that you’ve added some package to the machine that’s brought in Postfix (cron is the most likely culprit, in my experience).

I can think of a few potential ways to fix this:

  1. Remove the MTA package (most likely Postfix) entirely, potentially also removing whatever wants an MTA
  2. Install a “relay-only” stub MTA like ssmtp or nullmailer to support whatever wants an MTA (assuming it’s something simple that only uses /usr/sbin/sendmail, like cron), and configure it to relay… somewhere
  3. Ensure the MTA you’ve got on the host is only listening on localhost (inet_interfaces = loopback-only in Postfix) and tell Docker to forward just the public IP, not INADDR_ANY (0.0.0.0) by changing the SMTP expose parameter in containers/mail-receiver.yml to be <publicIP>:25:25.

If none of those suit, a description of what exactly you’re up to might brew up more suggestions.


(Joshua Rosenfeld) #19

Don’t know why it was installed, but it is now removed, and I was able to start the container. Will continue going through instructions - thanks!

Edit: And it works! Email-in worked without issue (rejected an unknown address and created new topic when configured with a proper address). Now testing reply to email.


(Joshua Rosenfeld) #20

OK. So emailing in works great, but Discourse (or perhaps the SMTP server) refuses to send out emails to users once reply-by-email is enabled. The following error is in the logs:

[Sender] 553 5.7.1 <replies+verp-828eb3dd63f224003dae4a928055c72a@leet.arc.rpi.edu>: Sender address rejected: not owned by user d_academic_research_co_zxg04790

Googling a 553 SMTP error seems to imply that there is something unhappy with relays, but this is beyond my email knowledge at this point.


(Eli the Bearded) #21

Sounds like your email provider (RPI) is not happy with VERP, which is a method of using sending unique addresses per recipient for better bounce / reply handling.


(Joshua Rosenfeld) #22

Makes sense - I’ll reach out to the Postmaster to see what can be done about this server side.


(Felix Freiberger) #23

I have set this up, and it works beautifully!

I had to deviate in one step, and wanted to document it here: I have postfix installed on the host in satellite mode, to send out system mail. For this function, postfix doesn’t need to listen on port 25, but it does, causing the

@jomaxro ran into. Since I didn’t want to uninstall postfix (sending local mail works, and configuring it was a pain), I ended editing /etc/postfix/master.cf, commenting out the line about smtp in the beginning, and running service postfix restart. Postfix is still happily sending out mail sent to root, but now, mail-receiver is purring like a cat :smile_cat:


(Matt Palmer) #24

With three reportedly successful installations, I’m going to rip out the warning signs and call this GA. Thanks for the beta testing and feedback!


(Sander Datema) #25

I love this as well. And I was wondering: will you be able to support multisite instances of Discourse as well?


(Matt Palmer) #26

Most of what’s needed for multisite is already in place – you can specify multiple domains in MAIL_DOMAINS (space-separated) and they’ll all be passed through. What’s missing is something to lookup which Host header to use for which incoming domain. For our (CDCK) hosting, we run the same base mail-receiver container that’s available publically, and then layer on top another container which looks up incoming domain -> container mappings in PostgreSQL. Something similar could be built for “ordinary” multisite containers, but that’s not something that I’m going to be able to build in the foreseeable future. A suitable PR would be welcome, and if someone’s serious about doing the work to make it happen, I’d be happy to give pointers.


(Sander Datema) #27

I understand. Unfortunately I only understand a little of what you say, so I won’t be able to create that PR. :slight_smile:


(Jose C Gomez) #28

I am new to discourse just starting running a site, so forgive me if I am misunderstanding, will this also allow sending of email directly from the discourse instance? I am right now using Amazon SES but that can prove quite expensive we have a very quickly growing community and are sending thousands of emails a day.