Straightforward direct-delivery incoming mail


(Matt Palmer) #45

It should be smooth as silk. You can have both POP3 polling and direct delivery active at once, so the sequence would be something like:

  1. Setup direct delivery.
  2. Change the forwarding rules (or MX config, or whatever) to deliver to the direct delivery MTA rather than the POP3 mailbox.
  3. PROFIT!
  4. After a while (once you’re pretty sure no more mail’s going to land in the POP3 mailbox, due to cached MX records or whatever), nuke the POP3 polling config.

You’ve just got to do something to map incoming envelope-to addresses to site hostnames, and then modify mail-receiver to set the Host: header on connection (if it’s a multisite container), or connect to a different host/port (if separate containers). Since that is exactly how we do it here at Discourse (we lookup the envelope-to address in our master hosting database, and then connect to the appropriate site), mail-receiver is structured to make that pretty easy – just build a new container with an alternate entrypoint script that includes the default script then overrides post_email. If you need assistance, I’m happy to give specific pointers and share bits and pieces of how we do things privately.


(Wes Osborn) #46

My Discourse site requires TLS. We’re using a publicly signed cert from Digicert, but when I attempt to receive mail via mail-receiver, I’m seeing the following in the logs:

<19>May 23 02:50:01 receive-mail[86]: Failed to POST the e-mail to https://discourse.clcohio.org/admin/email/handle_mail: SSL_connect returned=1 errno=0 state=error: certificate verify failed (OpenSSL::SSL::SSLError)
<19>May 23 02:50:01 receive-mail[86]:   /usr/local/lib/ruby/2.3.0/net/http.rb:933:in `connect_nonblock'
  /usr/local/lib/ruby/2.3.0/net/http.rb:933:in `connect'
  /usr/local/lib/ruby/2.3.0/net/http.rb:863:in `do_start'
  /usr/local/lib/ruby/2.3.0/net/http.rb:852:in `start'
  /usr/local/lib/ruby/2.3.0/net/http.rb:1398:in `request'
  /usr/local/bin/receive-mail:66:in `post_email'
  /usr/local/bin/receive-mail:39:in `main'
  /usr/local/bin/receive-mail:81:in `<main>'
<20>May 23 02:50:01 postfix/pipe[85]: warning: write/read private/defer socket: Permission denied

(Matt Palmer) #47

It doesn’t look like you’re sending a chain to a trust anchor:

$ openssl s_client -connect discourse.clcohio.org:443 -servername discourse.clcohio.org
CONNECTED(00000003)
depth=0 C = US, ST = Ohio, L = Columbus, O = Central Library Consortium, CN = *.clcohio.org
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 C = US, ST = Ohio, L = Columbus, O = Central Library Consortium, CN = *.clcohio.org
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
 0 s:/C=US/ST=Ohio/L=Columbus/O=Central Library Consortium/CN=*.clcohio.org
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 High Assurance Server CA

While some browsers will do chain construction and AIA fetching, it’s far from universal, and it certainly isn’t done by OpenSSL (which is what underlies the HTTP library used by mail-receiver). Configure your web server to present a complete chain to a trust anchor, and I suspect things will go much better for you.


(Wes Osborn) #48

Is that the way things would be setup if we used these instructions?

We followed those, but it has been a long time, so may be we need to revisit the configuration that we used.


(Matt Palmer) #49

Yes, if you followed those instructions, the chain would be presented (specifically, the line cat "Your PositiveSSL Certificate" "Intermediate CA Certificate" "Intermediate CA Certificate" >>ssl.crt). However, the chain is, most definitely, not being presented.


(Wes Osborn) #50

Thanks for the pointer. I see that info was added officially to the main instructions in April 2015, we did our original setup in April 2014 and hadn’t had any issues with “typical” clients, so we didn’t add the rest of the certs in the chain.

Just a note for others that make changes, I had to restart the Discourse app and the mail-receiver app after making the cert changes.


@mpalmer When I attempt to check the mail queue I get the following error:

root@proddiscourse:/var/discourse# ./launcher enter mail-receiver
proddiscourse-mail-receiver:/# mailq
postqueue: fatal: malformed showq server response

Any thoughts on what I’m doing wrong there?


(Matt Palmer) #51

:frowning: I have no idea what’s going on there. I’ve never seen that sort of behaviour on our mail receivers, and I’m having trouble even formulating a hypothesis to test. Have you done anything to the mail-receiver container (upgrade anything, do anything custom) at all? Because, short of futzing with something fairly fundamental (like having a half-upgraded Postfix), that’s a real “Shouldn’t Happen” kind of error.


(Robby O'Connor) #52

This works awesomely! I love it!


(Wes Osborn) #53

I rebuilt the mail-receiver container and still get the error:

proddiscourse-mail-receiver:/# mailq
postqueue: fatal: malformed showq server response

Here is what the .yml looks like:

root@proddiscourse:/var/discourse/containers# more mail-receiver.yml
## this is the incoming mail receiver container template
##
## After making changes to this file, you MUST rebuild
## /var/discourse/launcher rebuild mail-receiver
##
## BE *VERY* CAREFUL WHEN EDITING!
## YAML FILES ARE SUPER SUPER SENSITIVE TO MISTAKES IN WHITESPACE OR ALIGNMENT!
## visit http://www.yamllint.com/ to validate this file as needed

base_image: discourse/mail-receiver:1.0.0
update_pups: false

expose:
  - "25:25"   # SMTP

env:
  LANG: en_US.UTF-8

  ## Where e-mail to your forum should be sent.  In general, it's perfectly fine
  ## to use the same domain as the forum itself here.
  MAIL_DOMAIN: lists.clcohio.org

  ## The URL of the mail processing endpoint of your Discourse forum.
  ## This is simply your forum's base URL, with `/admin/email/handle_mail`
  ## appended.  Be careful if you're running a subfolder setup -- in that case,
  ## the URL needs to have the subfolder included!
  DISCOURSE_MAIL_ENDPOINT: 'https://discourse.clcohio.org/admin/email/handle_mail'

  ## The master API key of your Discourse forum.  You can get this from
  ## the "API" tab of your admin panel.
  DISCOURSE_API_KEY: REDACTED

  ## The username to use for processing incoming e-mail.  Unless you have
  ## renamed the `system` user, you should leave this as-is.
  DISCOURSE_API_USERNAME: system

volumes:
  - volume:
      host: /var/discourse/shared/mail-receiver/postfix-spool
      guest: /var/spool/postfix
root@proddiscourse:/var/discourse/containers#

The service seems to be delivering messages but I want to make sure that there isn’t something wrong that will come back and bite us in the future.


(Matt Palmer) #54

Yeah, I’d be a bit worried about that, too, given that error. I honestly couldn’t say for sure that there isn’t something larger at play.

The only new thing that’s sprung to mind is a corrupted mail spool… you could try stopping the container, moving the /var/discourse/shared/mail-receiver/postfix-spool directory out of the way (don’t nuke it, it might have useful info in it) and start the container again (it should be re-created on startup). If mailq suddenly starts working, that was the culprit, and I’d be very interested to see what that mail spool looks like… on the other hand, if it’s still fudged, the next step I’d recommend is a strace-oscopy.


(Christoph) #55

So is this the new (officially) recommended method for incoming email? If not, why not?


(Sam Saffron) #57

Why not, is cause it is significantly more complicated to setup than simply plugging in a bunch of credentials in the web UI.

It is “straightforward” but also much more involved that plugging in pop3 stuff.


(Wes Osborn) #58

That was it!

root@proddiscourse:/var/discourse/shared/mail-receiver/postfix-spool-2017-05-25# find . -type f
./flush/lists_clcohio_org
./pid/unix.cleanup
./pid/unix.defer
./pid/master.pid
./pid/unix.showq
./pid/unix.discourse
./pid/inet.smtp
./pid/unix.flush

But if you’d prefer, I tar it up and PM it to you via wetransfer. Let me know.


(Matt Palmer) #59

Please do. Also, just for completeness, can you drop me the image ID of the mail-receiver container you’re using, so I can make sure I’m running the exact same version in my tests?


(Matt Palmer) #60

Given that in order to use POP3 polling you’ve also got to setup a mailbox somewhere, probably forwarding e-mail from other places to there (with all the attendant amusements), and (if you’re using gmail) turn off all sorts of high security switches to let POP3 work at all, I’m thinking that configuring another container (which is copying a file, editing a couple of values, and running ./launcher) and setting an MX record is almost certainly less involved.

To close out @wesochuck’s mailq problems, I’m unable to reproduce the problem. That may be because I can’t seem to find the base image that he was running (discourse/mail-receiver:1.0.0 is b4f002bde510, but he somehow has an image with ID 5b435447174b), but given the downright strangeness of the problem, I’m inclined to think it was actually some sort of low-level filesystem problem that (a) got cleared with a new mail spool, and (b) didn’t make it through the tarball. Unless some further diagnostic info comes to light, I’m filing this one alongside the solar flares, neutrinos, and other random occurances.


(Robby O'Connor) #61

Also – by using this – you don’t have to set up aliases manually to go to the pop3 mailbox you set up…it simply just works.

You set this up once and it works for anything that discourse accepts. Far less work in my opinion. Not sure I agree it’s more work – there’s more work involved if I wanted to set up sayyy a new category and have a custom email address to post to if I were to use the pop3 polling since I have to set up an alias to the pop3 mailbox.


(Christoph) #62

Wait, are you saying you have a catch all MX record set up to have all mail for your domain sent to the mail-receiver container? Or why don’t you have to setup an alias for a new category custom email?


(Robby O'Connor) #63

Sort of, but no. Unless I set up a valid recipient in discourse, it doesn’t work. If you send to foo@forums.example.com – and nobody set it up – it gets rejected and the sender notified.


(Christoph) #64

I may be completely misunderstanding something, but isn’t that always the case, regardless of whether you use direct delivery or POP3 polling?

Here is what (I believe) you have to do when setting up a new category custom email in both scenarios:

POP3 polling using Gmail as your inbox:

  1. Create a new email-address and forward it to said Gmail account.
  2. Paste that new email address into Custom incoming email address in the category’s settings

Direct-delivery as described in the OP

  1. Create a new email-address and forward it to the mail-receiver container.
  2. Paste that new email address into Custom incoming email address in the category’s settings

Hm, as I write this, I realize where I might be thinking wrongly: the point is that this MX record will route all emails sent to that subdomain to the mail-receiver container:

Right? So that’s where it saves you some work, right? But then again, you could do that with Gmail too by setting up a catch-all * email forwarder to the Gmail account…


(Jay Pfaffman) #65

No. The cool thing with the mail-receiver is that if you want to add an address to a group or category, you just make up an address and stick it in the settings. You don’t need the first step.

I must be missing something. What would you expect to happen with mail sent to a non-existent address?