Straightforward direct-delivery incoming mail


(Matt Palmer) #49

Yes, if you followed those instructions, the chain would be presented (specifically, the line cat "Your PositiveSSL Certificate" "Intermediate CA Certificate" "Intermediate CA Certificate" >>ssl.crt). However, the chain is, most definitely, not being presented.


(Wes Osborn) #50

Thanks for the pointer. I see that info was added officially to the main instructions in April 2015, we did our original setup in April 2014 and hadn’t had any issues with “typical” clients, so we didn’t add the rest of the certs in the chain.

Just a note for others that make changes, I had to restart the Discourse app and the mail-receiver app after making the cert changes.


@mpalmer When I attempt to check the mail queue I get the following error:

root@proddiscourse:/var/discourse# ./launcher enter mail-receiver
proddiscourse-mail-receiver:/# mailq
postqueue: fatal: malformed showq server response

Any thoughts on what I’m doing wrong there?


(Matt Palmer) #51

:frowning: I have no idea what’s going on there. I’ve never seen that sort of behaviour on our mail receivers, and I’m having trouble even formulating a hypothesis to test. Have you done anything to the mail-receiver container (upgrade anything, do anything custom) at all? Because, short of futzing with something fairly fundamental (like having a half-upgraded Postfix), that’s a real “Shouldn’t Happen” kind of error.


(Robby O'Connor) #52

This works awesomely! I love it!


(Wes Osborn) #53

I rebuilt the mail-receiver container and still get the error:

proddiscourse-mail-receiver:/# mailq
postqueue: fatal: malformed showq server response

Here is what the .yml looks like:

root@proddiscourse:/var/discourse/containers# more mail-receiver.yml
## this is the incoming mail receiver container template
##
## After making changes to this file, you MUST rebuild
## /var/discourse/launcher rebuild mail-receiver
##
## BE *VERY* CAREFUL WHEN EDITING!
## YAML FILES ARE SUPER SUPER SENSITIVE TO MISTAKES IN WHITESPACE OR ALIGNMENT!
## visit http://www.yamllint.com/ to validate this file as needed

base_image: discourse/mail-receiver:1.0.0
update_pups: false

expose:
  - "25:25"   # SMTP

env:
  LANG: en_US.UTF-8

  ## Where e-mail to your forum should be sent.  In general, it's perfectly fine
  ## to use the same domain as the forum itself here.
  MAIL_DOMAIN: lists.clcohio.org

  ## The URL of the mail processing endpoint of your Discourse forum.
  ## This is simply your forum's base URL, with `/admin/email/handle_mail`
  ## appended.  Be careful if you're running a subfolder setup -- in that case,
  ## the URL needs to have the subfolder included!
  DISCOURSE_MAIL_ENDPOINT: 'https://discourse.clcohio.org/admin/email/handle_mail'

  ## The master API key of your Discourse forum.  You can get this from
  ## the "API" tab of your admin panel.
  DISCOURSE_API_KEY: REDACTED

  ## The username to use for processing incoming e-mail.  Unless you have
  ## renamed the `system` user, you should leave this as-is.
  DISCOURSE_API_USERNAME: system

volumes:
  - volume:
      host: /var/discourse/shared/mail-receiver/postfix-spool
      guest: /var/spool/postfix
root@proddiscourse:/var/discourse/containers#

The service seems to be delivering messages but I want to make sure that there isn’t something wrong that will come back and bite us in the future.


(Matt Palmer) #54

Yeah, I’d be a bit worried about that, too, given that error. I honestly couldn’t say for sure that there isn’t something larger at play.

The only new thing that’s sprung to mind is a corrupted mail spool… you could try stopping the container, moving the /var/discourse/shared/mail-receiver/postfix-spool directory out of the way (don’t nuke it, it might have useful info in it) and start the container again (it should be re-created on startup). If mailq suddenly starts working, that was the culprit, and I’d be very interested to see what that mail spool looks like… on the other hand, if it’s still fudged, the next step I’d recommend is a strace-oscopy.


(Christoph) #55

So is this the new (officially) recommended method for incoming email? If not, why not?


(Sam Saffron) #57

Why not, is cause it is significantly more complicated to setup than simply plugging in a bunch of credentials in the web UI.

It is “straightforward” but also much more involved that plugging in pop3 stuff.


(Wes Osborn) #58

That was it!

root@proddiscourse:/var/discourse/shared/mail-receiver/postfix-spool-2017-05-25# find . -type f
./flush/lists_clcohio_org
./pid/unix.cleanup
./pid/unix.defer
./pid/master.pid
./pid/unix.showq
./pid/unix.discourse
./pid/inet.smtp
./pid/unix.flush

But if you’d prefer, I tar it up and PM it to you via wetransfer. Let me know.


(Matt Palmer) #59

Please do. Also, just for completeness, can you drop me the image ID of the mail-receiver container you’re using, so I can make sure I’m running the exact same version in my tests?


(Matt Palmer) #60

Given that in order to use POP3 polling you’ve also got to setup a mailbox somewhere, probably forwarding e-mail from other places to there (with all the attendant amusements), and (if you’re using gmail) turn off all sorts of high security switches to let POP3 work at all, I’m thinking that configuring another container (which is copying a file, editing a couple of values, and running ./launcher) and setting an MX record is almost certainly less involved.

To close out @wesochuck’s mailq problems, I’m unable to reproduce the problem. That may be because I can’t seem to find the base image that he was running (discourse/mail-receiver:1.0.0 is b4f002bde510, but he somehow has an image with ID 5b435447174b), but given the downright strangeness of the problem, I’m inclined to think it was actually some sort of low-level filesystem problem that (a) got cleared with a new mail spool, and (b) didn’t make it through the tarball. Unless some further diagnostic info comes to light, I’m filing this one alongside the solar flares, neutrinos, and other random occurances.


(Robby O'Connor) #61

Also – by using this – you don’t have to set up aliases manually to go to the pop3 mailbox you set up…it simply just works.

You set this up once and it works for anything that discourse accepts. Far less work in my opinion. Not sure I agree it’s more work – there’s more work involved if I wanted to set up sayyy a new category and have a custom email address to post to if I were to use the pop3 polling since I have to set up an alias to the pop3 mailbox.


(Christoph) #62

Wait, are you saying you have a catch all MX record set up to have all mail for your domain sent to the mail-receiver container? Or why don’t you have to setup an alias for a new category custom email?


(Robby O'Connor) #63

Sort of, but no. Unless I set up a valid recipient in discourse, it doesn’t work. If you send to foo@forums.example.com – and nobody set it up – it gets rejected and the sender notified.


(Christoph) #64

I may be completely misunderstanding something, but isn’t that always the case, regardless of whether you use direct delivery or POP3 polling?

Here is what (I believe) you have to do when setting up a new category custom email in both scenarios:

POP3 polling using Gmail as your inbox:

  1. Create a new email-address and forward it to said Gmail account.
  2. Paste that new email address into Custom incoming email address in the category’s settings

Direct-delivery as described in the OP

  1. Create a new email-address and forward it to the mail-receiver container.
  2. Paste that new email address into Custom incoming email address in the category’s settings

Hm, as I write this, I realize where I might be thinking wrongly: the point is that this MX record will route all emails sent to that subdomain to the mail-receiver container:

Right? So that’s where it saves you some work, right? But then again, you could do that with Gmail too by setting up a catch-all * email forwarder to the Gmail account…


(Jay Pfaffman) #65

No. The cool thing with the mail-receiver is that if you want to add an address to a group or category, you just make up an address and stick it in the settings. You don’t need the first step.

I must be missing something. What would you expect to happen with mail sent to a non-existent address?


(Christoph) #66

Yes, that’s what dawned on me too, but then:


(Robby O'Connor) #67

Yes, but I don’t need pop3 polling – which means I don’t need to have a pop3 mailbox – I COULD set up a catch-all – but why? This works – in a straight-forward way too.


(Christoph) #68

Okay, fine. Now we closed the circle, which I believe started off by comparing whether or in what ways direct delivery was less work and you arguing that it was because

Anyway, I now understand that the difference is not all that big when it comes to the day-to-day work, but setup actually does seem more straightforward.

Now, regarding the catch-all email functionality (which comes by default with direct-delivery and which is optional, as it were, with POP3): may this not potentially cause problems when a lot of emails arrive that are not actually used in discourse at all (SPAM, in other words)? Is it not an advantage to have those not even reach your server in the first place?


(Matt Palmer) #69

Yes, if you use the reduced backscatter patch (coming someday to mainline, I swear!), attempts to deliver to non-existent addresses will be rejected before they’re accepted by your server, which is a Very Good Thing.