Straightforward direct-delivery incoming mail

(Robby O'Connor) #70

Make that someday – be in the near future pleaseeeeeeeeeeee – not that my forum has an issue with spam right now…

(Matt Palmer) #71

Yeah, unfortunately it’s not on the money-making side of things, and I’ve got to rework the patch to make it backwards compatible, so it’s not a trivial merge. It’ll get there, just not today.

(Robby O'Connor) #72

Is it usable in its current state? Is there anything I can do to expedite things, as in I’m willing to help.

(Matt Palmer) #73

Yes, as long as you don’t need to support an older config file with a newer container image (ie you’re a new user, or edit your mail-receiver.yml to use the new variables), the patch as currently available reportedly works great.

Well, if you’re up for a bit of programming…

The problem that keeps this from being merged and released is that existing mail-receiver.yml files will define the DISCOURSE_MAIL_ENDPOINT environment variable (because that’s what the existing template says to do), but the newer mail-receivercontainer has switched to using DISCOURSE_BASE_URL.

What’s needed is either:

  1. additional logic to recognise that DISCOURSE_BASE_URL isn’t set, but DISCOURSE_MAIL_ENDPOINT is, and deal with it appropriately (probably don’t enable backscatter protection, etc etc); or
  2. at the very least, detect that the config is out of date and provide suitably simple instructions to upgrade the configuration.

Then, the testing… that’s the time-consuming part, really, making sure everything still works.

If you put together a tested PR for one of those, it would be gratefully accepted.

(Robby O'Connor) #74

Perfect – built the image local and it’s running.

We can derive the DISCOURSE_BASE_URL from the DISCOURSE_MAIL_ENDPOINT since it is very likely that the base URL is the on the current domain…if it’s in a subdirectory (e.g, https://example.com/forums – then we’re in trouble…that’s the one issue – otherwise this is pretty simple to do…

(Ryan C. Gordon) #75

Shoot, I forgot this issue was still pending.

Would it be easier to just back out the one patch where we changed the environment variable? Having a base URL is a better idea in a general sense, but not if it risks breaking people’s installs.

(Robby O'Connor) #76

Oh dear god – no – I’m using this :slight_smile: Don’t break my install :stuck_out_tongue:

Is what I proposed sane…we’d have to do some gymnastics to deal with example.com/discourse – i’m not even sure that’s possible – or at least not easy

(Ryan C. Gordon) #77

The curse of software development!

There’s probably some startup script that sets these variables, surely we can just have it build the base URL variable from the old one, and vice versa, so everything Just Works for everyone involved.

(Matt Palmer) #78

Yes, figuring out what the site’s base URL is from the mail endpoint (strip off /admin/email/handle_mail) is another possible approach.

(Robby O'Connor) #79

This is the sane approach, but only if it’s not defined.

(Robin Ward) #80

Hi there!

I’m wondering if you’ve made any progress on the backwards compatibility here? We are looking to add some customizations to mail-receiver and I didn’t want to duplicate your work.

(Robin Ward) #81

I’ve created a PR To ensure backwards compatibility:

I was quite careful when writing it, but I’m not clear how to test it short of creating an entire production environment to do so, so I figured it’s worth having it reviewed first by more eyeballs.

(Matt Palmer) #82

Eyeballed. Puppies: kicked.

(Christoph) #83

I tried

MAIL_DOMAIN: mydomain.net forum.mydomain.net

but that gave me

/usr/sbin/postconf: fatal: missing '=' after attribute name: "forum.mydomain.net"
(Christoph) #84

I have set enabled manual polling for forum.mydomain.net and it works like a charm. Fascinating to see how fast the reply pops up on the forum!

Now, my problem is the following: I had previously not used the forum subdomain for incoming email.

Details

My reply by email address was forum-replies+%{reply_key}@mydomain.net and forum-replies@mydomain.net was (and still is) forwarded to a gmail account (from which discourse does the pop3 polling).

So, in order for the direct delivery to work, I had to change my reply by email address to ...@forum.mydomain.net while that works fine, I hesitate to turn the pop3 polling from the gmail account off because chances are that people will still be replying to the old address (without the subdomain) and since those end up in gmail, I need to keep polling it.

My question is: how do I safely shut down the old gmail polling based system?

I have considered forwarding those old emails to the new mail-receiver instead of gmail, but would it work? Because they will still be directed to @mydomain.net and I assume that the mail-receiver will not accept them. And it it will, I don’t know if discourse will accept them because the reply by email address no longer matches.

In fact, as I think of it, the latter may well be a problem already now when I still emails arriving via gmail. :scream:

(Matt Palmer) #85

OK, that’s been fixed in this commit; update the base_image in your mail-receiver.yml to use version 1.1.2 (from probably 1.0.0) and things should be much happier there.

Wait a while, until you’re pretty sure nobody’s going to reply (or you’re willing to tell 'em to be more timely in their responses) and then turn off POP3 polling.

There’s an “alternate reply by email addresses” setting that you can use to specify other addresses that Discourse should recognise, specifically for this situation.

(Christoph) #86

Thanks!

Given that the multi-domain bug is fixed, I suppose I could just as well change the target address for the old reply-to address from gmail to something@forum.mydomain.net, and then the old addresses will work forever, even without pop3 polling, right?

One thing comes to mind when comparing direct delivery with pop3 polling: what happens when my server is down (even if it’s just for rebooting)?

(Eli the Bearded) #87

Direct email delivery was designed in an age when computers could fall off the net for days at a time and the protocol can handle it.

High level overview:

  1. Alice writes a message to Bob, and presses send in her mailer.
  2. Alice’s mailer does a brief sanity check on the message, and then (in a typical scenario) forwards it to the local relay host.
  3. Alice’s relay host looks up the mail servers for the recipients in the message. This will be A, AAAA, or MX records for the hostname.
  4. Alice’s relay host will try to connect to the mail server found. If unable to deliver, the relay host will save it to a queue.
  5. Periodically the relay host will try to redeliver mail in the queue.
    Traditionally the relay host will send a bounce warning after N hours (1 <= N <= 48) if still trying and failing.
  6. After M hours (eg 72 <= M <= 168) of trying, a message might be returned to Alice as undeliverable.
  7. When Bob finally does get a message, he can examine the Received: headers to track what stops it made and how long it spent at each.

Three Received headers are typical for mail I get these days: Alice’s first stop, the relay, and my local MX. A spam filter, or more complicated local delivery can push up the Received header count.

Your case sounds like step 5 here.

Things get more complicated with multiple MX hosts, but not terribly so. There are many, many mailers out there, with many, many configurations. I can’t promise that all will try to redeliver, just the protocol is designed to allow that and most mailers support that case.

(Jay Pfaffman) #88

And in spite of all of the things that have changed about email delivery over the past 20 years because of spammers, this process has remained the same.

(Eli the Bearded) #89

Some of the things spammers have changed:

  1. No more open relays. Alice’s relay in step 3 will be very careful who it accepts mail from to send out.
  2. Bob’s mail receiver (MX) host has been optimized to reject email at delivery time so that mail which would bounce never goes into a local queue. Previously MX hosts were more open to “accept everything, figure out what’s going to a real address later”.
  3. Relay servers get a lot of scrutiny from MX hosts, including full round-trip DNS (if 123.45.67.8 claims to be mx.example.com, do a DNS lookup of mx.example.com and see that the IP matches, and do a DNS lookup of 8.67.45.123.in-addr.arpa and see that that name matches, too).

You probably know all this, but those reading along might not.