Reducing backscatter in email interface?

(Ryan C. Gordon) #1

This is a bit of a braindump, under the assumption there isn’t a “this already works if you click this checkbox” solution out there.

I’m experimenting with Discourse and its Mailing List Mode (which are both pretty great!), and I was thinking about some minor improvements with email backscatter.

I’m using the mail-receiver Docker container that Discourse offers, not POP3 polling. What I’m going to describe here only applies to handling one’s own incoming SMTP traffic.

The gist is this: Discourse is smart enough to reject incoming mail if the user isn’t allowed to post somewhere, or if the user wrote to a bogus email address. It sends a polite bounce email, but this has unintended consequences:

  • Spammers hitting the mail server will generate bounces, either for bogus addresses or real people that didn’t actually try to interact with Discourse, so you can get a flurry of backscatter, which is bad.
  • Legit email from legit users that made legit mistakes will get a legit bounce, and GMail will quietly put it in the user’s spam folder (at least, it did for me).

A better solution would be to reject mail during the SMTP transaction, when reasonable to do so. Making that work will:

  • Reduce unnecessary email bounce traffic due to joe-job spammers, etc.
  • Prevent risk of damage to a domain’s spam reputation.
  • Give users a more visible and immediate notice that something went wrong (email providers will tend to put a THIS DIDN’T SEND email in the user’s inbox).

Even if we just use the mail-receiver Docker container that Discourse offers, this still needs some support from the Discourse instance to make this happen. I couldn’t find any API to do this, but it would be useful if we could query with the system API key for something like this:

  • Can a@b.c send mail to x@y.z?

And this returns a boolean (yes or no on whether mail is acceptable) and maybe a simple string of why ("x@y.z isn’t a valid reply-to-thread slug, x@y.z isn’t a create-new-topic address, a@b.c isn’t allowed to post here, etc. It could even just be the existing “Email::Receiver::StrangersNotAllowedError” style error strings).

This could be more fine-grained (“is a@b.c a valid user?” “is x@y.z an available email target?”), but ultimately we want an API that says “should I drop this email right now or send it through to the system?” …it doesn’t have to, and probably shouldn’t, deal with rejections for things that need further processing (email is too short rejections, etc). We really just need to catch the obvious spam and malware vectors here.

Then that API call can be hooked up to /usr/local/bin/receive-mail in the mail-receiver Docker container, which can optionally return a permanent fail to the SMTP client instead of sending the email content to Discourse at all.

Does this sound like a reasonable approach? Does something like this exist already?

Straightforward direct-delivery incoming mail
(Matt Palmer) #2

Nothing like that exists already. It wouldn’t be impossible to add, but it wouldn’t be trivial, either. Postfix has some excellent hooks for allowing SMTP-time checks to occur, we’re just not using them because mail-receiver was very much a “simplest thing that could possibly work” implementation.

If you’d like to work on this, I’m happy to provide guidance, code review, and testing. I’d like to see it happen, it’s just not something that I think anyone here at CDCK World HQ will have any bandwidth to work on any time soon.

(Ryan C. Gordon) #3

I’ve posted a PR for the new API inside of Discourse here.

If I’m understanding Postfix correctly, we’d need something that talks over the Milter protocol or SMTP to properly reject email (it’s too late when /usr/local/bin/receive-mail runs). That seemed non-trivial, at least for the moment, so I’ll wait for feedback on the PR before worrying about that part.

(Matt Palmer) #4

I’ll leave the core PR to those more experienced in the app, but for the Postfix integration, yes, you need something that will talk milter, SMTP or Postfix policy protocol to Postfix and HTTP to Discourse, to translate. I’d strongly recommend going the Postfix policy route, as the protocol is easy to parse, and you can avoid the need to manage spawning your own daemons by having the Postfix master process do it for you.

(Ryan C. Gordon) #5

or Postfix policy protocol

Oh wow, this policy protocol thing is so much less painful! Thanks for mentioning that. :slight_smile:

Ok, I wrote a script to handle the SMTP portions of all this magic.

First, you’ll need the patch in the PR applied, and a sv restart unicorn or reboot or whatever to pick up the changes.

Then get this script:

discourse-smtp-fast-rejection.rb.gz (1.3 KB)

In the mail-receiver Docker container, you’ll want to put that file at /usr/local/bin/discourse-smtp-fast-rejection.rb, perhaps with a less verbose name if you want.

/etc/postfix/mail-receiver-environment.json in the mail-receiver Docker container needs a DISCOURSE_SMTP_SHOULD_REJECT_ENDPOINT entry, or just hardcode it in the script appropriately. It should look like this:

I don’t know how/where that JSON file gets generated. I ended up hardcoding it in my copy for now.

Add this to the end of /etc/postfix/

policy  unix  -       n       n       -       -       spawn
    user=nobody argv=/usr/local/bin/discourse-smtp-fast-rejection.rb

…and this to the end of /etc/postfix/

smtpd_recipient_restrictions = check_policy_service unix:private/policy

And run postfix reload inside the container, or just reboot or whatever.

Note that there’s a bug in my PR about reply-to addresses, but once that’s sorted out, this work might be complete. The mail-receiver container’s SMTP server is rejecting email appropriately before sending it on to Discourse. :slight_smile:

(Matt Palmer) #6

A PR against mail-receiver would be greatly appreciated, once the core changes have gone in.

(Ryan C. Gordon) #7

Okay, this is all sitting in various PRs now. You’ve probably gotten 12 emails with the links at this point, :), but here they are one more time for the general public:

(Allen - Watchman Monitoring) #8

This looks pretty slick, nice work.

There is mention of testing… I’ve got a new discourse instance that’s up and running now, but not scheduled to be in production for a week or so. I can offer it for real-world testing (and can grant access as needed).

As for the setup of this, looking at the new variable in the mail-receiver, I wonder:

If DISCOURSE_SMTP_SHOULD_REJECT_ENDPOINT is going to be optional, should that be commented out by default?

If it should be on by default, maybe there should be just one variable to set, like


I’m suggesting admin URL as it would have to take in to account any subfolder. Using this, the paths to /email/handle_mail and email/smtp_should_reject.json could be implied by the mail-receiver itself.

(Matt Palmer) #9

I was pondering that myself; rather than having to list a whole bunch of URLs all over the place, when the internal structure is known and fixed. @icculus, if you’re feeling frisky that would be a nice change to make, although in the interests of getting things merged I wouldn’t consider it a blocker, it can always be done in the future.

(Ryan C. Gordon) #10

This is a good idea, I’ll fix this up in the morning.

(Ryan C. Gordon) #11

I’ve got a live instance running this already, and I’m trying to decide if there’s any reason I can’t just blow away my existing mail-receiver and build a new one to make sure I didn’t screw up the new parts of the Docker container configuration.

@mpalmer, I assume this line in mail-receiver.yml pulls in the contents of GitHub - discourse/mail-receiver: Docker container for receiving e-mails and forwarding them to a Discourse instance from somewhere?

base_image: discourse/mail-receiver:1.0.0

Is there an easy way to override that, so it picks up my changes? Or is that literally just cloning from the github repo when it sees that?

(Matt Palmer) #12

That’s a reference to the docker hub image that comes pre-built. You can build your own image based on your changes, by running docker build -t my-funky-mail-receiver . in the mail-receiver repo, and then change base_image to my-funky-mail-receiver in the mail-receiver.yml, and everything will Just Work.

(Ryan C. Gordon) #13

If it should be on by default, maybe there should be just one variable to set, like

I went further and changed it to DISCOURSE_BASE_URL; we can tack on the “admin” part too. :slight_smile:

The PRs are updated with that now.

(Ryan C. Gordon) #14

change base_image to my-funky-mail-receiver in the mail-receiver.yml, and everything will Just Work.

This worked like a champ, thank you; is up and running with a freshly built mail-receiver container, so we can say with some confidence that the latest PR is safe to merge.

You can send email to any address to get a rejection to see it working on your end. :slight_smile:

(Robby O'Connor) #16

Should I work on this, or do you have it @icculus?

(Ryan C. Gordon) #17

If you have time now, go for it!

(Robby O'Connor) #18

Okay – just lemme know if you do before I do…I might do it if I get bored…maybe…