Reducing backscatter in email interface?

This is a bit of a braindump, under the assumption there isn’t a “this already works if you click this checkbox” solution out there.

I’m experimenting with Discourse and its Mailing List Mode (which are both pretty great!), and I was thinking about some minor improvements with email backscatter.

I’m using the mail-receiver Docker container that Discourse offers, not POP3 polling. What I’m going to describe here only applies to handling one’s own incoming SMTP traffic.

The gist is this: Discourse is smart enough to reject incoming mail if the user isn’t allowed to post somewhere, or if the user wrote to a bogus email address. It sends a polite bounce email, but this has unintended consequences:

  • Spammers hitting the mail server will generate bounces, either for bogus addresses or real people that didn’t actually try to interact with Discourse, so you can get a flurry of backscatter, which is bad.
  • Legit email from legit users that made legit mistakes will get a legit bounce, and GMail will quietly put it in the user’s spam folder (at least, it did for me).

A better solution would be to reject mail during the SMTP transaction, when reasonable to do so. Making that work will:

  • Reduce unnecessary email bounce traffic due to joe-job spammers, etc.
  • Prevent risk of damage to a domain’s spam reputation.
  • Give users a more visible and immediate notice that something went wrong (email providers will tend to put a THIS DIDN’T SEND email in the user’s inbox).

Even if we just use the mail-receiver Docker container that Discourse offers, this still needs some support from the Discourse instance to make this happen. I couldn’t find any API to do this, but it would be useful if we could query with the system API key for something like this:

  • Can a@b.c send mail to x@y.z?

And this returns a boolean (yes or no on whether mail is acceptable) and maybe a simple string of why ("x@y.z isn’t a valid reply-to-thread slug, x@y.z isn’t a create-new-topic address, a@b.c isn’t allowed to post here, etc. It could even just be the existing “Email::Receiver::StrangersNotAllowedError” style error strings).

This could be more fine-grained (“is a@b.c a valid user?” “is x@y.z an available email target?”), but ultimately we want an API that says “should I drop this email right now or send it through to the system?” …it doesn’t have to, and probably shouldn’t, deal with rejections for things that need further processing (email is too short rejections, etc). We really just need to catch the obvious spam and malware vectors here.

Then that API call can be hooked up to /usr/local/bin/receive-mail in the mail-receiver Docker container, which can optionally return a permanent fail to the SMTP client instead of sending the email content to Discourse at all.

Does this sound like a reasonable approach? Does something like this exist already?

9개의 좋아요

Nothing like that exists already. It wouldn’t be impossible to add, but it wouldn’t be trivial, either. Postfix has some excellent hooks for allowing SMTP-time checks to occur, we’re just not using them because mail-receiver was very much a “simplest thing that could possibly work” implementation.

If you’d like to work on this, I’m happy to provide guidance, code review, and testing. I’d like to see it happen, it’s just not something that I think anyone here at CDCK World HQ will have any bandwidth to work on any time soon.

9개의 좋아요

I’ve posted a PR for the new API inside of Discourse here.

If I’m understanding Postfix correctly, we’d need something that talks over the Milter protocol or SMTP to properly reject email (it’s too late when /usr/local/bin/receive-mail runs). That seemed non-trivial, at least for the moment, so I’ll wait for feedback on the PR before worrying about that part.

3개의 좋아요

I’ll leave the core PR to those more experienced in the app, but for the Postfix integration, yes, you need something that will talk milter, SMTP or Postfix policy protocol to Postfix and HTTP to Discourse, to translate. I’d strongly recommend going the Postfix policy route, as the protocol is easy to parse, and you can avoid the need to manage spawning your own daemons by having the Postfix master process do it for you.

4개의 좋아요

or Postfix policy protocol

Oh wow, this policy protocol thing is so much less painful! Thanks for mentioning that. :slight_smile:

Ok, I wrote a script to handle the SMTP portions of all this magic.

First, you’ll need the patch in the PR applied, and a sv restart unicorn or reboot or whatever to pick up the changes.

Then get this script:

discourse-smtp-fast-rejection.rb.gz (1.3 KB)

In the mail-receiver Docker container, you’ll want to put that file at /usr/local/bin/discourse-smtp-fast-rejection.rb, perhaps with a less verbose name if you want.

/etc/postfix/mail-receiver-environment.json in the mail-receiver Docker container needs a DISCOURSE_SMTP_SHOULD_REJECT_ENDPOINT entry, or just hardcode it in the script appropriately. It should look like this:

https://discourse.example.com/admin/email/smtp_should_reject.json

I don’t know how/where that JSON file gets generated. I ended up hardcoding it in my copy for now.

Add this to the end of /etc/postfix/master.cf …

policy  unix  -       n       n       -       -       spawn
    user=nobody argv=/usr/local/bin/discourse-smtp-fast-rejection.rb

…and this to the end of /etc/postfix/main.cf …

smtpd_recipient_restrictions = check_policy_service unix:private/policy

And run postfix reload inside the container, or just reboot or whatever.

Note that there’s a bug in my PR about reply-to addresses, but once that’s sorted out, this work might be complete. The mail-receiver container’s SMTP server is rejecting email appropriately before sending it on to Discourse. :slight_smile:

4개의 좋아요

A PR against mail-receiver would be greatly appreciated, once the core changes have gone in.

6개의 좋아요

Okay, this is all sitting in various PRs now. You’ve probably gotten 12 emails with the links at this point, :), but here they are one more time for the general public:

https://github.com/discourse/discourse/pull/4793

https://github.com/discourse/mail-receiver/pull/2

https://github.com/discourse/discourse_docker/pull/344

4개의 좋아요

This looks pretty slick, nice work.

There is mention of testing.. I’ve got a new discourse instance that’s up and running now, but not scheduled to be in production for a week or so. I can offer it for real-world testing (and can grant access as needed).

As for the setup of this, looking at the new variable in the mail-receiver, I wonder:

If DISCOURSE_SMTP_SHOULD_REJECT_ENDPOINT is going to be optional, should that be commented out by default?

If it should be on by default, maybe there should be just one variable to set, like

DISCOURSE_ADMIN_URL

I’m suggesting admin URL as it would have to take in to account any subfolder. Using this, the paths to /email/handle_mail and email/smtp_should_reject.json could be implied by the mail-receiver itself.

2개의 좋아요

I was pondering that myself; rather than having to list a whole bunch of URLs all over the place, when the internal structure is known and fixed. @icculus, if you’re feeling frisky that would be a nice change to make, although in the interests of getting things merged I wouldn’t consider it a blocker, it can always be done in the future.

3개의 좋아요

This is a good idea, I’ll fix this up in the morning.

I’ve got a live instance running this already, and I’m trying to decide if there’s any reason I can’t just blow away my existing mail-receiver and build a new one to make sure I didn’t screw up the new parts of the Docker container configuration.

@mpalmer, I assume this line in mail-receiver.yml pulls in the contents of GitHub - discourse/mail-receiver · GitHub from somewhere?

base_image: discourse/mail-receiver:1.0.0

Is there an easy way to override that, so it picks up my changes? Or is that literally just cloning from the github repo when it sees that?

1개의 좋아요

That’s a reference to the docker hub image that comes pre-built. You can build your own image based on your changes, by running docker build -t my-funky-mail-receiver . in the mail-receiver repo, and then change base_image to my-funky-mail-receiver in the mail-receiver.yml, and everything will Just Work.

1개의 좋아요

If it should be on by default, maybe there should be just one variable to set, like
DISCOURSE_ADMIN_URL

I went further and changed it to DISCOURSE_BASE_URL; we can tack on the “admin” part too. :slight_smile:

The PRs are updated with that now.

3개의 좋아요

change base_image to my-funky-mail-receiver in the mail-receiver.yml, and everything will Just Work.

This worked like a champ, thank you; discourse.libsdl.org is up and running with a freshly built mail-receiver container, so we can say with some confidence that the latest PR is safe to merge.

You can send email to any address @discourse.libsdl.org to get a rejection to see it working on your end. :slight_smile:

4개의 좋아요

Should I work on this, or do you have it @icculus?

2개의 좋아요

If you have time now, go for it!

Okay – just lemme know if you do before I do…I might do it if I get bored…maybe…

오래된 스레드를 다시 꺼내는데, 적용되지 않았다고 생각했던 메일 수신기 패치의 어떤 형태가 결국 버전 관리 시스템에 반영되었기 때문입니다. 시간이 지나며 시스템이 노후화되면서, 제가 패치를 적용한 컨테이너가 어느 시점부터 작동하지 않게 되었습니다.

메일 수신기 컨테이너를 삭제하고 새로운 컨테이너와 API 키를 설정한 후, 제 Discourse 인스턴스에서 수신 메일이 다시 작동하기 시작했습니다… 하지만 시스템이 2017년 때처럼 거부된 메일을 백스캐터링(backscattering)하는 것 같습니다.

역시나, 이 거부 응답 메일 두 통 모두 제 받은 편지함으로 돌아왔습니다. BadDestinationAddress인 경우는 SMTP 레벨에서 Discourse로 추가 처리 및 응답 메일 전송 전에 거부되었어야 하는데 말이죠. 스팸이 이 서버에 가짜 이메일 주소를 사용하여 접근하면, 이는 백스캐터를 생성하게 됩니다.

SMTP 서버와 직접 통신해 보니, 가짜 이메일을 거부하려는 시도를 하지 않는 것을 확인할 수 있었습니다.

root@discourse:/var/discourse# telnet localhost 25
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
220 ESMTP server
HELO sdfsdfsdf
250 discourse-mail-receiver.localdomain
MAIL FROM: sdfsdf@example.com
250 2.1.0 Ok
RCPT TO: sdfsdfsdf@discourse.libsdl.org
250 2.1.5 Ok

…결국 이런 이야기인데: 기본 메일 수신기 이미지에는 빠른 거부(fast-rejection) 스크립트가 연결되어 있는 것을 볼 수 있지만, 실제로는 거부되지 않는 것 같습니다…?

/etc/postfix/master.cf:

policy     unix  -       n       n       -       -       spawn user=nobody
    argv=/usr/local/bin/discourse-smtp-fast-rejection

/etc/postfix/main.cf:

smtpd_recipient_restrictions = check_policy_service unix:private/policy

이것이 작동하도록 조정해야 하는 것이 있는지, 아니면 작동하지 않는 이유를 더 깊이 파고들 수 있는 방법이 있을까요? 다른 사람들에게는 작동하고 있을까요?

감사합니다!

1개의 좋아요

업데이트… 다른 분들도 이 문제가 있나요?

저도 이 문제를 겪고 있으며, 최근 스팸이 회신 주소(replies+abc123@instance)를 From 헤더로 사용하여 Discourse가 자신에게 자동 응답을 보내는 연쇄 반응을 일으키는 문제를 발견했습니다.

제 판단에 따르면, 빠른 거부(fast rejection) 기능은 모두 병합되었지만, mail-receiver의 readme에는 빠른 거부의 일부인 BLACKLISTED_SENDER_DOMAINS 사용에 대해 언급하고 있을 뿐, 빠른 거부 기능은 기본적으로 비활성화되어 있습니다.

위 변경 사항은 하위 호환성에 대한 조사가 필요하다는 이유로 병합되지 않았습니다. 또한 DISCOURSE_BASE_URL이 부재한 상태에서는 다음 줄에 의해 빠른 거부가 비활성화됩니다.

아래에 표시된 바와 같이 메일 리시버 앱은 DISCOURSE_BASE_URL 또는 DISCOURSE_MAIL_ENDPOINT 중 하나를 처리하도록 작성되어 있으며, readme에는 후자에 대한 언급이 없습니다. 따라서 샘플의 최근 변경 사항을 반영하여 http 대신 https를 사용하여 해당 풀 리퀘스트의 변경 사항을 적용하는 것이 안전할 것 같습니다.

풀 리퀘스트에서 제안된 변경 사항은 mail-receiver의 신규 운영자(샘플을 시작점으로 복사하는 운영자)에게는 기본적으로 빠른 거부가 활성화됨을 의미합니다.

기존 운영자는 yml을 수동으로 업데이트하여 DISCOURSE_BASE_URL을 사용할 때까지 이 기능이 비활성화된 상태를 유지하게 됩니다. 만약 제가 여기서 한 주장이 정확하다면, 샘플이 업데이트된 후 #feature:announcements 채널에 이 사항에 대해 게시하는 것이 좋을 것입니다.

handle_mail 엔드포인트와 매우 유사한 방식으로 빠른 거부 엔드포인트에 대해서도 DISCOURSE_BASE_URL이 존재하지 않을 경우 DISCOURSE_MAIL_ENDPOINT에서 엔드포인트를 추론하도록 하는 것도 간단하게 구현할 수 있습니다.

이렇게 하면 신규 및 기존 운영자 모두에게 기본적으로 빠른 거부가 활성화될 수 있습니다. 변경 사항 이후 mail-receiver를 재빌드하는 모든 사용자에게 활성화하는 것이 바람직하다면, 이 변경 사항을 위한 풀 리퀘스트를 작성하는 데 기꺼이 참여하겠습니다.

2개의 좋아요