I’m trying to figure out how to get discourse to clean up email lists, specifically bounced emails. The site is using a private smtp server to send emails but the response is set to came back to a gmail address for POP access by discourse.
So I see bounced emails in the Received emails dashboard.
Discourse uses the reply_key to associate bounces with sent emails. When Discourse sends mail, it uses the reply_by_email_address value as the SMTP envelope sender.
You need to ensure your reply_by_email_address includes a {reply_key} so your instance can associate the bounces with the correct sent mail.
For instance, here on meta it’s set to incoming+%{reply_key}@meta.discoursemail.com and anything set to that address gets delivered to this instance.
Thanks for the response. Unfortunately I’ve had to turn off the reply_key from the response to email because on our email relay server doesn’t recognize the + addressing. Is there any other option?
Okay to clarify, the destination SMTP domain server which receives the bounced email doesn’t recognize the + addressing, so it only forwards emails based on the To field to the gmail address which is picked up by discourse via POP. That if the To field includes the reply_key then it won’t forward that email to the gmail account used by discourse.
So I can’t put the reply_key in the email address, can it put somewhere else? In the subject field or body or somewhere where discourse may be able to parse it?
The email sent by Discourse goes from discourse@xxx.com via a dedicated domain SMTP server setup for discourse (using the discourse domain address).
Any replies or when email bounces back, the domain SMTP server forwards the email to discourse.xxxx@gmail.com. This domain SMTP server cannot recognize the + addressing, so if I include the reply_key in the reply to email address, it’ll get dropped by the domain SMTP server. I can only set rules to forward discrete/unique incoming email addresses.
The discourse forum then uses POP via GMail to access those forwarded replies/bounced emails and parses them.
I understand what you’re saying. Unfortunately due to a limitation of the SMTP server rules I cannot setup forwarding for sub domains, I can only configure it to forward unique To emails id’s.
However I have a clarification here - more like a discrepancy in how discourse appears to be working:
When a user replies to an email post, it comes in correctly - replies seem to work perfectly even without any {reply_key} explicitly configured anywhere (see the screenshots above)
However, when an email gets bounced, Discourse categorizes it under Received rather than Bounced
The error logs are showing that something in discourse recognizes that it’s a bounced email (see my first post error log). So if something in discourse is recognizing that it’s a bounced email, why doesn’t is show up under Bounced instead of Received?
So why is that when a user replies to a post it is processed correctly by discourse but not a bounced email (and there also appears to be a disconnect between the error logs and the dashboard for bounced emails). Am I missing something in the configuration?
I’ve also tried setting the reply by email address settings in discourse to discourse.xxx+%{reply_to}@gmail.com but then when the email gets delivered, gmail seems to think that the domain yyy.com (discourse SMTP domain) is trying to spoof the gmail domain and it ends up marking it as spam. It appears that setting a reply to domain different from the senders domain is triggering a DMARC and SPF failure.
ARC-Authentication-Results: i=1; mx.google.com;
spf=softfail (google.com: domain of transitioning discussion.xxx+verp-b9c40db917ca04993dd3433cc9748518@gmail.com does not designate y.y.y.y as permitted sender) smtp.mailfrom=discussion.xxx+verp-b9c40db917ca04993dd3433cc9748518@gmail.com;
dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yyy.com
Return-Path: <discussion.xxx+verp-b9c40db917ca04993dd3433cc9748518@gmail.com>
You’ve turned off find related posts with key (I hope you read the warning) thus Discourse is using the in-reply-to mail header to determine to which topic/post the reply should be referencing.
It can’t do that for bounces - Discourse needs to know the specific message which bounced and that information is only guaranteed to be in one spot - the To: address (which comes from the envelope-from address of the original message).
For this to work, when Discourse sends a message, it needs to receive the reply to the address from which it came. Discourse looks in the To: header for this (not the envelope-to).
is spoofing the gmail domain
If you want to send mail from a gmail address, you need to send it via their servers. But they don’t like that.
Doesn’t look like you’ve configured DKIM for yyy.com; you should do that. If you get that right, DMARC should pass.
Yes it’s configured and the email “sent” by discourse via the yyy.com SMTP server passes SPF, DKIM and DMARC without issues (atleast from the “Send test email” page in the admin console).
This issue is happening if I set a gmail address as the reply_by_email_address to a gmail.com address instead of the yyy.com address. Is there something I can do for this setup so that DMARC doesn’t fail when set to reply_by_email_address to a gmail address while keeping the outgoing server as yyy.com ?
Can it not parse the rest of the content/attachments in the bounced email to extract that information if it can’t what it need in the “obvious” place (or atleast provide an option in settings to do that with the necessary warnings about impersonation)
DMARC alignment fails since the reply_by_email_address is used as the envelope-from address in this situation.
Pretty much the only thing guaranteed to be intact on a bounce is the address.
Maybe the subject, but it would not be practical to put this information there.
I do see that some systems include the original message as an attachment… it’s theoretically possible that we could check bounces for an attachment to get more information if they exist.
If I’m understanding this correctly, the reply_by_email_address should be set in the reply-to field of the envelope sent by the Discourse (from yyy.com). So when the user replies, they reply to the reply-to email (gmail.com) rather than the original (yyy.com) address. So in the email response envelope, the from address should be the users email address and the to is the gmail.com address.
Why would the reply_by_email_address be used as the from address?
the reply_by_email_address is not used as the from address but the envelope-from, specifically so bounces work
Here’s an image that demonstrates how each address is used. The addresses themselves are specific to our hosting, but should be sufficient for an example.
So basically the reply_by_email_address is used in the from header so that it comes back to that email address if it bounces. And the same email address is also set on the reply-to header if replies by email are enabled.
So, if my understanding above is correct, then if discourse can have a separate setting (reply_to_email) for the reply-to header, then that would solve the issue for the DMARC failure. When using the yyy.com domain (taken from reply_by_email_address) for from while sending and gmail.com for the reply-to (taken from the a new reply_to_email setting). If the email bounces it will still be returned to the reply_by_email_address but if the user replies then it would go to reply_to_email.
That’s the SMTP envelope-from, not the From header. The From header is not used for bounces.
i.e. ① not ②:
To pass DMARC, either the SPF (which validates the envelope-from in combination with the sending IP) or DKIM (which validates the From and checksums fields of the message) have to align.
It sounds like the whole point of this exercise is to have a “pretty” reply-to address, that users reply to?
You want something like this?
envelope-from: outgoing+12309847801923840923502389423@yyy.com
…
From: notifications@yyy.com
To: user@contoso.com
Reply-to: my_sweet_forum+12309847801923840923502389423@gmail.com
You must have that envelope-from like so, or bounces will not work.
Yes, that’s correct. The idea to have the envelope-from different from the reply-to.
Since the envelope-from domain and the sending IP match, the SPF should pass but at the same time the reply can go to GMail to process responses and if the email should bounce it’ll head back to the original domain server which can then forward the bounce back to the GMail inbox as well.
It would actually look like this:
envelope-from: outgoing@yyy.com
…
From: notifications@yyy.com
To: user@contoso.com
Reply-to: my_sweet_forum+12309847801923840923502389423@gmail.com
In my setup the outgoing won’t have a VERP because my incoming SMTP doesn’t support VERP (i.e. the bounce back won’t have a VERP address), that’s why the reply-to is being sent to GMail because it supports VERP. This shouldn’t cause a DMARC failure as it does right now.
