Discourse setting Reply-To value in `From` header field when using reply by eMail feature

[RBoy]

This issue was also reported on different topic related to setting up email replies. As suggested by @mpalmer I’m creating a new topic.

The other users who reported this issue on this topic:
https://meta.discourse.org/t/set-up-reply-via-email-support/14003/96

I’ve searched the forum and can’t seem to find an answer so after debugging this using a hMailServer SMTP server I’m beginning to think this may be a bug. This issue is causing issues like the SMTP server not DKIM signing the message since the sender domain doesn’t match the from header and also being flagged as potential spam by www.mail-tester.com for the same reason.

So I did some more testing to see what’s going on. I did 2 tests with the same setup from the same network and both are setup with the same SMTP server name, username/password, port and authentication. One machine running outlook and another discourse, sent a test eMail to the same email address from discourse and from outlook.

In outlook I set the “reply-to” feature similar to what discourse is is supposed to set for a reply to address. (using gmail here for the reply-to)

I looked at the SMTP server records and I noticed that the from header from outlook was set correctly.

“RECEIVED: MAIL FROM: discussion@xxxx.com”

However the logs from the discourse transaction shows this for the from header:

“RECEIVED: MAIL FROM: my.discussion+verp-eb90be8c2b93e18896c910d4fe2cdc05@gmail.com”

That’s the root cause why receivers are flagging as spam and why the SMTP server won’t DKIM sign the message sent from discourse since the from address doesn’t match the senders domain.

Why is discourse setting the from flag with the reply-to address?

[RBoy]

I thinking it has something to do with this piece of code in sender.rb line 164

       # WARNING: RFC claims you can not set the Return Path header, this is 100% correct
        # however Rails has special handling for this header and ends up using this value
        # as the Envelope From address so stuff works as expected
        @message.header[:return_path] = SiteSetting.reply_by_email_address.sub("%{reply_key}", "verp-#{email_log.bounce_key}")

I’m wondering if this header should be set:

What is the behavior difference between return-path, reply-to and from?

This seems to indicate that the return_path should not be set by the client software but instead by the mail server

Only the recipient’s mail server is supposed to add a Return-Path header to the top of the email

EDIT: From the Rails documentation:

https://api.rubyonrails.org/v5.1.6/classes/ActionMailer/Base.html

When a :return_path is specified as header, that value will be used as the ‘envelope from’ address for the Mail message. Setting this is useful when you want delivery notifications sent to a different address than the one in :from . Mail will actually use the :return_path in preference to the :sender in preference to the :from field for the ‘envelope from’ value.

This could explain why the mail servers are using the reply-to instead of the from while sending the eMail. It looks like this is causing the issue and should probably not be set.

[RBoy]

I’ve submitted a patch for this. Essentially if the reply-to domain and the from domain don’t match, then discourse shouldn’t use the return-path header as this will cause a DKIM/SPF failure otherwise the message may be flagged a spam/phishing address since the senders domain won’t match the from domain.

https://github.com/discourse/discourse/pull/6718

[mpalmer (Matt Palmer)]

@RBoy

Why is discourse setting the from flag with the reply-to address?

It… isn’t? The envelope FROM address is being set to the VERP bounce handling address, just as it should be.

Your anti-spam problems are all down to trying to send e-mail as coming from a provider which doesn’t support sending e-mail from alternate locations. Perhaps consider not doing that?

@RBoy

Essentially if the reply-to domain and the from domain don’t match, then discourse shouldn’t use the return-path header as this will cause a DKIM/SPF failure

Nope. It works Just Fine. We (CDCK) send e-mail this way for some of our biggest Enterprise customers, and believe me, if it wasn’t working, they’d be all over us. Your problems are due to your choice of mail provider, and possibly other misconfigurations in your SPF/DKIM setup.

[RBoy]

@mpalmer

Your anti-spam problems are all down to trying to send e-mail as coming from a provider which doesn’t support sending e-mail from alternate locations. Perhaps consider not doing that?

RBoy:

I’m running hMailServer on my own domain and it’s working perfectly as per the RFC’s and www.mail-tester.com (the official spam test server for Discourse) has verified it a safe and compliant server, everything including SPF, DKIM and DMARC. No issues here.

@mpalmer

Your problems are due to your choice of mail provider, and possibly other misconfigurations in your SPF/DKIM setup.

Nope, and I’m not the only one who reported it, see my links above. The reason you don’t probably see if from your enterprise customers is because they’re their down domain for a reply to address. This bug only affects Discourse when it’s setup to use a reply to address on a different domain.

If one is to follow the guidelines as outlined in this guide to use gMail as a reply to inbox: Set up reply by email with POP3 polling then the emails sent by Discourse are in violation of the RFC standards.

As per the standard:

Only the recipient’s mail server is supposed to add a Return-Path header to the top of the email

Infact even the comment in the code says it’s violating the RFC standard as a “hack” because of the oddity of the way Ruby works:

      # WARNING: RFC claims you can not set the Return Path header, this is 100% correct
        # however Rails has special handling for this header and ends up using this value
        # as the Envelope From address so stuff works as expected

From the Ruby docs:

When a :return_path is specified as header, that value will be used as the ‘envelope from’ address for the Mail message. Setting this is useful when you want delivery notifications sent to a different address than the one in :from . Mail will actually use the :return_path in preference to the :sender in preference to the :from field for the ‘envelope from’ value.

I mean it’s pretty clear that the code is violating the RFC standard and because of which the From header is being set incorrectly under certain conditions. The From header domain should NEVER be different from the senders domain. That by the very definition is how spammers/phishing servers work.

If Discourse is going to have a reply-to from a different domain that should NEVER be set in the from address and as per the Ruby docs and code comments above, that’s what’s it doing.

The issue here is that there’s no sanity check being done while trying to use the Ruby “hack” for the Return-Path:

• The VERP bounce handling address should be set in the Reply-To header, which is being done correctly and this should be enough
• If Discourse also wants to set this in the From header, which is acceptable if it’s the same domain, it needs for first verify that the senders domain is the same the reply to domain. Otherwise the guide published above on how to use GMail as a inbox for Discourse is violating the essence of spam management and no compliant SMTP server will DKIM sign the message.

EDIT: FYI, I have verified this by setting the Reply-To email as one from own domain and then the SMTP server DKIM signs the message and even www.mail-tester.com verifies that the message is complaint and not spam. This is because even with the VERP bounce address handling the From domain matches the senders domain and hence it’s a verifiable message with a clean chain of command. So the SMTP configuration is correct. You can also refer to the SMTP server logs showing the Discourse is sending a from address which doesn’t match the senders domain, why would a server DKIM sign such a message?

[mpalmer (Matt Palmer)]

@RBoy

Only the recipient’s mail server is supposed to add a Return-Path header to the top of the email

Do the e-mails as sent over the wire from Discourse contain a Return-Path header? I’m pretty sure they don’t. That the way to set the envelope FROM in Rails is to seemingly set the Return-Path header in the code is unfortunate, but not standards-breaking as far as I’m aware.

@RBoy

The From header domain should NEVER be different from the senders domain

“Citation needed”, as the wikipedeans say.

@RBoy

The VERP bounce handling address should be set in the Reply-To header

And how, prey tell, is an SMTP bounce going to know to look in the Reply-To header?

@RBoy

You can also refer to the SMTP server logs showing the Discourse is sending a from address which doesn’t match the senders domain, why would a server DKIM sign such a message?

Because that’s how DKIM works. The phrase you’re looking for is “DKIM alignment”.

[RBoy]

@mpalmer

That the way to set the envelope FROM in Rails is to seemingly set the Return-Path header in the code is unfortunate, but not standards-breaking as far as I’m aware.

So you’re saying that it’s okay to have a From address domain that doesn’t belong to the sender? Isn’t that the definition of spam?

[mpalmer (Matt Palmer)]

@RBoy

So you’re saying that it’s okay to have a From address domain that doesn’t belong to the sender?

That question is insufficiently precise to be answerable in a reasonable volume of electrons.

@RBoy

Isn’t that the definition of spam?

No, the definition of spam is “unsolicited bulk or commercial e-mail”.

[RBoy]

@mpalmer

No, the definition of spam is “unsolicited bulk or commercial e-mail”.

A message that cannot be authenticated or appears to be deceptive in nature which is then flagged by an antispam message system is considered spam (the adage, if it’s looks like X, smells like X it’s probably X). That’s what happens with eMail and spam filters.

So when sending a message from mysite.com with the from header saying gmail.com and when www.mail-tester.com says, sniff, sniff, something ain’t right here - you aren’t authorized to represent yourself as gmail.com, I’m going to flag you a spam; it’s going to end up in a spam mailbox for many users.

So whatever the intent might have been, discourse is now sending emails which are likely to be flagged as spam for domain impersonation. Again don’t take my word for it, take www.mail-tester.com results

The purpose of VERP is to help reduce undelivered/bounced email. The side of effect of the way it’s implemented in discourse is that it’s setting off anti spam filters under certain conditions, so instead of trying to track down undelivered emails now you’re ending up in the spam folder.

If Discourse verifies that the sender domain matches the from envelope than VERP may actually work instead of ending up in a spam folder.

Yes, folks who’re using a different domain for reply to addresses (like gmail as prescribed by Discourse) won’t get the benefit of VERP but that’s a lot better than having emails end up in a spam folder.