When turning on emails for every post to a forum, the emails do not thread correctly, at least in Outlook 2013. It looks like there’s an attempt to make it work, based on the In-Reply-To headers, but somehow it’s still not working :(. I imagine Outlook has some strange heuristics you might have to appease…
Can you provide information on these heuristics?
I don’t know them myself, and can’t find anything with a quick Google. But, comparing to GitHub notification emails (which are correctly threaded), here are some differences I notice:
- Discourse emails have different Sender headers for each email, whereas GitHub emails do not have a Sender header. (This manifests in the UI via Outlook’s “$Sender on behalf of $From” for Discourse, whereas GitHub just uses “$From”)
- GitHub emails have List-Post headers; Discourse emails do not.
- Discourse emails have different Reply-To addresses; GitHub emails use the same one for each thread. (Both use different Reply-To names.)
I put together a gist with the headers of two Discourse emails in the same thread, plus two GitHub emails in the same thread, which can be analyzed further. I’ll try tweeting to see if anyone can connect me with the Outlook team as well for a more definitive answer.
7 Likes
Great stuff. I’d really like to fix this once we figure out exactly what we need to change.
There are a lot of List- headers here:
List-ID: felixge/node-sandboxed-module
<node-sandboxed-module.felixge.github.com>
List-Archive: https://github.com/felixge/node-sandboxed-module
List-Post: <mailto:reply+i-21866655-GUID-617481@reply.github.com>
List-Unsubscribe: <mailto:unsub+i-21866655-GUID-617481@reply.github.com>,
<https://github.com/notifications/unsubscribe/617481__GUID>
The only one we have is
List-Unsubscribe: <http://discourse.specifiction.org/my/preferences>
We can certainly add these headers:
List-ID, List-Archive, List-Post
It looks like:
List-ID = topic title, e.g. URL fragment? /t/topic-title/1234
List-Archive = topic URL
List-Post = duplicate the email reply address
The difference in Reply-To could also help?
Reply-To: felixge/node-sandboxed-module
<reply+i-21866655-GUID-617481@reply.github.com>
versus
Reply-To: domenic <discourse+GUID@specifiction.org>
2 Likes
Ok I added
List-ID
Which was oddly in the code already as a function, but not present on any outgoing emails at all, anywhere, on any site, for… reasons I don’t understand. It was also in the code as lower case List-Id, whereas GitHub puts it in as upper case List-ID. At any rate, this header was not present on any Discourse emails, which I think is a big, big problem.
In the case that email replies are enabled, I also added
List-Post
which is simply a copy of the Reply-To header the way GitHub does it, so we’ll copy that header into that field if email replies are enabled.
I need @eviltrout’s help to build List-Archive since I can’t figure out a way to get to the current URL of the topic in that part of the code.
It looks like Mandrill may be interfering here, and stripping List-ID and List-Post. I verified the header arrives in the logged API call on their end, but it never arrives on the client… I also opened a support ticket on our paid Mandrill account.
2 Likes
Here is a gist with headers from a YahooGroups email and reply.
https://gist.github.com/peternlewis/cca726bbb8ea7e5b458e
Interesting candidate headers include:
Sender: keyboard_maestro@yahoogroups.com
Mailing-List: list keyboard_maestro@yahoogroups.com; contact keyboard_maestro-owner@yahoogroups.com
Delivered-To: mailing list keyboard_maestro@yahoogroups.com
List-Id: <keyboard_maestro.yahoogroups.com>
Precedence: bulk
List-Unsubscribe: <mailto:keyboard_maestro-unsubscribe@yahoogroups.com>
1 Like
Apparently In-Reply-To is another possibly important header. http://www.ietf.org/rfc/rfc2822.txt documents in detail what the References and In-Reply-To headers are meant to be, but basically as near as I can figure it:
In-Reply-To: lists the message ids of all the messages this reply is replying to (so whatever message was selected for the “Reply” button, plus any message quoted. Just the former would probably be sufficient.
References: lists the parent’s References header followed by the parent’s message ID.
If you look at ours currently, it looks like this:
Message-ID: <539a986f8300_5a3fae28acd8c8201e9@forum.mail>
In-Reply-To: <topic/96@forum.keyboardmaestro.com>
References: <topic/96@forum.keyboardmaestro.com>
and
Message-ID: <539a9904d40bd_5a3fae28d5b8242036d@forum.mail>
In-Reply-To: <topic/96@forum.keyboardmaestro.com>
References: <topic/96@forum.keyboardmaestro.com>
which is clearly wrong. The first one was the original PM, so should not have an In-Reply-To or References header, and the second one (the reply, should look like this:
Message-ID: <539a9904d40bd_5a3fae28d5b8242036d@forum.mail>
In-Reply-To: <539a9904d40bd_5a3fae28d5b8242036d@forum.mail>
References: <539a986f8300_5a3fae28acd8c8201e9@forum.mail>
And a reply to that reply should look like this:
Message-ID: <adsassadfdfsdfs_sddfsdfsaererfgdsf@forum.mail>
In-Reply-To: <539a986f8300_5a3fae28acd8c8201e9@forum.mail>
References: <539a986f8300_5a3fae28acd8c8201e9@forum.mail> <539a9904d40bd_5a3fae28d5b8242036d@forum.mail>
Does that help?
1 Like
All topic replies are conceptually “in reply to” the topic, though.
Do you mean that the first posted message is a reply to the (empty, created) topic, or do you mean that all following replies are replies to the original message?
Either way, the “References” and “In-Reply-To” list message ids, not topics. They need to list the message IDs of previous emails.
The first post (post_number=1) should not have References or In-Reply-To headers because there is no previously existing message ID.
All following replies could simply be Reference: , In-Reply-To . That would be fine (minimal, but fine) as far as I can see.
If you want the References and In-Reply-To headers to be “topic/134@domain.name”, that would be fine, but only if you make sure that first email has message id “topic/134@domain.name” which might be ok, but is probably a bit dangerous. Otherwise, topic/134@domain.name wont be a message ID of any existing email message so it’s not going to be legitimate.
And on the topic of Mail.app, I’ve done some more testing, and as far as I can see the “Re:” is by far the most important thing for its threading. It will generally thread the emails together as long as the subjects have “Re:”, and will not thread them together if they don’t no matter what headers they have as far as I can tell.
2 Likes
Confirmed
Thanks for generating that email and sending it to us. I appreciate your diligence and effort. In this case it does look like these headers are getting stripped out when we break apart the SMTP message and convert it to an API call through our SMTP Integration. We’ve previously added support for a few common headers as we’ve continued to develop our product, but at this time we don’t have support for the List-ID and List-Archive headers. These might be ones that we can similarly allow through, but I don’t have a timeframe for when that may be implemented.
I have shared all of this with our developers so that they are aware of the need for this and can work on incorporating that feature.
We need to switch away from recommending Mandrill while this is the case. Absence of these List-* headers can break threading in mail clients, depending on the mail client of course…
But then they just replied
Thanks for getting back to us! We should actually be able to push a change pretty shortly to add support for these headers, and I’m glad to let you know once that change has gone live.
So maybe playing the Blogger Jackass Card worked in this case.
update:
The changes are complete. The List-ID and List-Archive headers will no longer be stripped out during conversion of your SMTP messages.
Yay! So hopefully this will start working for all sites that use Mandrill now…
2 Likes
Getting back to the subject of the topic: can anyone confirm that, with changes since June, email notifications from Discourse are properly threaded in Outlook 2013?
1 Like
Trying to reply to this post to see what happens…
(EDIT: ok, I forgot to turn on email notifications wnen visiting the site recently for meta.discourse.org before doing this, apparently. So instead of spamming this thread twice more, I’ll just subscribe to the currently-most-active thread and report back.)
I can confirm that the emails are still not properly threaded in Outlook 2013 
New gist at Discourse headers for 2014-10-13 · GitHub
Still no List-Post header, although who knows if that’s the important one.
I wasn’t originally subscribed to the topics I just subscribed to, so it’s possible if you guys are using the “In-Reply-To the top post” strategy (instead of the “In-Reply-To most recent post on the thread”) threading won’t work unless you receive the email for the first post. I’ve stayed subscribed to all of meta.discourse.org so if that happens to be the case we should know after a few posts.
I wonder if the "Re: " subject prefix is important!?
No idea… someone with knowledge of Outlook 2013 internals will need to step up, otherwise it’ll just be the way it is.
From your gist, here are the current relevant headers that I know of:
Subject: [Discourse Meta] [feature] Possibility to selectively (or completely) mark notifications as read
From: Mittineague <info@discourse.org>
Reply-To: Discourse Meta <reply+29fb8d02de845c19eb575be161de8855@discourse.org>
In-Reply-To: <topic/20227@meta.discourse.org>
References: <topic/20227@meta.discourse.org>
Auto-Submitted: auto-generated
List-ID: <feature.meta.discourse.org>
List-Archive: https://meta.discourse.org/t/possibility-to-selectively-or-completely-mark-notifications-as-read/20227
List-Unsubscribe: <https://meta.discourse.org/my/preferences>
Precedence: list
One thing I did notice , comparing your GitHub email header gists:
Reply-To: felixge/node-sandboxed-module
<reply+i-21866655-ede66648bf2cdb8a3707a09c0e03d118d318ad1e-617481@reply.github.com>
versus
Reply-To: Discourse Meta <reply+29fb8d02de845c19eb575be161de8855@discourse.org>
@techapj can you improve us here so we are using topic/20227 instead of Discourse Meta here as the “friendly” reply-to name here, like GitHub does?
Couple other worries about magical incantations needed:
- Re: in the email title? So ghetto, but we do now support it (off by default).
- Message-ID: might need to be something other than the random generic technobabble string our mail server seems to slap on there. Is it safe for us to include this header @michael?
Where is that setting? I can’t find it in user preferences at least, and although I am an admin for discourse.specifiction.org, I don’t see any real adjustable settings there.
Happy to report that since upgrading to the latest Discourse and turning on the optional Re: prefix, the emails are now threading.
2 Likes
Well, it does not really work in every client:
Roundcube:
KMail:
Thunderbird:
Thunderbird seems to grab the "Re: " prefix and the other clients don’t. As the In-Reply-To headers are wrong these clients could not do any threading. As mentioned before the specs are here (chapter 3.6.4): http://www.ietf.org/rfc/rfc2822.txt
I have edited the In-Reply-To and References Headers manually according to the specs. I have replaced these topic links with the appropriate message IDs. Suprise, surprise, it threads well in every client:
KMail:
Roundcube:
@codinghorror: Have a look at rfc2822 and @peternlewis post above. That’s the real issue with its intended solution. Adding an Re: prefix is just a workaround for some clients.