```
Again, an ok `Message-ID`, but completely nonsensical `In-Reply-To` and `References`.
This _should_ be easy to fix. The first message should have neither `In-Reply-To` nor `References` headers. The second message should have the first message’s `Message-ID` in the `In-Reply-To` and `References` headers.
Please see RFC5322 section 3.6.4 for specifics:
> **[Home](https://authors.ietf.org/)**
>
> Welcome to the Internet-Draft/RFC author resources site
As things are, email users see flat unstructured discussions. With these fixes, they can have sensible easy to follow threaded display.
---
### Author:  [@ericsnowcurrently](https://meta.discourse.org/u/ericsnowcurrently)
#### Post date: [July 21, 2022, 3:26pm UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/3 "2022-07-21T15:26:17Z")
In case anyone is interested, the archive of the discussion to which Cameron is referring is found at [https://mail.python.org/archives/list/python-dev@python.org/message/VHFLDK43DSSLHACT67X4QA3UZU73WYYJ/](https://mail.python.org/archives/list/python-dev@python.org/message/VHFLDK43DSSLHACT67X4QA3UZU73WYYJ/).
---
### Author:  [@RGJ](https://meta.discourse.org/u/RGJ)
#### Post date: [July 21, 2022, 3:59pm UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/4 "2022-07-21T15:59:13Z")
That seems to be a regression, see this [old topic and the fix](https://meta.discourse.org/t/discourse-emails-not-threaded-properly-in-some-email-clients/44444/12).
---
### Author:  [@cameron-simpson](https://meta.discourse.org/u/cameron-simpson)
#### Post date: [July 21, 2022, 10:50pm UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/5 "2022-07-21T22:50:05Z")
Just having a look at the diff between HEAD and that fix.
It seems to me that current still always sets `References`, even if there’s no antecedant - the `topic_canonical_reference_id` is used as a fallback. I still think that’s wrong, because there is no email message with that id.
The `In-Reply-To` is a little more correct, in that it is only set if `post.post_number!=1`, but it still falls back to `topic_canonical_reference_id`:
```
@message.header['In-Reply-To'] = referenced_post_message_ids[0] || topic_canonical_reference_id
```
This seems to have 2 problems to my eye:
- the fallback should be the `Message-ID` of post #1 if there are no `referenced_post_message_ids`, and _not_ `topic_canonical_reference_id`
- something in the receipt-of-reply-emails code must be dropping the `In-Reply-To` header of the reply messages, because they should have correctly populated the `referenced_post_message_ids` array (“list”? I’m new to Ruby)
---
### Author:  [@martin](https://meta.discourse.org/u/martin)
#### Post date: [July 22, 2022, 12:25am UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/6 "2022-07-22T00:25:23Z")
Cameron, thanks for opening up this topic for discussion and providing a lot of detail in your posts. I am responsible for this can of worms, from these two commits:
[https://github.com/discourse/discourse/commit/3b13f1146b2a406238c50d6b45bc9aa721094f46](https://github.com/discourse/discourse/commit/3b13f1146b2a406238c50d6b45bc9aa721094f46)
[https://github.com/discourse/discourse/commit/82cb67e67b83c444f068fd6b3006d8396803454f](https://github.com/discourse/discourse/commit/82cb67e67b83c444f068fd6b3006d8396803454f)
We have been aware of some issues around threading for a little while now in email clients such as Thunderbird but it has not represented a large number of consumers of email threading from Discourse so it’s been punted on, but now this is coming to light we need to spend some time reexamining the issue and working on a fix.
> [@cameron-simpson](#):
>
> It is the first message. It should not have a `References` header, because there’s no message anywhere with that id.
Interestingly, we added this `References` header to the first sent email and every subsequent one at the time since it makes threading work correctly in Gmail, but I agree it’s not ideal and is likely causing the threading issues along with not using the original `Message-ID` in the subsequent email `In-Reply-To` and `References` headers.
Please bear with me as I look through old discussions and the code and work through this. In the meantime, are you aware of other email clients that are being used and are experiencing issues? For example I know that this is an issue in Thunderbird, but what about any others? Thanks.
---
### Author:  [@cameron-simpson](https://meta.discourse.org/u/cameron-simpson)
#### Post date: [July 22, 2022, 3:27am UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/7 "2022-07-22T03:27:39Z")
Wrote a long reply, but got:
```
We're sorry, but your email message to
["incoming+8349bd9eb1f2b582df4f32dbe85c3363@meta.discoursemail.com"]
(titled Re: [Discourse Meta] [bug] Discourse email messages are
incorrectly threaded) didn't work.
Reason:
Sorry, new users can only put 2 links in a post.
If you can correct the problem, please try again.
```
I’ll go put in in in the forum where I can catch and revise…
---
### Author:  [@cameron-simpson](https://meta.discourse.org/u/cameron-simpson)
#### Post date: [July 22, 2022, 3:34am UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/8 "2022-07-22T03:34:14Z")
> Cameron, thanks for opening up this topic for discussion and providing
> a lot of detail in your posts. I am responsible for this can of worms,
> from these two commits:
>
> 3b13f1146b2a406238c50d6b45bc9aa721094f46
This looks fine. Does it save this id with the db record so that inbound
replies can be tied to the antecedant forum message?
Also, do you want me to vet that the suffix is syntacly legal for
RFC5322, in terms of permitted characters?
> 82cb67e67b83c444f068fd6b3006d8396803454f
This second commit seems to address another problem we have seen: if a
post comes from an email, the outbound message-id sent to email users is
not the message-id of the source message from the author. This results
in two different messages from the point of view of a mail client, and
probably breaks replies made to the original as opposed to the
forum-sent copy. For example:
```
To: the forum
CC: one of the participants
```
The partitpant will (well, may) receive a copy from the forum and a
direct copy from the author, and these will be distinct messages at
their end because they will have different message-ids.
I was going to make a second bug report about this issue after sorting
the in-reply-to and references headers issue, which is far more
important.
> We have been aware of some issues around threading for a little while now in email clients such as Thunderbird but it has not represented a large number of consumers of
> email threading from Discourse so it’s been punted on, but now this is coming to light we need to spend some time reexamining the issue and working on a fix.
I and several others use mutt. I’m happy to do whatever is needed to aid
in debugging this and reviewing code. I’ve also been a mail sysadmin for
yonks in former lives.
> [quote=“Cameron Simpson, post:1, topic:233499,
> username:cameron-simpson”]
> It is the first message. It should not have a `References` header, because there’s no message anywhere with that id.
> [/quote]
>
> Interestingly, we added this `References` header to the first sent email and every subsequent one at the time since it makes threading work correctly in Gmail,
I think a correct `References` header (absent in the first post, like
in-reply-to in replies) should also work. But GMail has a rather loose
relationship with mail standards at times. I have a gmail accord; I can
do some debugging there too. And in principle we can use this very
discussion as the test bed, maybe.
> but I agree it’s not ideal and is likely causing the threading issues
> along with not using the original `Message-ID` in the subsequent email
> `In-Reply-To` and `References` headers.
>
> Please bear with me as I look through old discussions and the code and work through this.
No worries.
> In the meantime, are you aware of other email clients that are being
> used and are experiencing issues? For example I know that this is an
> issue in Thunderbird, but what about any others? Thanks.
Definitely mutt. At least with mutt is it very easy to see the headers
and also to see the reply tree chain, which is often obscured in other
clients.
Mail threading is entirely defined by the `Message-ID` and `In-Reply-To`
headers. The `References` header started with USENET for followups, and
supported (there) multiple message-ids; the `In-Reply-To` supports just
one. It looks like `References` is now also present in RFC5322, and I’ll
check into its semantics.
---
### Author:  [@martin](https://meta.discourse.org/u/martin)
#### Post date: [July 22, 2022, 4:41am UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/9 "2022-07-22T04:41:24Z")
I am just collecting my thoughts in a big post about this for later today, thank you for the extra information so far!
---
### Author:  [@martin](https://meta.discourse.org/u/martin)
#### Post date: [July 22, 2022, 6:24am UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/10 "2022-07-22T06:24:31Z")
Okay this is kind of huge, please bear with me. First, thanks for another detailed reply and the offer of debugging / review, it is really helpful 👍 I’ve actually been looking into this this morning and, surprisingly, the threading in a unified view works in Thunderbird for **most cases** , and I think the `References` header consistently pointing to the OP helps with that (for example the topic `Reference` in this chain which is always present is ``.
The case where the threading does not working as intended is:
1. A post is created within discourse and an email is sent out to those watching the topic _then_
2. Someone else replies to that post and an email is sent out to those watching the topic
In the case of the second email, it gets an incorrect `In-Reply-To` and `References` header since it generates one on this line [discourse/lib/email/sender.rb at 98bacbd2c6b9fe57167cd32af5eb4839b4a5d1f6 · discourse/discourse · GitHub](https://github.com/discourse/discourse/blob/98bacbd2c6b9fe57167cd32af5eb4839b4a5d1f6/lib/email/sender.rb#L152) rather than using an existing one. It should be using the `Message-ID` for the email that was sent first. In the screenshot, this is where the messages following this pattern _should_ be placed:
> [@cameron-simpson](#):
>
> This looks fine. Does it save this id with the db record so that inbound
> replies can be tied to the antecedant forum message?
The answer is – it depends. If a post is created in Discourse from an inbound email, such as [this one](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/7) of yours, we use that post’s original inbound `Message-ID` when someone replies to it for the `In-Reply-To` and `References` headers as per:
>
Otherwise we are just using the topic OP reference and just generating a new reference, which obviously is what is causing all the issues. In all cases we generate a new `Message-ID` every time an outbound email is sent, which seems correct and on par with other mail clients.
> [@cameron-simpson](#):
>
> This second commit seems to address another problem we have seen: if a
> post comes from an email, the outbound message-id sent to email users is
> not the message-id of the source message from the author. This results
> in two different messages from the point of view of a mail client, and
> probably breaks replies made to the original as opposed to the
> forum-sent copy.
I think I see what you mean, does it go like this:
1. cameron sends email to Discourse from mutt which gets `Message-ID: 74398756983476983@mail.com`
2. Discourse creates a post and stores the `Message-ID` with against the post with an `IncomingEmail` record
3. johndoe is watching the topic, so they get sent an email from Discourse with a `Message-ID: topic/222/44@discourse.com` and no reference to the original `Message-ID: 74398756983476983@mail.com`
Does that sound correct, that we should just “pass on” that `Message-ID` to those watching the topic instead of generating our own since it’s already unique? What then happens in `johndoe`’s mail client if
cameron also CC’d him on that original outbound message? This does sound like a separate issue so it would be good to open another bug topic for it.
> [@cameron-simpson](#):
>
> I and several others use mutt.
I will set up a mutt client locally to see what you are also seeing, I have never tested this functionality in a text-based client (only Gmail and Thunderbird) so I am keen to see how it looks anyway.
* * *
My line of thinking to address these issues this morning was to dispose with the randomly generated suffixes generated when we send `Message-ID` headers in emails and instead change to a scheme where we use the `user_id` of both the sending and receiving user. The benefit of this is that there is no need to store the `Message-ID` anywhere (apart from when an inbound email creates a post) and so `References` and `In-Reply-To` headers will always be consistent. Let me give an example. Say we have these users:
- martin - user\_id 25
- cameron - user\_id 44
- sam - user\_id 78
- bob - user\_id 999
And then we have this topic, topic\_id 233499, with posts starting from post\_id 100 as the OP. The format would become `topic/#{topic_id}/#{post_id}.s#{sender_user_id}r#{receiver_user_id}`. The order of operations would look like this:
1. **martin** creates the OP
- **cameron** is sent an email with these headers:
- `Message-ID: topic/233499.s25r44@meta.discourse.org`
- `References: topic/233499@meta.discourse.org`
- **sam** is sent an email with these headers:
- `Message-ID: topic/233499.s25r78@meta.discourse.org`
- `References: topic/233499@meta.discourse.org`
1. **cameron** replies via email
- **discourse** is sent an email with these headers from mutt:
- `Message-ID: 43585349859734@test.com`
- `References: topic/233499@meta.discourse.org topic/233499.s25r44@meta.discourse.org`
- `In-Reply-To: topic/233499.s25r44@meta.discourse.org`
1. **discourse** (as **cameron** , from the above email) creates post 101
- **sam** is sent an email from discourse with these headers:
- `Message-ID: topic/233499/101.s44r78@meta.discourse.org`
- `References: 43585349859734@test.com topic/233499@meta.discourse.org`
- `In-Reply-To: 43585349859734@test.com`
1. **sam** replies via email to cameron
- **discourse** is sent an email with these headers from gmail:
- `Message-ID: 5346564746574@gmail.com`
- `References: topic/233499/101.s44r78@meta.discourse.org topic/233499@meta.discourse.org`
- `In-Reply-To: topic/233499/101.s44r78@meta.discourse.org`
1. **discourse** (as **sam** , from the above email) creates post 102
- **cameron** is sent an email from discourse with these headers:
- `Message-ID: topic/233499/102.s78r44@meta.discourse.org`
- `References: 5346564746574@gmail.com topic/233499@meta.discourse.org`
- `In-Reply-To: 5346564746574@gmail.com`
1. **bob** creates post 103 in the topic, not in reply to anyone (note that the References here includes the `Message-ID` sent to both users for the OP email)
- **cameron** is sent an email with these headers:
- `Message-ID: topic/233499/103.s999r44@meta.discourse.org`
- `References: topic/233500@meta.discourse.org topic/23499.s25r44@meta.discourse.org`
- **sam** is sent an email with these headers:
- `Message-ID: topic/233499/103.s999r78@meta.discourse.org`
- `References: topic/233499@meta.discourse.org topic/23499.s25r78@meta.discourse.org`
1. **cameron** replies via email
- **discourse** is sent an email with these headers from mutt:
- `Message-ID: 6759850728742572@test.com`
- `References: topic/233499@meta.discourse.org topic/233499/103.s999r44@meta.discourse.org`
- `In-Reply-To: topic/233499/103.s999r44@meta.discourse.org`
cameron’s inbox
- martin - topic OP
- SENT → to: discourse, RE: topic OP
- sam - reply to second post
- bob - reply in topic not to any particular post
- SENT → to: discourse, RE: bob’s post
sam’s inbox
- martin - topic OP
- cameron - second post
- SENT → to: discourse, RE: second post
- bob - reply in topic not to any particular post
I think this is correct, can you just take a look over what I have written in these headers and verify that is what you would expect from this scenario? The only thing I am a little unsure about is whether I have covered all the References, and of course I would be testing this on a live set of emails in a dev branch before rolling it out. I have not tested anything in mutt yet either.
* * *
As a side note, I also looked into what GitHub do with their notification emails, and noticed they do a similar thing where they have an ever-present `Reference` (`discourse/discourse/pull/252@github.com`) that is used in all the emails related to that “topic” which in this case is a GitHub pull request:
```plaintext
References:
In-Reply-To:
```
---
### Author:  [@cameron-simpson](https://meta.discourse.org/u/cameron-simpson)
#### Post date: [July 23, 2022, 12:44am UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/11 "2022-07-23T00:44:07Z")
By Martin Brennan via Discourse Meta at 22Jul2022 06:34:
> Okay this is kind of huge, please bear with me. First, thanks for
> another detailed reply and the offer of debugging / review, it is
> really helpful 👍 I’ve actually been looking into this this morning
> and, surprisingly, the threading in a unified view works in Thunderbird
> for **most cases** , and I think the `References` header consistently
> pointing to the OP helps with that (for example the topic `Reference`
> in this chain which is always present is
> ``.
I’ve just reread RFC5322 section 3.6.4 closely. It has moved on from
earlier versions (822 and 2822), and has merged the email `In-Reply-To`
headers, USENET `References` headers and modern
reply-citing-more-that-one previous messages.
The short summary:
- The `Message-ID` is a single persisent identifier for a message
- The `In-Reply-To` contains all the message-ids of which this message
is a direct reply, so if I reply to a pair of messages it will have
those 2 message-ids
- The `References` is a reply chain of antecedant message-ids from the
OP to the preceeding message. So indeed it should _always_ start with
the OP message-id.
So for a discussions like this, pretending that labels are message-ids:
```
OP
-> reply1
-> reply2 ---+
-> reply3 |
-> reply4 |
-> reply5 <+
```
The reply5 would have:
- message-id=reply5
- in-reply-to=“reply2 reply4”
- references=“OP reply3 reply4”
It is also leagel to include “reply1 reply2” in the references (the
other chain to reply5) but the RFC explicitly recommends against that
becaause some clients expect the references to be a single linear chain
of replies, not some flattened digraph.
So my recommendation for constructing the references is to use the
references of the “primary” antecedant message with the primary
antecedant message’s message-id appended. That way you always get a
linear chain in the correct order.
> 
Interestingly there seems to be some threading there.
But notice: the top post has a little “is a reply” arrow. Even though it
is post 1. I expect that is because of the “topic” references entry,
which make TB think there was a earlier message (which of course there
was not).
In mutt-land we see almost no threading at all:
```
23Jul2022 06:24 Olha via Discus - ┌>[Py] [Users] I need an advise discuss-users 5.7K
22Jul2022 17:12 Paul Jurczak vi - ├>[Py] [Users] I need an advise discuss-users 5.5K
22Jul2022 13:21 Rob via Discuss - ├>[Py] [Users] I need an advise discuss-users 6.8K
22Jul2022 12:53 vasi-h via Disc - ├>[Py] [Users] I need an advise discuss-users 5.5K
22Jul2022 11:38 Cameron Simpson - ├>[Py] [Users] I need an advise discuss-users 14K
22Jul2022 10:27 Rob via Discuss - ├>[Py] [Users] I need an advise discuss-users 6.6K
22Jul2022 06:14 vasi-h via Disc r ┴>[Py] [Users] I need an advise discuss-users 6.5K
```
which is because every message’s `In-Reply-To` points directly at the
fictitious “topic” message-id. Mutt probably ignores the `References`
because it is a mail reader, and `References` originates in USENET news.
Maybe Thunderbird is using the references or augumenting the in-reply-to
with references information.
You only need to consult one of `In=-Reply-To` or `References` to do
threading; the former comes from email and the latter from USENET.
You’re supporting both (which is great!) so we need to make them
consistent.
(Aside: there’s also discussion about USENET mirroring, because several
python people consume the lists via a USENET interface. Again, a
separate topic.)
[…]
> [quote=“Cameron Simpson, post:8, topic:233499,
> username:cameron-simpson”]
> This looks fine. Does it save this id with the db record so that inbound
> replies can be tied to the antecedant forum message?
> [/quote]
>
> The answer is – it depends. If a post is created in Discourse from an inbound email, such as [this one](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/7) of yours, we use that post’s original inbound `Message-ID` when someone replies to it for the `In-Reply-To` and `References` headers as per:
>
> [discourse/lib/email/sender.rb at 98bacbd2c6b9fe57167cd32af5eb4839b4a5d1f6 · discourse/discourse · GitHub](https://github.com/discourse/discourse/blob/98bacbd2c6b9fe57167cd32af5eb4839b4a5d1f6/lib/email/sender.rb#L146-L147)
>
> Otherwise we are just using the topic OP reference and just generating a new reference, which obviously is what is causing all the issues. In all cases we generate a new `Message-ID` every time an outbound email is sent, which seems correct and on par with other mail clients.
Alas, not quite. If you’re the origin of the message (i.e. authored in
Discourse), generating the message-id is fine. If there’s no message-id
(illegal) generating one is standard practice (usually by MTAs). But if
you’re passing a message on (authored in email), the existing message-id
should be preserved.
To my mind you need to be doing 3 things:
1. having a stable message-id and not replacing the message-id from an
inbound message
2. generating correct `In-Reply-To`, which is easily computed from the
immediate antecedant message(s) i.e. antecedant(s)-`Message-ID`
3. generating correct `References`, which is easily computed as
antecedant-`References` + antecedant-`Message-ID`
For point 1, looking at the code you cite, you probably want the email
message id to be (Pythonish syntax, sorry):
```
def message_id(post):
return post.incoming_email.message_id or discourse_message_id(post)
```
i.e. to be the post’s email message-id if it originated from email,
otherwise the Discourse message-id using something like the algorithm
you outline later in this message: anything (a) stable and (b)
syntacticly valid.
Then computing the `In-Reply-To` and `References` fields is simple
mechanical stuff as in points 2 and 3.
> > [@cameron-simpson](#):
> >
> > This second commit seems to address another problem we have seen: if a
> > post comes from an email, the outbound message-id sent to email users is
> > not the message-id of the source message from the author. This results
> > in two different messages from the point of view of a mail client, and
> > probably breaks replies made to the original as opposed to the
> > forum-sent copy.
>
> I think I see what you mean, does it go like this:
>
> 1. cameron sends email to Discourse from mutt which gets `Message-ID: 74398756983476983@mail.com`
> 2. Discourse creates a post and stores the `Message-ID` with against the post with an `IncomingEmail` record
Correct.
> 1. johndoe is watching the topic, so they get sent an email from Discourse with a `Message-ID: topic/222/44@discourse.com` and no reference to the original `Message-ID: 74398756983476983@mail.com`
No. You really want to pass through `IncomingEmail.message_id` as the
`Message-ID` in the email to johndoe. It’s the same message.
> Does that sound correct, that we should just “pass on” that `Message-ID` to those watching the topic instead of generating our own since it’s already unique? What then happens in `johndoe`’s mail client if
> cameron also CC’d him on that original outbound message? This does sound like a separate issue so it would be good to open another bug topic for it.
By passing it on, the original message (cameron-\>cc:johndoe) and the
Discourse forwarded message (cameron-\>Discourse-\>johndoe) have the same
message-id and the same message contents. The receiving mail system
stores both. The mail reader sees both, and either presents both or
keeps just one (this is a policy decision of the mail reader - keeping
just one is common). Because they’re the same message, in general it
does not matter which is kept.
If we ignored discourse and considered a message which was
a copy of the message via the list and also via direct email. They’re
the same message, with the same message-id.
> > [@cameron-simpson](#):
> >
> > I and several others use mutt.
>
> I will set up a mutt client locally to see what you are also seeing, I have never tested this functionality in a text-based client (only Gmail and Thunderbird) so I am keen to see how it looks anyway.
Happy to help with settings. For threaded view you need to set the
sorting to threadeed. Mutt is very configurable.
> My line of thinking to address these issues this morning was to dispose
> with the randomly generated suffixes generated when we send
> `Message-ID` headers in emails and instead change to a scheme where we
> use the `user_id` of both the sending and receiving user. The benefit
> of this is that there is no need to store the `Message-ID` anywhere
> (apart from when an inbound email creates a post) and so `References`
> and `In-Reply-To` headers will always be consistent.
Yes, that is much better. Noting that the inbound email message-id
should override the Discourse derived message-id for the outbound email.
(Most mail systems use random strings because there’s no surrounding
context such as the discourse topic message structure - messages are
considered alone; but the only real requirement is persistent
uniqueness.)
> Let me give an example. Say we have these users:
>
> - martin - user\_id 25
> - cameron - user\_id 44
> - sam - user\_id 78
> - bob - user\_id 999
>
> And then we have this topic, topic\_id 233499, with posts starting from post\_id 100 as the OP. The format would become `topic/#{topic_id}/#{post_id}.s#{sender_user_id}r#{receiver_user_id}`.
>
> The order of operations would look like this:
>
> 1. **martin** creates the OP
>
> - **cameron** is sent an email with these headers:
> - `Message-ID: topic/233499.s25r44@meta.discourse.org`
> - `References: topic/233499@meta.discourse.org`
>
> - **sam** is sent an email with these headers:
> - `Message-ID: topic/233499.s25r78@meta.discourse.org`
> - `References: topic/233499@meta.discourse.org`
1. There should _not_ be a `References` header in the OP. It isn’t
needed for threading and effectively pretends there’s some “post 0”
which doesn’t exist. It meeans every OP (a) looks like a reply, which it
is not and (b) looks like the thing to which it is a reply is missing
from the reader’s mailbox.
2. This makes different message-ids for each outbound copy of the OP.
That’s bad. They need to be the same. Supposing **sam** CCs **cameron**
directly in a reply. The `In-Reply-To` will cite a mesage-id **cameron**
has never received.
You can just _drop_ the `sender_user_id` and `receiver_user_id` from the
message-id field and get a single unique id which _every_ receiver sees.
The uniqueness constraint is the post itself, not the individual
email-level “message” object.
Re the `References`, the OP should not have one. TB and everything else
will be fine. If they’re threading using `References` instead of
`In-Reply-To`, the `References` in the reply messages are enough.
Here’s the start of a mailing list discussion thread in Mutt:
```
16Jul2022 01:09 Rob Boehne - │├>[Python-Dev] Re: [SPAM] Re: Swit python-dev 9.2K
16Jul2022 01:33 Peter Wang - │├> python-dev 3.0K
16Jul2022 00:24 Skip Montanaro - ├>[Python-Dev] Re: Switching to Dis python-dev 4.2K
16Jul2022 04:49 Erlend Egeberg - ├>[Python-Dev] Re: Switching to Dis python-dev 10K
16Jul2022 04:20 Mariatta - ├>[Python-Dev] Re: Switching to Dis python-dev 10K
15Jul2022 21:18 Petr Viktorin - [Python-Dev] Switching to Discourse python-dev 4.2K
```
Ignore that I sort my email newest-on-top. See that there’s no arrow on
the initial post (at the bottom). That messgae has no `References` and
no `In-Reply-To`. All the others have `In-Reply-To` (and possibly
`References`, but this is an email mailing list so not necessarily; as I
mentioned before they’re complimentary.)
If I repeat my Discourse example from earlier:
```
23Jul2022 06:24 Olha via Discus - ┌>[Py] [Users] I need an advise discuss-users 5.7K
22Jul2022 17:12 Paul Jurczak vi - ├>[Py] [Users] I need an advise discuss-users 5.5K
22Jul2022 13:21 Rob via Discuss - ├>[Py] [Users] I need an advise discuss-users 6.8K
22Jul2022 12:53 vasi-h via Disc - ├>[Py] [Users] I need an advise discuss-users 5.5K
22Jul2022 11:38 Cameron Simpson - ├>[Py] [Users] I need an advise discuss-users 14K
22Jul2022 10:27 Rob via Discuss - ├>[Py] [Users] I need an advise discuss-users 6.6K
22Jul2022 06:14 vasi-h via Disc r ┴>[Py] [Users] I need an advise discuss-users 6.5K
```
See they _all_ have a leading arrow? That is because the mail client
believes they are all replies to a common (and _missing_) root message,
which is because of the “topic” message-id in the `References` header.
Whereas post 1 is actually the bottom message displayed above.
Summary:
- your plan is good, provided you drop the sender and receiver from the
message-id - they’re unnecessary and in fact the receiver will cause
trouble (the sender is just redundant).
- drop the “topic” pseudo-message-id from the `References` - it misleads
email clients (including TB, even if it isn’t visually evident)
> 1. **cameron** replies via email
>
> - **discourse** is sent an email with these headers from mutt:
> - `Message-ID: 43585349859734@test.com`
> - `References: topic/233499@meta.discourse.org topic/233499.s25r44@meta.discourse.org`
> - `In-Reply-To: topic/233499.s25r44@meta.discourse.org`
Yes, again with the caveat that there should not be a “topic” reference.
As expected, there is a reference to the OP message-id. Though it should
be the same message-id that **sam** sees for the OP.
> 1. **discourse** (as **cameron** , from the above email) creates post 101
>
> - **sam** is sent an email from discourse with these headers:
> - `Message-ID: topic/233499/101.s44r78@meta.discourse.org`
> - `References: 43585349859734@test.com topic/233499@meta.discourse.org`
> - `In-Reply-To: 43585349859734@test.com`
And here it goes wrong. The `Message-ID` should be
`43585349859734@test.com` from the `.incoming_post.message_id` field.
(Well, in my mind this is `post.message_id()`, which returns
`post.incoming_post.message_id` for an email generated post and your
Discourse generated one otherwise).
Consider: I compose and send my reply with message-id
`43585349859734@test.com`. For continuity reasons, I keep a copy of that
in my local folder, where it shows as a reply to the OP. Ideally
Discourse also sends me a copy of my own post (this is a policy setting
on many mailing lists), so I get Discourse’s version also. That should
have the _same_ message-id, because it is the same message, just via a
different route.
Discourse’s message is _not_ “in reply to” my message. It _is_ my
message, just forwarded.
This effect cascades through your following examples. The actual process
should be simpler than you’ve made it.
Think of it this way. If I reply to a post from email, it effectively is
like me emailing **sam** (and the others) via Discourse. Discourse
forwards my message to the email-receiving subscribers, and
“incidentally” keeps a copy on the forum 🙂
> As a side note, I also looked into what GitHub do with their
> notification emails, and noticed they do a similar thing where they
> have an ever-present `Reference`
> (`discourse/discourse/pull/252@github.com`) that is used in all the
> emails related to that “topic” which in this case is a GitHub pull
> request:
>
> ```plaintext
> References:
> In-Reply-To:
>
> ```
Hoo, github. What a disaster their issue emails are 🙂
However, in their scenario, the PR _is_ the OP. So a reference directly
to the pull is sane. You _could_ use the “topic” message-id for post 1,
provided you didn’t also use the “topic/1” id as well. But there seems
little point - it is extra effort to special case post 1 - I’d just use
“topic/1” myself.
To add some complication. As I understand it, an admin can move a post
or topic. Doesn’t that break the “generate the message-id” scheme,
particularly if they move just a post? I’m somewhat of the opinion that
every _post_ should have a `_message_id` field, filled in from the
incoming message (from email) or generated (posting via Discourse). Then
it is persistent and stable and robust against any shuffling of posts or
changes of algorithm.
Finally, there’s a small security consideration: you should ignore the
inbound email message-id (and potentially bounce the message) if it
claims the message-id of an existing post. Since as an author, I can put
anything I like in that header 🙂 I’d go with just dropping the
message-id - accept the post, but don’t let it lie about being some
other post - give your copy the Discourse-generated id and then proceed
as normal.
---
### Author:  [@martin](https://meta.discourse.org/u/martin)
#### Post date: [July 25, 2022, 12:18am UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/12 "2022-07-25T00:18:38Z")
Wow thanks again for this wonderfully in-depth response. It will probably take a little while for me to process this and turn into actionable items, so please bear with us (as well as this I have some high priority other internal projects I am currently working on). I think with this information we will be able to make our threading systems a lot more robust and to-spec. I may have more questions as I go through your post, thanks Cameron.
---
### Author:  [@cameron-simpson](https://meta.discourse.org/u/cameron-simpson)
#### Post date: [July 25, 2022, 2:59am UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/15 "2022-07-25T02:59:05Z")
By Martin Brennan via Discourse Meta at 25Jul2022 00:28:
> Wow thanks again for this wonderfully in-depth response. It will
> probably take a little while for me to process this and turn into
> actionable items, so please bear with us (as well as this I have some
> high priority other internal projects I am currently working on). I
> think with this information we will be able to make our threading
> systems a lot more robust and to-spec. I may have more questions as I
> go through your post, thanks Cameron.
Sure. Cheers, Cameron Simpson
---
### Author:  [@cameron-simpson](https://meta.discourse.org/u/cameron-simpson)
#### Post date: [July 25, 2022, 3:02am UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/16 "2022-07-25T03:02:12Z")
BTW, I notice this followup post from you has these headers:
```
Message-ID:
In-Reply-To:
References:
```
i.e. it has preserved my original email message-id. So the `In-Reply-To`
is correct, and the `References` at least has my email message-id in it.
That wasn’t what we were observing over at [discuss.python.org](http://discuss.python.org).
Cheers,
Cameron Simpson
---
### Author:  [@martin](https://meta.discourse.org/u/martin)
#### Post date: [July 26, 2022, 12:17am UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/19 "2022-07-26T00:17:36Z")
> [@cameron-simpson](#):
>
> But notice: the top post has a little “is a reply” arrow. Even though it
> is post 1. I expect that is because of the “topic” references entry,
> which make TB think there was a earlier message (which of course there
> was not).
Ah that is an interesting observation, I hadn’t noticed the little arrow.
> [@cameron-simpson](#):
>
> Mutt probably ignores the `References`
> because it is a mail reader, and `References` originates in USENET news.
> Maybe Thunderbird is using the references or augumenting the in-reply-to
> with references information.
>
> You only need to consult one of `In=-Reply-To` or `References` to do
> threading; the former comes from email and the latter from USENET.
> You’re supporting both (which is great!) so we need to make them
> consistent.
This is also super interesting. I believe (without examining the source) Thunderbird does do that, and likely the Gmail UI as well since it does the same thing.
> [@cameron-simpson](#):
>
> But if
> you’re passing a message on (authored in email), the existing message-id
> should be preserved.
> [@cameron-simpson](#):
>
> i.e. it has preserved my original email message-id. So the `In-Reply-To`
> is correct, and the `References` at least has my email message-id in it.
> [@cameron-simpson](#):
>
> No. You really want to pass through `IncomingEmail.message_id` as the
> `Message-ID` in the email to johndoe. It’s the same message.
We do seem to be doing this but I guess not consistently? Basically we need to make sure that:
- TODO #1 - **If a post has an associated IncomingEmail record, we always use that Message-ID when sending email.**
> [@cameron-simpson](#):
>
> There should _not_ be a `References` header in the OP. It isn’t
> needed for threading and effectively pretends there’s some “post 0”
> which doesn’t exist. It meeans every OP (a) looks like a reply, which it
> is not and (b) looks like the thing to which it is a reply is missing
> from the reader’s mailbox.
> [@cameron-simpson](#):
>
> drop the “topic” pseudo-message-id from the `References` - it misleads
> email clients (including TB, even if it isn’t visually evident)
- TODO #2 - **Do not use a `References` when sending out emails related to the OP of the topic**. @cameron-simpson one question though – if the OP was created via an inbound email, would we use that `Message-ID` in `References` for the OP or still exclude it?
> [@cameron-simpson](#):
>
> You can just _drop_ the `sender_user_id` and `receiver_user_id` from the
> message-id field and get a single unique id which _every_ receiver sees.
>
> The uniqueness constraint is the post itself, not the individual
> email-level “message” object.
> [@cameron-simpson](#):
>
> your plan is good, provided you drop the sender and receiver from the
> message-id - they’re unnecessary and in fact the receiver will cause
> trouble (the sender is just redundant).
> [@cameron-simpson](#):
>
> To add some complication. As I understand it, an admin can move a post
> or topic. Doesn’t that break the “generate the message-id” scheme,
> particularly if they move just a post? I’m somewhat of the opinion that
> every _post_ should have a `_message_id` field, filled in from the
> incoming message (from email) or generated (posting via Discourse). Then
> it is persistent and stable and robust against any shuffling of posts or
> changes of algorithm.
This is interesting, I thought every recipient of the email had to have a unique `Message-ID`? In fact I believe this is why we went down the path of adding uniqueness to each recipient’s `Message-ID`, to avoid spam behaviours, looking back on our internal topic. Perhaps @supermathie , who is on our infra team and was doing a bunch of testing with email earlier in the year, could weigh in here too?
What you are saying is that it’s more that the post should be the thing determining a single `Message-ID` for all recipients. So perhaps we just generate one for each post that generates an email? Then we could also move the `IncomingEmail.message_id` to here as well. Tentatively, the change we would need to make is:
- TODO #3 - **Add a `outbound_message_id` to the Post table. Generate it once when an email is first sent in relation to the post. Use if for subsequent `References` and `In-Reply-To` headers. Set its value when a post is created from an IncomingEmail. Format should be `topic/:topic_id/:post_id/:random_alphanumeric_string@host` e.g. `topic/233499/33545/gvy8475y7c45y87554c@meta.discourse.org`**
After this change my first example would become this:
1. **martin** creates the OP
- **cameron** is sent an email with these headers:
- `Message-ID: topic/233499/33545/gvy8475y7c45y87554c@meta.discourse.org`
- **sam** is sent an email with these headers:
- `Message-ID: topic/233499/33545/gvy8475y7c45y87554c@meta.discourse.org`
> [@cameron-simpson](#):
>
> But there seems
> little point - it is extra effort to special case post 1 - I’d just use
> “topic/1” myself.
With the consideration also that the OP does not have special handling, it will no longer be in the format `topic/:topic_id@hostname`.
> [@cameron-simpson](#):
>
> - generating correct `In-Reply-To`, which is easily computed from the
> immediate antecedant message(s) i.e. antecedant(s)-`Message-ID`
> - generating correct `References`, which is easily computed as
> antecedant-`References` + antecedant-`Message-ID`
- TODO #4 - **Ensure that correct In-Reply-To and References headers are generated based on PostReply records and the new outbound\_message\_id column on the Post table**
> [@cameron-simpson](#):
>
> Finally, there’s a small security consideration: you should ignore the
> inbound email message-id (and potentially bounce the message) if it
> claims the message-id of an existing post. Since as an author, I can put
> anything I like in that header 🙂 I’d go with just dropping the
> message-id - accept the post, but don’t let it lie about being some
> other post - give your copy the Discourse-generated id and then proceed
> as normal.
I _think_ we have some consideration for this, I will double-check.
> [@cameron-simpson](#):
>
> This effect cascades through your following examples. The actual process
> should be simpler than you’ve made it.
It definitely seems that way 😅
* * *
Can you confirm the TODOs here sound reasonable Cameron? It really doesn’t seem like much now that I look at it. I also wonder, when I get to this work would you be open to joining a testing Discourse instance with me that will have the WIP changes deployed to it so we can email back and forth and test that things are working correctly? I will of course do testing of my own before I involve you.
If not, that’s fine too – I have Thunderbird and will be setting up `mutt` and I can test it all out there 🙂
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [July 26, 2022, 2:24am UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/20 "2022-07-26T02:24:55Z")
@cameron-simpson one thing I did want to clarify here is “message\_id” scoping.
The thing that kicked off this whole dance was a strong suspicion by @supermathie that our non unique message\_ids were causing issues.
Discourse generates **unique** emails per user for every email it sends. So for example say 2 users are watching this topic:
- User 1 gets **payload 1** with a distinct unsubscribe link directed at user 1
- User 2 gets **payload 2** with a distinct unsubscribe link directed at user 2
If in **both** cases our message id was say `discourse_topic_100/23` (`topic_id/post_number`) then we will be telling MTAs out there that `discourse_topic_100/23` can be 2 distinct payloads, the hypothesis is that they treat this as a spam signal
> Hey Discourse … you just sent two emails called `discourse_topic_100/23` what is up?
Since Discourse is in control of all email transport and emails are not added to a BCC list or CC list like traditional mailing lists, we can afford to have clean per user unsubscribe links.
What are your thoughts here? What about the simple change of using `discourse_topic_100/23/7333` eg (topic\_id, post\_number, user\_id) as the unique identifier for mail, it is certainly a unique payload and we can easily refer back to it when generating mails for a user.
---
### Author:  [@cameron-simpson](https://meta.discourse.org/u/cameron-simpson)
#### Post date: [July 26, 2022, 2:46am UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/21 "2022-07-26T02:46:53Z")
By Martin Brennan via Discourse Meta at 26Jul2022 00:27:
> [quote=“Cameron Simpson, post:11, topic:233499,
> username:cameron-simpson”]
> Mutt probably ignores the `References`
> because it is a mail reader, and `References` originates in USENET news.
> Maybe Thunderbird is using the references or augumenting the in-reply-to
> with references information.
>
> You only need to consult one of `In=-Reply-To` or `References` to do
> threading; the former comes from email and the latter from USENET.
> You’re supporting both (which is great!) so we need to make them
> consistent.
> [/quote]
>
> This is also super interesting. I believe (without examining the source) Thunderbird does do that, and likely the Gmail UI as well since it does the same thing.
I think mutt will use both, but probably just `In-Reply-To` if present,
_falling back_ to `References`. I’d need to check the source.
With `References` you do at least know the full chain to the OP; with
`In-Reply-To` you more or less need the antecedant messages around to
stitch things together. For mailing lists I usually keep the whole
thread locally until it’s done anyway, and I expect that is common.
> > [@cameron-simpson](#):
> >
> > But if
> > you’re passing a message on (authored in email), the existing message-id
> > should be preserved.
>
> > [@cameron-simpson](#):
> >
> > i.e. it has preserved my original email message-id. So the `In-Reply-To`
> > is correct, and the `References` at least has my email message-id in it.
>
> > [@cameron-simpson](#):
> >
> > No. You really want to pass through `IncomingEmail.message_id` as the
> > `Message-ID` in the email to johndoe. It’s the same message.
>
> We do seem to be doing this but I guess not consistently? Basically we need to make sure that:
>
> - TODO #1 - **If a post has an associated IncomingEmail record, we always use that Message-ID when sending email.**
Yes. This is why I was thinking it might be sanest to have an explicit
field for the message-id, and to fill it in once. Then use that from
then on always, regardless of any changes to the process in which the
message-id is manufactured in the code later.
> > [@cameron-simpson](#):
> >
> > There should _not_ be a `References` header in the OP. It isn’t
> > needed for threading and effectively pretends there’s some “post 0”
> > which doesn’t exist. It meeans every OP (a) looks like a reply, which it
> > is not and (b) looks like the thing to which it is a reply is missing
> > from the reader’s mailbox.
>
> > [@cameron-simpson](#):
> >
> > drop the “topic” pseudo-message-id from the `References` - it misleads
> > email clients (including TB, even if it isn’t visually evident)
>
> - TODO #2 - **Do not use a `References` when sending out emails related to the OP of the topic**.
Yes. The OP has no antecedant, so there’s no `References` or
`In-Reply-To`.
> @cameron-simpson one question though – if the OP was created via an
> inbound email, would we use that `Message-ID` in `References` for the
> OP or still exclude it?
Still exclude. But use it as the persistent message-id for the OP.
So a message authored by email (OP or reply) gets its message-id from
the email. One authored on the web gets one when the user presses
Submit, generated by Discourse. From then on, that’s the message-id,
however created.
> [quote=“Cameron Simpson, post:11, topic:233499,
> username:cameron-simpson”]
> You can just _drop_ the `sender_user_id` and `receiver_user_id` from the
> message-id field and get a single unique id which _every_ receiver sees.
>
> The uniqueness constraint is the post itself, not the individual
> email-level “message” object.
> [/quote]
>
> > [@cameron-simpson](#):
> >
> > your plan is good, provided you drop the sender and receiver from the
> > message-id - they’re unnecessary and in fact the receiver will cause
> > trouble (the sender is just redundant).
>
> > [@cameron-simpson](#):
> >
> > To add some complication. As I understand it, an admin can move a post
> > or topic. Doesn’t that break the “generate the message-id” scheme,
> > particularly if they move just a post? I’m somewhat of the opinion that
> > every _post_ should have a `_message_id` field, filled in from the
> > incoming message (from email) or generated (posting via Discourse). Then
> > it is persistent and stable and robust against any shuffling of posts or
> > changes of algorithm.
>
> This is interesting, I thought every recipient of the email had to have a unique `Message-ID`?
No. The message-id identifies the “message”. Not the individual copy. I
might post to the forum and CC someone directly. If that someone gets a
copy direct from me and also via the forum, they should have the same
message-id.
> In fact I believe this is why we went down the path of adding
> uniqueness to each recipient’s `Message-ID`, to avoid spam behaviours,
> looking back on our internal topic. Perhaps @supermathie , who is on
> our infra team and was doing a bunch of testing with email earlier in
> the year, could weigh in here too?
Maybe. But on that face of it, threading is indeed broken. Certainly
sending the same message to many people should have the same message-id,
and generally, as a forwarder (email-\>discourse-\>email-recipients)
discourse shoud not be modifying the message-ids.
> What you are saying is that it’s more that the post should be the thing determining a single `Message-ID` for all recipients. So perhaps we just generate one for each post that generates an email?
Every post should have one stable unique message-id for use in the email
side. If the post originated from an email, that original message-id
should be used. Otherwise (via the web interface) Discourse should be
generating a message-id and storing it with the post.
> Then we could also move the `IncomingEmail.message_id` to here as well.
Sure. Having a distinct set of fields (message-id seems enough)
containing the email-side state should do it.
> Tentatively, the change we would need to make is:
>
> - TODO #3 - \*\*Add a `outbound_message_id` to the Post table. Generate
> it once when an email is first sent in relation to the post.
If you _got_ the post from an email, you should be using that, not
generating a new one.
> Use if for subsequent `References` and `In-Reply-To` headers. Set its
> value when a post is created from an IncomingEmail.
Yes. _To the message-id from the email._
> Format should be
> `topic/:topic_id/:post_id/:random_alphanumeric_string@host` e.g.
> `topic/233499/33545/gvy8475y7c45y87554c@meta.discourse.org`\*\*
For ones you generate yourselves, this looks good to me.
> After this change my first example would become this:
>
> 1. **martin** creates the OP
>
> - **cameron** is sent an email with these headers:
> - `Message-ID: topic/233499/33545/gvy8475y7c45y87554c@meta.discourse.org`
> - **sam** is sent an email with these headers:
> - `Message-ID: topic/233499/33545/gvy8475y7c45y87554c@meta.discourse.org`
Yes.
But note: the message-id only needs to be stable and unique. If the
`topic/:topid_id/:post_id@host` is stable and will never be regenerated,
that will do. But if you’re concerned about that (eg db restores or
migrations or imports bringing those same numbers) then the random
string will make it robust against collision.
Note that the message-id left part is `dot-atom-text`, defined here:
> **[RFC 5322: Internet Message Format](https://datatracker.ietf.org/doc/html/rfc5322#section-3.2.3)**
>
> RFC 5322 RFC - Draft Standard | This document specifies the Internet Message Format (IMF), a syntax for text messages that are sent between computer users, within the framework of "electronic mail" messages. This specification is a revision of...
which is alphas and digits and a limited set of punctuation characters
(which includes “/”).
Um, your headers. They should have:
```
Message-ID:
```
Note the angle brackets. The message-id is formally the bit between the
angle brackets, and the angle brackets are mandatory. Syntax here:
> **[RFC 5322: Internet Message Format](https://datatracker.ietf.org/doc/html/rfc5322#section-3.6.4)**
>
> RFC 5322 RFC - Draft Standard | This document specifies the Internet Message Format (IMF), a syntax for text messages that are sent between computer users, within the framework of "electronic mail" messages. This specification is a revision of...
> > [@cameron-simpson](#):
> >
> > But there seems
> > little point - it is extra effort to special case post 1 - I’d just use
> > “topic/1” myself.
>
> With the consideration also that the OP does not have special handling, it will no longer be in the format `topic/:topic_id@hostname`.
Sounds good.
> > [@cameron-simpson](#):
> >
> > - generating correct `In-Reply-To`, which is easily computed from the
> > immediate antecedant message(s) i.e. antecedant(s)-`Message-ID`
> > - generating correct `References`, which is easily computed as
> > antecedant-`References` + antecedant-`Message-ID`
>
> - TODO #4 - **Ensure that correct In-Reply-To and References headers are generated based on PostReply records and the new outbound\_message\_id column on the Post table**
Thanks.
> > [@cameron-simpson](#):
> >
> > Finally, there’s a small security consideration: you should ignore the
> > inbound email message-id (and potentially bounce the message) if it
> > claims the message-id of an existing post. Since as an author, I can put
> > anything I like in that header 🙂 I’d go with just dropping the
> > message-id - accept the post, but don’t let it lie about being some
> > other post - give your copy the Discourse-generated id and then proceed
> > as normal.
>
> I _think_ we have some consideration for this, I will double-check.
+1
> > [@cameron-simpson](#):
> >
> > This effect cascades through your following examples. The actual process
> > should be simpler than you’ve made it.
>
> It definitely seems that way 😅
>
> Can you confirm the TODOs here sound reasonable Cameron?
They seem correct to me.
> It really doesn’t seem like much now that I look at it. I also wonder,
> when I get to this work would you be open to joining a testing
> Discourse instance with me that will have the WIP changes deployed to
> it so we can email back and forth and test that things are working
> correctly? I will of course do testing of my own before I involve you.
Certainly. Happy to help in whatever way.
> If not, that’s fine too – I have Thunderbird and will be setting up
> `mutt` and I can test it all out there 🙂
I can help you with mutt if you want it too.
---
### Author:  [@cameron-simpson](https://meta.discourse.org/u/cameron-simpson)
#### Post date: [July 26, 2022, 3:50am UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/22 "2022-07-26T03:50:02Z")
I think you can still send distinct messages with the same message-id, even with small differences like this.
Ordinary mailing lists do this all the time to a greater or lesser degree. At the least some header mucking around always happens. But the message body is also sometimes modified. An egrarious example is `python-list`, which discards not-text attachments. The message goes through with the same message-id though. And almost all lists put a rider at the bottom with, say, a link to the list admin page or an unsubscribe link. That will not have been on the message when it arrived.
And there have been long discussions on content signing which revolve around what should be covered by a signature.
So I’d be entirely ok with you adding your recipient-specific unsub link and preserving the original message-id. The benefits _far_ _far_ outweigh the loss of threading if you gave each message _copy_ an individual message-id.
Again, consider the email user. I can reply to a discourse message and _add_ a CC to an interested outside person. Maybe they get a copy from discourse, maybe not. But if they did, it should have the source message-id on it even with your additional rider. Otherwise they’re got 2 copies of my message, but their mail system doesn’t know they’re copies of the one message. Badness ensues.
So in short: I do not think that your very minor additional unsub text warrants distinct message-ids. Keep just the one.
---
### Author:  [@supermathie](https://meta.discourse.org/u/supermathie)
#### Post date: [July 27, 2022, 1:56pm UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/23 "2022-07-27T13:56:08Z")
Sorry, I’m just catching up now, here are some thoughts, some of which have already been addressed…
> [@cameron-simpson](#):
>
> This second commit seems to address another problem we have seen: if a
> post comes from an email, the outbound message-id sent to email users is
> not the message-id of the source message from the author.
> [@cameron-simpson](#):
>
> The `Message-ID` is a single persistent identifier for a message
The difficulty here is that what is sent _out_ from Discourse _is_ a different message than the inbound. It has different metadata (for this purpose, To/From/Reply-to/Unsubscribe/etc.) and a different body (it’s customised per user (I think? Does this not happen in mailling list mode?)).
What _exactly_ is the message? Treating 5322 as gospel:
> A message consists of header fields, optionally followed by a message body.
> The “Message-ID:” field provides a unique message identifier that refers to a **particular version** of a particular message.
[emphasis mine]
It’s that “particular version” that makes me think it would be inappropriate to re-send an incoming message with a different Message-ID. Though, if you change your point of view from Discourse as “Forum Software” to Discourse being “Mailing List Software” then it _kind of_ makes sense to do so, so I get where you’re coming from. 5322 also says:
> There are many instances when messages are “changed”, but those changes do
> not constitute a new instantiation of that message, and therefore the message
> would not get a new message identifier. For example, when messages are
> introduced into the transport system, they are often prepended with
> additional header fields such as trace fields (described in section 3.6.7)
> and resent fields (described in section 3.6.6). The addition of such header
> fields does not change the identity of the message and therefore the original
> “Message-ID:” field is retained. In all cases, it is the meaning that the
> sender of the message wishes to convey (i.e., whether this is the same
> message or a different message) that determines whether or not the
> “Message-ID:” field changes, not any particular syntactic difference that
> appears (or does not appear) in the message.
I suppose it comes down to, does the sender of the message change when Discourse sends it out?
Maybe we should use Resent-Message-ID and friends?
> [@cameron-simpson](#):
>
> It looks like `References` is now also present in RFC5322
It’s always been there, all the back to 822. But as you say later, yes it’s been updated.
5322 also speaks directly to the way Discourse and Github use it:
> The “In-Reply-To:” field may be used to identify the message (or messages) to
> which the new message is a reply, while the “References:” field may be used to
> identify a “thread” of conversation.
Possibly slightly improperly, likely due to the lack of a suitable “Thread Identifier” header. But this interpretation may not be what the RFC authors intended… it doesn’t address messages with a “References” but without “In-Reply-To”.
> [@martin](#):
>
> This is interesting, I thought every recipient of the email had to have a unique `Message-ID`? In fact I believe this is why we went down the path of adding uniqueness to each recipient’s `Message-ID`, to avoid spam behaviours, looking back on our internal topic. Perhaps @supermathie , who is on our infra team and was doing a bunch of testing with email earlier in the year, could weigh in here too?
The tricky bit of this is that we aren’t sending out _one_ email, we’re sending out N - one per recipient - so that their individual metadata (Unsubscribe, etc.) can be correct.
And yes, I did see strong indications during testing that spam determination would be tied to a Message-ID. If it was later seen again (same user _or_ different user) it would be much more likely to be marked spam.
> [@cameron-simpson](#):
>
> So I’d be entirely ok with you adding your recipient-specific unsub link and preserving the original message-id. The benefits _far_ _far_ outweigh the loss of threading if you gave each message _copy_ an individual message-id.
The benefits here, to be fair, are _entirely_ around threading the emails correctly in certain mail clients at the expense of deliverability.
The current `topic/#{topic_id}/#{post_id}.s#{sender_user_id}r#{receiver_user_id}` at least makes it consistent _for a user in their mailbox_. The assumption
_My_ biggest concern is the deliverability - it’s hard enough to get email delivered when there is zero visibility from the major providers.
But I do see a strong argument for making Discourse behave more like mailing list software _in mailing list mode_. @martin I believe we don’t customise the message body in mailing list mode? Do you think it makes sense to take a more strict approach around preserving and reusing Message-IDs _in mailing list mode_?
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [July 27, 2022, 9:22pm UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/24 "2022-07-27T21:22:30Z")
I don’t want to be in a situation where perfect is the enemy of good enough here.
>
We use “random suffix” now in messages and this is unquestionably causing pain.
We have 3 options on the table:
1. Random message ids that can not be referred back to
2. Message ids **stable** per topic/post/user
3. Message ids **stable** per topic/post pair
We are currently in planet (1) which is wreaking havoc.
I worry that we can reach decision paralysis between (2) and (3).
Perhaps we simply start with (2) acknowledging that adding extra ccs to an email from Discourse may cause unexpected behavior, and at least stop the majority of the pain here?
---
### Author:  [@supermathie](https://meta.discourse.org/u/supermathie)
#### Post date: [July 27, 2022, 10:27pm UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/25 "2022-07-27T22:27:28Z")
> [@sam](#):
>
> We are currently in planet (1) which is wreaking havoc.
ah! I thought we were already doing: `topic/#{topic_id}/#{post_id}.s#{sender_user_id}r#{receiver_user_id}`
I would be inclined to, in the interest of balancing concerns of email uniqueness & deliverability vs. those of mailing-list-mode, do (2) for mailinglist-mode _disabled_ and (3) for mailinglist-mode _enabled_.
Similarly, with the `References` header, I would be inclined to have it _absent_ for post #1 in a topic and have it referencing the topic (so `topic/#{topic_id}`) and the post to which it’s replying, if any.
[Next page](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499.md?page=2)