# Discourse email messages are incorrectly threaded
**URL:** https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499
**Category:** Bug
**Created:** [July 21, 2022, 6:50am UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499 "2022-07-21T06:50:36Z")
**Posts on this page:** 20
**Page:** 2
### Author:  [@cameron-simpson](https://meta.discourse.org/u/cameron-simpson)
#### Post date: [July 27, 2022, 11:20pm UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/26 "2022-07-27T23:20:28Z")
Apologies in advance for some of the tone below. I sound exasperated,
because I am a little exasperated.
By Michael Brown via Discourse Meta at 27Jul2022 14:06:
> Sorry, I’m just catching up now, here are some thoughts, some of which
> have already been addressed…
>
> > [@cameron-simpson](#):
> >
> > This second commit seems to address another problem we have seen: if a
> > post comes from an email, the outbound message-id sent to email users is
> > not the message-id of the source message from the author.
>
> > [@cameron-simpson](#):
> >
> > The `Message-ID` is a single persistent identifier for a message
>
> The difficulty here is that what is sent _out_ from Discourse _is_ a different message than the inbound. It has different metadata (for this purpose, To/From/Reply-to/Unsubscribe/etc.) and a different body (it’s customised per user (I think? Does this not happen in mailling list mode?)).
>
> What _exactly_ is the message? Treating 5322 as gospel:
>
> > A message consists of header fields, optionally followed by a message body.
>
> > The “Message-ID:” field provides a unique message identifier that refers to a **particular version** of a particular message.
> > [emphasis mine]
>
> It’s that “particular version” that makes me think it would be inappropriate to re-send an incoming message with a different Message-ID. Though, if you change your point of view from Discourse as “Forum Software” to Discourse being “Mailing List Software” then it _kind of_ makes sense to do so, so I get where you’re coming from.
Well, unfortunately this depends on an overly literal reading, maybe
reading conext which isn’t there.
_Every_ email messages gets its headers modified as mail system pass it
along. If nothing else, `Received:` headers get added at every step, and
several systems add various headers indicating spam filtering results
and signatures. None of those trigger a message-id modification, and
indeed doing so would make the message-id totally dysfunctional.
Regarding content, as already mentioned, almost every mailing list adds
content to the body text, usually a footer with a link to the list admin
page or an unsubscribe link. There _also_ do not trigger a message-id
change.
In fact, almost nothing which forwards a message changes the message-id.
Because that would break threading and duplicate detection for end user
clients.
I see you go on to quote what I was just about to cite 🙂
> 5322 also says:
>
> > There are many instances when messages are “changed”, but those changes do
> > not constitute a new instantiation of that message, and therefore the message
> > would not get a new message identifier. For example, when messages are
> > introduced into the transport system, they are often prepended with
> > additional header fields such as trace fields (described in section 3.6.7)
> > and resent fields (described in section 3.6.6). The addition of such header
> > fields does not change the identity of the message and therefore the original
> > “Message-ID:” field is retained. In all cases, it is the meaning that the
> > sender of the message wishes to convey (i.e., whether this is the same
> > message or a different message) that determines whether or not the
> > “Message-ID:” field changes, not any particular syntactic difference that
> > appears (or does not appear) in the message.
>
> I suppose it comes down to, does the sender of the message change when Discourse sends it out?
I think you’ve misread things here. Let me emphasise:
```
In all cases, it is the meaning that the sender of the message
wishes to convey (i.e., whether this is the same message or a
different message) that determines whether or not the "Message-ID:"
field changes
```
The _sender_ is the author, not an MTA such as Discourse.
If I post to Discourse via email, I want my message to reach the readers
as it is, semanticly speaking. Any riders like unsub links do not change
the semantics of what I have said in my message.
It’s still _the same message_.
> Maybe we should use Resent-Message-ID and friends?
Absolutely not. They are for a _user_ resubmitting a message. For
example, if I forwarded a message on to someone else. They’re not for
mail relays (such as lists and Discourse).
> > [@cameron-simpson](#):
> >
> > It looks like `References` is now also present in RFC5322
>
> It’s always been there, all the back to 822. But as you say later, yes it’s been updated.
Ouch. I thought it was USENET only at that point. I stand corrected.
> 5322 also speaks directly to the way Discourse and Github use it:
>
> > The “In-Reply-To:” field may be used to identify the message (or messages) to
> > which the new message is a reply, while the “References:” field may be used to
> > identify a “thread” of conversation.
>
> Possibly slightly improperly, likely due to the lack of a suitable “Thread Identifier” header. But this interpretation may not be what the RFC authors intended… it doesn’t address messages with a “References” but without “In-Reply-To”.
It says to me that the two fields cover the same information:
- `References` shows a linear (usually) thread back to the OP
- `In-Reply-To` shows the parent, and implies the same thread in
aggregate with the previous messages back to the OP
> > [@martin](#):
> >
> > This is interesting, I thought every recipient of the email had to have a unique `Message-ID`? In fact I believe this is why we went down the path of adding uniqueness to each recipient’s `Message-ID`, to avoid spam behaviours, looking back on our internal topic. Perhaps @supermathie , who is on our infra team and was doing a bunch of testing with email earlier in the year, could weigh in here too?
>
> The tricky bit of this is that we aren’t sending out _one_ email, we’re sending out N - one per recipient - so that their individual metadata (Unsubscribe, etc.) can be correct.
This isn’t tricky. The meaning of the messages is the same, the
customisations are minor and semanticly irrelevant. They _do not_
warrant new or distinct message-ids.
> And yes, I did see strong indications during testing that spam determination would be tied to a Message-ID. If it was later seen again (same user _or_ different user) it would be much more likely to be marked spam.
Can you show some of these instances. Because message-ids allow
deduplication are the end user’s end. And bear in mind that many
“antispam” measures are misguides rubbish. The number of things I’ve had
rejected as potential spam for utterly spurious reasons… breaking
email to work around broken spam misfiltering is a poor choice.
To this day I never CC people with GMail addresses because GMail’s spam
filtering knows me and drops things on the floor. If I send only to the
list, they get it. If I CC ther GMail address it (a) marks it as spam
and (b) then _also_ marks the mailing list message as spam as well (same
message-id!) The end user doesn’t see my message. This logic is utterly
spurious and unrepairable.
> [quote=“Cameron Simpson, post:22, topic:233499,
> username:cameron-simpson”]
> So I’d be entirely ok with you adding your recipient-specific unsub link and preserving the original message-id. The benefits _far_ _far_ outweigh the loss of threading if you gave each message _copy_ an individual message-id.
> [/quote]
>
> The benefits here, to be fair, are _entirely_ around threading the emails correctly in certain mail clients at the expense of deliverability.
Sigh. To _all_ email clients. And a _major_ reason people over in
Pythonland are saying they will just not go to Discourse is that the
email side threading is broken. _Many_ people do not use forums, because
each forum requires them to _visit_ it. Email comes to them, they get to
use _their_ preferred reader and _their_ preferred editor, and threading
lets people see the discussion flow clearly. When it works.
> The current `topic/#{topic_id}/#{post_id}.s#{sender_user_id}r#{receiver_user_id}` at least makes it consistent _for a user in their mailbox_. The assumption
>
> _My_ biggest concern is the deliverability - it’s hard enough to get email delivered when there is zero visibility from the major providers.
I would like to see evidence. Mailing lists do this correctly all over
the planet. Discourse definitely and objectively breaks this. I’m trying
to get it fixed.
Let me reiterate the two basic problems here:
- The OP `In-Reply-To` and `References` cite a fictitious “pre-OP”
“topic” message-id, so no email user has a threwad with a starting
message (the OP) - _everything_ including the OP looks like a followup
- The emails received via Discourse and the emails received directly eg
via CC have different message-ids even though they are the same
message semanticly speaking; this breaks threading and deduplication
> But I do see a strong argument for making Discourse behave more like mailing list software _in mailing list mode_. @martin I believe we don’t customise the message body in mailing list mode? Do you think it makes sense to take a more strict approach around preserving and reusing Message-IDs _in mailing list mode_?
Their are people over in Pythonland who found “mailing list mode” too
much of a firehose. They want to get email for targeted topics but not
everything. The message-id handling should e the same for _all_ of the
email side.
I’m a “mailing list mode” person on [discuss.python.org](http://discuss.python.org). But I turned it
on here ([discourse.org](http://discourse.org)) and \_immediately turned it off again. I need
targeted mode over here.
---
### Author:  [@cameron-simpson](https://meta.discourse.org/u/cameron-simpson)
#### Post date: [July 27, 2022, 11:26pm UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/27 "2022-07-27T23:26:52Z")
By Michael Brown via Discourse Meta at 27Jul2022 22:37:
> > [@sam](#):
> >
> > We are currently in planet (1) which is wreaking havoc.
>
> ah! I thought we were already doing: `topic/#{topic_id}/#{post_id}.s#{sender_user_id}r#{receiver_user_id}`
`{receiver_user_id}` puts you into distinct message-ids per end user for
the same source post. That is bad as soon as end users communicate
outside discourse or get copies not via discourse.
> I would be inclined to, in the interest of balancing concerns of email uniqueness & deliverability vs. those of mailing-list-mode, do (2) for mailinglist-mode _disabled_ and (3) for mailinglist-mode _enabled_.
And mentioned in my recent post, _mailing list mode_ only covers one
flavour of Discourse email reception. All the same concerns apply
whether the email receiver is in mailing list mode or just
email-for-some-topics/tags mode.
> Similarly, with the `References` header, I would be inclined to have it _absent_ for post #1 in a topic
Likewise the `In-Reply-To`. neither should be present, becaue to be
present they have to reference a fictitious per-to-OP message.
> and have it referencing the topic (so `topic/#{topic_id}`) and the post
> to which it’s replying, if any.
You can’t refer to the “topic” message-id _unless there was a post with
that message-id_ which went out as email. If you want to go that way,
special case the message-id of the OP to be the “topic” message=id
instead of `...../1`.
---
### Author:  [@cameron-simpson](https://meta.discourse.org/u/cameron-simpson)
#### Post date: [July 28, 2022, 2:49am UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/29 "2022-07-28T02:49:47Z")
This should be “prior-to-OP”. Sorry, Cameron Simpson
---
### Author:  [@supermathie](https://meta.discourse.org/u/supermathie)
#### Post date: [July 28, 2022, 2:57am UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/30 "2022-07-28T02:57:59Z")
> [@cameron-simpson](#):
>
> Can you show some of these instances.
As you say, _this_ is exactly the problem frustrating us:
> [@cameron-simpson](#):
>
> To this day I never CC people with GMail addresses because GMail’s spam
> filtering knows me and drops things on the floor. If I send only to the
> list, they get it. If I CC ther GMail address it (a) marks it as spam
> and (b) then _also_ marks the mailing list message as spam as well (same
> message-id!) The end user doesn’t see my message. This logic is utterly
> spurious and unrepairable.
> [@cameron-simpson](#):
>
> The OP `In-Reply-To` and `References` cite a fictitious “pre-OP”
> “topic” message-id, so no email user has a threwad with a starting
> message (the OP) - _everything_ including the OP looks like a followup
I agree this should be changed. The OP message-id should be (in lieu of one coming over mail) (simplified) `topic/1` and not reference another message.
The message ID wouldn’t change, even if it was only ever a Discourse post and never an email.
_Further_ messages can reference that one.
> [@cameron-simpson](#):
>
> You can’t refer to the “topic” message-id _unless there was a post with
> that message-id_ which went out as email.
Why must an email exist? Semantically, having only the post fits the criteria. The message it’s responding to exists, just not in that person’s email folder. We’ve just come to the conclusion that the _message_ is what’s important, be that Post body or Email body. It follows that the `topic/#{topic_id}/1@site` is a unique message ID referencing that post, whether it’s in an email message or not.
It’s no different than receiving a reply to an email that references an email not in your inbox. It’s _still_ a reply, so `References` is legitimate and correct.
Fundamentally I agree with you. The purist in me wants this correct. But the practicality of needing to get email into people’s inboxes was what led to this. For worse, a _ton_ of people use gmail and never train its filters, use it properly, and “Unsubscribe” by reporting as spam\[1\].
> [@cameron-simpson](#):
>
> breaking email to work around broken spam misfiltering is a poor choice
I agree, I think we were a bit too literal in reading
> A message identifier pertains to exactly one instantiation of a particular message
After mulling this over for a while I do think we ought to go back to what we had before (remove randomisation) and lock down a single message-id per post and it should be:
- message\_id\_from\_incoming\_email || `topic/#{topic_id}/#{post_num}@site` (post\_num of OP is 1)
And whenever we send out an email, I think it is correct to add `References` to parents all the way back to the OP and set `In-Reply-To` to the appropriate stable post message-ID (or the OP if replying to topic) since the _Message_ is the post. But those fields for the OP should be blank, yes.
* * *
1. not that gmail reports this to us, despite us implementing feedback-loop.
---
### Author:  [@martin](https://meta.discourse.org/u/martin)
#### Post date: [July 29, 2022, 12:26am UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/31 "2022-07-29T00:26:08Z")
> [@cameron-simpson](#):
>
> Let me reiterate the two basic problems here:
>
> - The OP `In-Reply-To` and `References` cite a fictitious “pre-OP”
> “topic” message-id, so no email user has a threwad with a starting
> message (the OP) - _everything_ including the OP looks like a followup
> - The emails received via Discourse and the emails received directly eg
> via CC have different message-ids even though they are the same
> message semanticly speaking; this breaks threading and deduplication
> [@cameron-simpson](#):
>
> But note: the message-id only needs to be stable and unique. If the
> `topic/:topid_id/:post_id@host` is stable and will never be regenerated,
> that will do.
> [@supermathie](#):
>
> After mulling this over for a while I do think we ought to go back to what we had before (remove randomisation) and lock down a single message-id per post and it should be:
>
> - message\_id\_from\_incoming\_email || `topic/#{topic_id}/#{post_num}@site` (post\_num of OP is 1)
>
> And whenever we send out an email, I think it is correct to add `References` to parents all the way back to the OP and set `In-Reply-To` to the appropriate stable post message-ID (or the OP if replying to topic) since the _Message_ is the post. But those fields for the OP should be blank, yes.
Thank you for your responses @supermathie and @cameron-simpson , I do believe we have reached consensus. Pulling out the TODOs into a single post, and I hope to be able to begin work on these quite soon:
1. Change the generated `Message-ID` format to always be ``, this is unique enough, it’s basically reverting to what we used to do. Referring to the OP will just use the first post ID now instead of just the bare topic ID.
2. If a post has an associated `IncomingEmail` record, we always use that `Message-ID` when sending email, otherwise we generate one using the format above.
3. Do not use a `References` when sending out emails for the OP of the topic, there is nothing to `Reference` yet because it is the first email in the thread.
4. Ensure that correct `In-Reply-To` and `References` headers are generated based on `PostReply` records.
This has the potential to leave things in a bit of a murky state thread-wise for already sent emails, but I will try my best to allow for the format we are moving away from for a changover period as well. Thanks for bearing with us!
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [July 29, 2022, 12:32am UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/32 "2022-07-29T00:32:44Z")
> [@martin](#):
>
> :hostname
Just to clarify … this would not be the `hostname` of the server this originated from, but the url of the site? If it is hostname then we lose all stability when 3 different hosts serve the same site.
---
### Author:  [@martin](https://meta.discourse.org/u/martin)
#### Post date: [July 29, 2022, 12:34am UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/33 "2022-07-29T00:34:59Z")
Sorry yeah I mean the site domain e.g. `meta.discourse.org` which comes from `Email::Sender.host_for(Discourse.base_url)`, what we already use.
---
### Author:  [@supermathie](https://meta.discourse.org/u/supermathie)
#### Post date: [July 29, 2022, 1:12am UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/34 "2022-07-29T01:12:14Z")
> [@martin](#):
>
> Add a new `outbound_message_id` to the `Post` table, so we can be sure the thread survives even if a post moves topics or anything like that, store the `Message-ID` here for both of the above cases.
Good call, didn’t think about moves. Is `:post_id` the post’s ID (`post.id`) or number (within the topic)
If it’s the post ID, we can simplify and just use `` since that will never change, then we don’t need to store the Message-ID unless it’s overridden from default.
If not… we may as well use the post ID here? no reason this part has to be long, it just has to be unique.
---
### Author:  [@martin](https://meta.discourse.org/u/martin)
#### Post date: [July 29, 2022, 1:20am UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/35 "2022-07-29T01:20:05Z")
> [@supermathie](#):
>
> Is `:post_id` the post’s ID (`post.id`) or number (within the topic)
It’s the actual ID not the post number.
> [@supermathie](#):
>
> If it’s the post ID, we can simplify and just use `` since that will never change, then we don’t need to store the Message-ID unless it’s overridden from default.
That is a good point, `` will probably work just fine and avoids the extra column requirement. Maybe to make it more Discourse-specific we could add `discourse` to the front e.g. `` (keeping in mind many sites will have no mention of Discourse in the hostname). It’s splitting hairs at this point though.
I will try to think of ways this can be messed up. I guess if you move a post to another topic and then someone replies to the post via email their reply will end up in the new topic instead of the original topic. Maybe that is fine? Other risk is that the post is moved into a private category but I think we already have that same risk and we handle it.
Just thinking out loud, should be fine, will cover these things when I test out the changes anyway 👍
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [July 29, 2022, 1:28am UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/36 "2022-07-29T01:28:37Z")
The argument for including topic\_id is that you can deliberately break threading if people split a post from a topic into another topic.
I am mixed on it. Can go either way. But that would be the idea.
---
### Author:  [@martin](https://meta.discourse.org/u/martin)
#### Post date: [July 29, 2022, 1:37am UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/37 "2022-07-29T01:37:39Z")
The argument for using only the post ID is that it is more static which is what we want, since if you move a post to another topic the post ID will be the same in the `Message-ID` but the topic will not be the same.
I _think_ if we end up moving the post and sending out emails from the new topic the new thread will be created correctly anyway in the mail client, since the `References` and `In-Reply-To` header chains will be different. Anyway, I will make sure to test this scenario out and see if it does what we expect as well. Nothing will be merged into core until the various scenarios work as expected.
---
### Author:  [@martin](https://meta.discourse.org/u/martin)
#### Post date: [July 29, 2022, 1:46am UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/38 "2022-07-29T01:46:59Z")
Based on these further discussions @cameron-simpson I updated the TODOs to this, posting them here so you get the update since the Discourse edits will not arrive via email:
> 1. Change the generated `Message-ID` format to always be ``, this is unique enough, it’s basically reverting to what we used to do. Referring to the OP will just use the first post ID now instead of just the bare topic ID.
> 2. If a post has an associated `IncomingEmail` record, we always use that `Message-ID` when sending email, otherwise we generate one using the format above.
> 3. Add a new `outbound_message_id` column to the `Post` records which will be filled by either a) the Message-ID of the incoming email if it is creating the post or b) the outgoing `Message-ID` that we generate in the case of posts created by the Discourse web UI
> 4. Do not use a `References` or `In-Reply-To` headers when sending out emails for the OP of the topic, there is nothing to `Reference` or reply to yet because it is the first email in the thread.
> 5. Ensure that correct `In-Reply-To` and `References` headers are generated based on `PostReply` records.
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [July 29, 2022, 2:21am UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/39 "2022-07-29T02:21:37Z")
> [@martin](#):
>
> Ensure that correct `In-Reply-To` and `References` headers are generated based on `PostReply` records.
Does this cover quotes as well (Eg: a post quoted 10 different other posts, so it references them?)
---
### Author:  [@cameron-simpson](https://meta.discourse.org/u/cameron-simpson)
#### Post date: [July 29, 2022, 2:49am UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/40 "2022-07-29T02:49:03Z")
By Sam Saffron via Discourse Meta at 29Jul2022 02:31:
> > [@martin](#):
> >
> > Ensure that correct `In-Reply-To` and `References` headers are generated based on `PostReply` records.
>
> Does this cover quotes as well (Eg: a post quoted 10 different other
> posts, so it references them?)
`In-Reply-To` can only cite one antecedant, so pick one. `References`
can reference more than one but the RFC explicitly recommends against
this because not all client apps might expect other than a linear chain
from this post back to the OP.
I’d be ok with either for the `References` but would lean to the
conservative one. The easy computation is:
- `In-Reply-To`: use the message-id of the first quoted message (or
whatever single quote you pick based on some policy)
- `References`: the `References` of the same single chosen antecedant
post above plus the message-id of that same post
These would be stable, predictable and correct.
Cheers,
Cameron Simpson [cs@cskk.id.au](mailto:cs@cskk.id.au)
---
### Author:  [@supermathie](https://meta.discourse.org/u/supermathie)
#### Post date: [July 29, 2022, 2:50am UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/41 "2022-07-29T02:50:08Z")
`References` is [discouraged](https://www.rfc-editor.org/rfc/rfc5322#section-3.6.4) from being used this way:
> Note: Some implementations parse the “References:” field to display the “thread of the discussion”. These implementations assume that each new message is a reply to a single parent and hence that they can walk backwards through the “References:” field to find the parent of each message listed there. Therefore, trying to form a “References:” field for a reply that has multiple parents is discouraged; how to do so is not defined in this document.
---
### Author:  [@cameron-simpson](https://meta.discourse.org/u/cameron-simpson)
#### Post date: [July 29, 2022, 2:58am UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/42 "2022-07-29T02:58:12Z")
By Martin Brennan via Discourse Meta at 29Jul2022 01:57:
> Based on these further discussions @cameron-simpson I updated the TODOs
> to this, posting them here so you get the update since the Discourse
> edits will not arrive via email:
>
> > 1. Change the generated `Message-ID` format to always be ``, this is unique enough, it’s basically reverting to what we used to do. Referring to the OP will just use the first post ID now instead of just the bare topic ID.
> > 2. If a post has an associated `IncomingEmail` record, we always use that `Message-ID` when sending email, otherwise we generate one using the format above.
> > 3. Do not use a `References` when sending out emails for the OP of the topic, there is nothing to `Reference` yet because it is the first email in the thread.
I would also omit the `In-Reply-To` in the OP emails.
> > 1. Ensure that correct `In-Reply-To` and `References` headers are
> > generated based on `PostReply` records.
Yes.
Personally, I would go the extra step of having a column for the
email-side message-id. That way, once you’ve allocated a message-id for
the post (from the source email if from email, or generated if from the
web interface), it remains stable regardless of whatever else might
happen in the code now or later. i.e. even if there’s no `IncomingEmail`
the message-id generation happens just the once, rather than being
recomputed (which could thus change).
i.e. make it stable once made by storing it.
You have an `IncomingEmail` relation by the look of it. Maybe you have
(or could use) an `OutgoingEmail` relation for the additional state for
the outbound email messages, made the first time a post is forwarded by
email.
I know the flow is basicly that that happens when a post is made, but I
can imagine some later user feature being things like:
- please forward me emails for this whole topic, now that I’m interested
- if an edit happens to a post, consider sending the updated message out
with the same message-id
The reason the second example comes to mind is that we’ve got more
things to report 🙂 One is that Discourse seems to do some effort to
drop the quoted part of top posted replies to keep the post concise, or
something like that. I wrote a long post over in Python land a few weeks
back which got severe mistruncated. i went in and edited it on the forum
with the original text from my personal copy. But a receipient said he
had the full thing, and i was wondwering if Discourse sent edit updates
out as replacement messages with the same id. Which would be quite neat
depending on the end user client handling of that.
---
### Author:  [@cameron-simpson](https://meta.discourse.org/u/cameron-simpson)
#### Post date: [July 29, 2022, 3:00am UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/43 "2022-07-29T03:00:04Z")
By Martin Brennan via Discourse Meta at 29Jul2022 00:36:
> 1. Add a new `outbound_message_id` to the `Post` table, so we can be
> sure the thread survives even if a post moves topics or anything like
> that, store the `Message-ID` here for both of the above cases.
Yes, i think this is important, however implemented (relation or column
of whatever). I think I said that in your revised TODOs.
> 1. Do not use a `References` when sending out emails for the OP of the topic, there is nothing to `Reference` yet because it is the first email in the thread.
> 2. Ensure that correct `In-Reply-To` and `References` headers are generated based on `PostReply` records and the new `outbound_message_id` column on the `Post` table.
>
> This has the potential to leave things in a bit of a murky state thread-wise for already sent emails, but I will try my best to allow for the format we are moving away from for a changover period as well. Thanks for bearing with us!
Nothing we can do for the existing emails. Just get things well threaded
going forward!
Thanks!
Cameron Simpson [cs@cskk.id.au](mailto:cs@cskk.id.au)
---
### Author:  [@martin](https://meta.discourse.org/u/martin)
#### Post date: [July 29, 2022, 3:08am UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/44 "2022-07-29T03:08:29Z")
> [@cameron-simpson](#):
>
> You have an `IncomingEmail` relation by the look of it. Maybe you have
> (or could use) an `OutgoingEmail` relation for the additional state for
> the outbound email messages, made the first time a post is forwarded by
> email.
We do have `EmailLog` but those records get cleaned up every 90 days, and I don’t think it would be a good fit to this. Will just do this:
> [@cameron-simpson](#):
>
> > 1. Add a new `outbound_message_id` to the `Post` table, so we can be
> > sure the thread survives even if a post moves topics or anything like
> > that, store the `Message-ID` here for both of the above cases.
>
> Yes, i think this is important, however implemented (relation or column
> of whatever). I think I said that in your revised TODOs.
---
### Author:  [@supermathie](https://meta.discourse.org/u/supermathie)
#### Post date: [July 29, 2022, 3:15am UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/45 "2022-07-29T03:15:39Z")
> [@martin](#):
>
> That is a good point, `` will probably work just fine and avoids the extra column requirement.
It was about avoiding storing it at all… but now I think of it the post ID will never change but hostname might. So we should store it immediately after saving in all cases.
Couldn’t hurt to have `messageid` be a property of every Post, forever immutable…
> [@cameron-simpson](#):
>
> - if an edit happens to a post, consider sending the updated message out
> with the same message-id
Wouldn’t this be a different version of the message? From the spec:
> The “Message-ID:” field provides a unique message identifier that refers to a particular version of a particular message. … A message identifier pertains to exactly one version of a particular message; subsequent revisions to the message each receive new message identifiers.
So probably our generated message-id ought to be: `` (possibly leaving off `/rev/:revision_num` for the first revision). This would allow email recipients to _get_ the edit updates in the first place, considering that
> [@cameron-simpson](#):
>
> message-ids allow deduplication at the end user’s end
---
### Author:  [@martin](https://meta.discourse.org/u/martin)
#### Post date: [July 29, 2022, 3:19am UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/46 "2022-07-29T03:19:13Z")
> [@supermathie](#):
>
> Couldn’t hurt to have `messageid` be a property of every Post, forever immutable…
Yep will do that. As for these other discussions about edits and revisions, I think that is a whole other huge kettle of fish that we shouldn’t get into right now…let’s fix our threading sins first 🙂
[Previous page](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499.md?page=1)
[Next page](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499.md?page=3)