` which is
> the OP as well.
This is all correct.
Cheers,
Cameron Simpson [cs@cskk.id.au](mailto:cs@cskk.id.au)
---
### Author:  [@martin](https://meta.discourse.org/u/martin)
#### Post date: [August 24, 2022, 1:25am UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/72 "2022-08-24T01:25:35Z")
> [@cameron-simpson](#):
>
> This is all correct.
Thank you. I also forgot to mention that we indeed do have to go to the database to recreate the `References` chain based on the `PostReply` records, you will see that in the latest commit.
---
### Author:  [@martin](https://meta.discourse.org/u/martin)
#### Post date: [August 24, 2022, 11:28pm UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/73 "2022-08-24T23:28:48Z")
@cameron-simpson I would like to get the internal review process going with this code next week (as well as just general polish of what I have done), since I am going on holiday on the 3rd. That way if the review is completed I can merge it and deploy it out once I get back. If you have any further feedback please let me know before then, otherwise I will assume all is good (which it did seem to be during our testing yesterday 🙂 ) .
---
### Author:  [@cameron-simpson](https://meta.discourse.org/u/cameron-simpson)
#### Post date: [August 24, 2022, 11:41pm UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/74 "2022-08-24T23:41:13Z")
I think things are mostly good. My notes from hand sweeping the email thread at my end, which I’ve mostly hand swept for headers:
```plaintext
Topic for testing threading 2022-08-23
post msg-id detail
59/1 98 OP new topics for testing email threading
59/11 109 reply-to-topic in-reply-to 98 ref 98
59/2 100 reply-to-topic in-reply-to 98 ref 98
59/3 via-email uuid@discourse yes welcome in-reply-to 100 ref 98,100
not noted as a reply in the web ui
??? me-via-email ...kr@cskk glad to be here in-reply-to uuid@discourse no refs
not noted as a reply in the web ui
59/10 108 reply to earlier post in-reply-to ...kr@cskk ref 98,100,0aa@discourse,kr@cskk
59/5 103 thanks cameron in-reply-to kr@cskk refs 98,100,0aa@discourse,kr@cskk
???104 me-via-email ...zp@cskk in-reply-to 103 no refs
not noted as a reply in the web ui
59/7 105 no problem in-reply-to zp@cskk refs 98,100,00a@discourse,kr@cskk,103,zp@cskk
posted on web, reply to 104? aka zp@@cskk
not noted as a reply in the web UI (so, new-top-topic?)
quotes kr@cskk "glad to be here"
NEEDS REVIEW
59/9 107 expected or a bug in-reply-to 106 ref 98,100,0aa@discourse,kr@cskk,103,zp@cskk,105,106
quotes 59/8
Notes:
- email replies are not shown as replies in the web ui
- web multireplies only get one msg-id in the in-reply-to
- users do not get email copies of their own posts (email or web), would be nice to have an option in prefs for this
- web msg-ids seem to be forum post.id, would be nicer if topic.id/in-topic.id for easier tracing in headers
```
In summary: I did not find any incorrect headers, but note some shortcomings above.
I haven’t had a chance to review your updated code, but functionally things seem correct.
Thank you,
Cameron
---
### Author:  [@martin](https://meta.discourse.org/u/martin)
#### Post date: [August 25, 2022, 12:41am UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/75 "2022-08-25T00:41:03Z")
Thanks Cameron!
> [@cameron-simpson](#):
>
> `email replies are not shown as replies in the web ui`
Can you elaborate on this a little? Do you mean you can’t see this?
Or that you cannot see this part? For the latter, we only show the Reply with the arrow if the post you are replying to is further up the topic, not just one previous:
> [@cameron-simpson](#):
>
> `web multireplies only get one msg-id in the in-reply-to`
Oh I did not think this is required, so if you reply to N posts via the web, all of those post’s Message-IDs should show in the `In-Reply-To` header, and then `References` follows the current logic of a thread from OP down to a single parent (in our case I choose the most recently created post as the single parent)?
> [@cameron-simpson](#):
>
> `users do not get email copies of their own posts (email or web), would be nice to have an option in prefs for this`
Yes this is by design to not send you things that you have “seen” already, we could spin that out in a different TODO to see if there is a want for this from more users.
> [@cameron-simpson](#):
>
> `web msg-ids seem to be forum post.id, would be nicer if topic.id/in-topic.id for easier tracing in headers`
The problem with topic ID is that it is too brittle and not specific/unique enough, also it will seem a little confusing when a post gets moved between topics. Maybe we can include a custom header in the email, e.g. `X-Discourse-Topic-ID` or something (if that is allowed) to allow for easier visual tracking?
---
### Author:  [@cameron-simpson](https://meta.discourse.org/u/cameron-simpson)
#### Post date: [August 25, 2022, 1:11am UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/76 "2022-08-25T01:11:31Z")
> [@martin](#):
>
> > [@cameron-simpson](#):
> >
> > `email replies are not shown as replies in the web ui`
>
> Can you elaborate on this a little? Do you mean you can’t see this? [Email envelope icon.]
No, I see that icon.
> [@martin](#):
>
> Or that you cannot see this part? [reploy-icon+parent-author] For the latter, we only show the Reply with the arrow if the post you are replying to is further up the topic, not just one previous:
Ah. Whereas I, as an email nonforum person expected that indicator on every reply because I’m not thinking about this as an instant messaging layout (maybe). So my expectations are at variance with what you’ve chosen to do.
> [@martin](#):
>
> > [@cameron-simpson](#):
> >
> > `web multireplies only get one msg-id in the in-reply-to`
>
> Oh I did not think this is required,
It isn’t required. Think of it as “quality of service”. You explicitly go:
```
@message.header['In-Reply-To'] = referenced_post_message_ids[0] || topic_canonical_reference_id
```
and you could just drop the `[0]` there. Clients could then use just one message-id or do something very funky at their whim and it would all be valid.
> [@martin](#):
>
> so if you reply to N posts via the web, all of those post’s Message-IDs should show in the `In-Reply-To` header, and then `References` follows the current logic of a thread from OP down to a single parent (in our case I choose the most recently created post as the single parent)?
“Should” is a strong word. You _could_ include all the message-ids if they’re easily to hand. You’re not obliged, and the code is valid as is.
> [@martin](#):
>
> > [@cameron-simpson](#):
> >
> > `users do not get email copies of their own posts (email or web), would be nice to have an option in prefs for this`
>
> Yes this is by design to not send you things that you have “seen” already, we could spin that out in a different TODO to see if there is a want for this from more users.
Aye. I know I like it myself so that I know my post made it to the list/forum - email being very queue based and some (cough, large Australian telco, cough) ISP mail handlers being very… unreliable, slow, etc etc. Occasionally I have seen other people want this (in lists, but that’s the mode we’re effectively talking about here). The default for such an option should probably be false.
As a nerd, I like being at least able to get an unfiltered feed so that I can make my own policy decisions.
> [@martin](#):
>
> > [@cameron-simpson](#):
> >
> > `web msg-ids seem to be forum post.id, would be nicer if topic.id/in-topic.id for easier tracing in headers`
>
> The problem with topic ID is that it is too brittle and not specific/unique enough, also it will seem a little confusing when a post gets moved between topics.
Given that the `Message-ID` is effectively opaque/set-once I don’t view that as a problem unless there’s scope for _reissuing_ the same message-id - if all your counters are strictly monotonic I would not expect that to happen. I just found it very tedious to match up `post.id` eg `98` to topic/post eg `59/1`. It would have been handy to have something like `category.id/topic.id/post-in-topic.id` there instead of the `98`.
> [@martin](#):
>
> Maybe we can include a custom header in the email, e.g. `X-Discourse-Topic-ID` or something (if that is allowed) to allow for easier visual tracking?
That would also be sufficient. This is just convenience on the debugging headers side.
Cheers,
Cameron
---
### Author:  [@martin](https://meta.discourse.org/u/martin)
#### Post date: [August 25, 2022, 1:17am UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/77 "2022-08-25T01:17:07Z")
> [@cameron-simpson](#):
>
> It isn’t required. Think of it as “quality of service”. You explicitly go:
>
> ```plaintext
> @message.header['In-Reply-To'] = referenced_post_message_ids[0] || topic_canonical_reference_id
>
> ```
That is still the old code you are looking at there, you only need to look at `add_experimental_identification_field_headers` . But your point still stands.
> [@cameron-simpson](#):
>
> As a nerd, I like being at least able to get an unfiltered feed so that I can make my own policy decisions.
There is someone on our infrastructure team who would be agreeing firmly with this statement, they share a similar workflow and outlook to you 😆
> [@cameron-simpson](#):
>
> Given that the `Message-ID` is effectively opaque/set-once I don’t view that as a problem unless there’s scope for _reissuing_ the same message-id - if all your counters are strictly monotonic I would not expect that to happen. I just found it very tedious to match up `post.id` eg `98` to topic/post eg `59/1`. It would have been handy to have something like `category.id/topic.id/post-in-topic.id` there instead of the `98`.
That’s fair enough, perhaps I will re-introduce topic ID as well, since `outbound_message_id` is indeed set once and not changed (based on whether the post is generated in the Discourse web UI or via an incoming email).
> [@cameron-simpson](#):
>
> That would also be sufficient. This is just convenience on the debugging headers side.
Probably won’t need this now since I am adding the topic ID into the `Message-ID` once more.
---
### Author:  [@cameron-simpson](https://meta.discourse.org/u/cameron-simpson)
#### Post date: [August 25, 2022, 1:20am UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/78 "2022-08-25T01:20:03Z")
> [@martin](#):
>
> That is still the old code you are looking at there, you only need to look at `add_experimental_identification_field_headers` . But your point still stands.
Ah, should have seen this I guess:
```
most_recent_post = referenced_posts.first
most_recent_post_message_id = Email::MessageIdService.generate_or_use_existing(most_recent_post)
@message.header["In-Reply-To"] = most_recent_post_message_id
```
Anyway, the existing stuff is already valid. Entirely your call.
---
### Author:  [@martin](https://meta.discourse.org/u/martin)
#### Post date: [August 26, 2022, 1:53am UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/79 "2022-08-26T01:53:29Z")
> [@cameron-simpson](#):
>
> > [@martin](#):
> >
> > > [@cameron-simpson](#):
> > >
> > > `web msg-ids seem to be forum post.id, would be nicer if topic.id/in-topic.id for easier tracing in headers`
> >
> > The problem with topic ID is that it is too brittle and not specific/unique enough, also it will seem a little confusing when a post gets moved between topics.
>
> Given that the `Message-ID` is effectively opaque/set-once I don’t view that as a problem unless there’s scope for _reissuing_ the same message-id - if all your counters are strictly monotonic I would not expect that to happen. I just found it very tedious to match up `post.id` eg `98` to topic/post eg `59/1`. It would have been handy to have something like `category.id/topic.id/post-in-topic.id` there instead of the `98`.
I’m actually not sure about this, I’ve got a gut feeling we don’t really want the topic ID in those Message-IDs, just having the post ID makes it super simple. Will try this instead:
> [@martin](#):
>
> Maybe we can include a custom header in the email, e.g. `X-Discourse-Topic-ID` or something (if that is allowed) to allow for easier visual tracking?
---
### Author:  [@martin](https://meta.discourse.org/u/martin)
#### Post date: [August 26, 2022, 1:56am UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/80 "2022-08-26T01:56:54Z")
Actually we have them already and remove them here:
[https://github.com/discourse/discourse/blob/d135d0613f44812566b739e77537225f9e7d5c90/lib/email/sender.rb#L179-L181](https://github.com/discourse/discourse/blob/d135d0613f44812566b739e77537225f9e7d5c90/lib/email/sender.rb#L179-L181)
We can just keep them in, doesn’t hurt and helps with visual debugging!
---
### Author:  [@cameron-simpson](https://meta.discourse.org/u/cameron-simpson)
#### Post date: [August 26, 2022, 1:58am UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/81 "2022-08-26T01:58:03Z")
That’s fine by me. I just found it particularly hard to tie email messages headers to posts because the `post.id` isn’t evident in the UI or the URL for the particular post on the web. Having that present in an additional context header would be just as good.
Thanks,
Cameron
---
### Author:  [@cameron-simpson](https://meta.discourse.org/u/cameron-simpson)
#### Post date: [September 20, 2022, 12:03am UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/82 "2022-09-20T00:03:04Z")
Just doing a quick ping - the PR seems to have been quiet for quite a while. - Cameron
---
### Author:  [@martin](https://meta.discourse.org/u/martin)
#### Post date: [September 20, 2022, 12:05am UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/83 "2022-09-20T00:05:04Z")
Hi Cameron, I’ve been at our company meetup then on vacation for the last 2 weeks, just getting back into the swing of things today. If it’s not merged this week then it will definitely be merged next week. Apologies for the delays!
---
### Author:  [@cameron-simpson](https://meta.discourse.org/u/cameron-simpson)
#### Post date: [September 20, 2022, 12:30am UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/84 "2022-09-20T00:30:25Z")
By Martin Brennan via Discourse Meta at 20Sep2022 00:17:
> Hi Cameron, I’ve been at our company meetup then on vacation for the
> last 2 weeks, just getting back into the swing of things today. If it’s
> not merged this week then it will definitely be merged next week.
> Apologies for the delays!
No need for apologies. I knew you’d been away, wasn’t sure about your
timing or return. And I guess it might still be Monday where you are ☹
Thank you,
Cameron Simpson [cs@cskk.id.au](mailto:cs@cskk.id.au)
---
### Author:  [@martin](https://meta.discourse.org/u/martin)
#### Post date: [September 25, 2022, 11:19pm UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/85 "2022-09-25T23:19:49Z")
I just merged the PR, I will try to get the Python forum deployed later today so you can start using it in earnest.
---
### Author:  [@cameron-simpson](https://meta.discourse.org/u/cameron-simpson)
#### Post date: [September 26, 2022, 12:27am UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/86 "2022-09-26T00:27:05Z")
By Martin Brennan via Discourse Meta at 25Sep2022 23:29:
> I just merged the PR, I will try to get the Python forum deployed later
> today so you can start using it in earnest.
That would be amazing. Thanks, Cameron
---
### Author:  [@martin](https://meta.discourse.org/u/martin)
#### Post date: [September 26, 2022, 1:51am UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/87 "2022-09-26T01:51:38Z")
That’s been done now, please let me know here if you experience any issues 👍
---
### Author:  [@cameron-simpson](https://meta.discourse.org/u/cameron-simpson)
#### Post date: [September 26, 2022, 8:12am UTC](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499/88 "2022-09-26T08:12:47Z")
Thank you. I’ll let you know how it looks. Cameron
[Previous page](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499.md?page=3)
[Next page](https://meta.discourse.org/t/discourse-email-messages-are-incorrectly-threaded/233499.md?page=5)