Discourse email messages are incorrectly threaded

[cameron-simpson]

@martin

@cameron-simpson

email replies are not shown as replies in the web ui

Can you elaborate on this a little? Do you mean you can’t see this? [Email envelope icon.]

No, I see that icon.

@martin

Or that you cannot see this part? [reploy-icon+parent-author] For the latter, we only show the Reply with the arrow if the post you are replying to is further up the topic, not just one previous:

Ah. Whereas I, as an email nonforum person expected that indicator on every reply because I’m not thinking about this as an instant messaging layout (maybe). So my expectations are at variance with what you’ve chosen to do.

@martin

@cameron-simpson

web multireplies only get one msg-id in the in-reply-to

Oh I did not think this is required,

It isn’t required. Think of it as “quality of service”. You explicitly go:

@message.header['In-Reply-To'] = referenced_post_message_ids[0] || topic_canonical_reference_id

and you could just drop the [0] there. Clients could then use just one message-id or do something very funky at their whim and it would all be valid.

@martin

so if you reply to N posts via the web, all of those post’s Message-IDs should show in the In-Reply-To header, and then References follows the current logic of a thread from OP down to a single parent (in our case I choose the most recently created post as the single parent)?

“Should” is a strong word. You could include all the message-ids if they’re easily to hand. You’re not obliged, and the code is valid as is.

@martin

@cameron-simpson

users do not get email copies of their own posts (email or web), would be nice to have an option in prefs for this

Yes this is by design to not send you things that you have “seen” already, we could spin that out in a different TODO to see if there is a want for this from more users.

Aye. I know I like it myself so that I know my post made it to the list/forum - email being very queue based and some (cough, large Australian telco, cough) ISP mail handlers being very… unreliable, slow, etc etc. Occasionally I have seen other people want this (in lists, but that’s the mode we’re effectively talking about here). The default for such an option should probably be false.

As a nerd, I like being at least able to get an unfiltered feed so that I can make my own policy decisions.

@martin

@cameron-simpson

web msg-ids seem to be forum post.id, would be nicer if topic.id/in-topic.id for easier tracing in headers

The problem with topic ID is that it is too brittle and not specific/unique enough, also it will seem a little confusing when a post gets moved between topics.

Given that the Message-ID is effectively opaque/set-once I don’t view that as a problem unless there’s scope for reissuing the same message-id - if all your counters are strictly monotonic I would not expect that to happen. I just found it very tedious to match up post.id eg 98 to topic/post eg 59/1. It would have been handy to have something like category.id/topic.id/post-in-topic.id there instead of the 98.

@martin

Maybe we can include a custom header in the email, e.g. X-Discourse-Topic-ID or something (if that is allowed) to allow for easier visual tracking?

That would also be sufficient. This is just convenience on the debugging headers side.

Cheers,
Cameron

[martin (Martin Brennan)]

@cameron-simpson

It isn’t required. Think of it as “quality of service”. You explicitly go:

@message.header['In-Reply-To'] = referenced_post_message_ids[0] || topic_canonical_reference_id

That is still the old code you are looking at there, you only need to look at add_experimental_identification_field_headers . But your point still stands.

@cameron-simpson

As a nerd, I like being at least able to get an unfiltered feed so that I can make my own policy decisions.

There is someone on our infrastructure team who would be agreeing firmly with this statement, they share a similar workflow and outlook to you

@cameron-simpson

Given that the Message-ID is effectively opaque/set-once I don’t view that as a problem unless there’s scope for reissuing the same message-id - if all your counters are strictly monotonic I would not expect that to happen. I just found it very tedious to match up post.id eg 98 to topic/post eg 59/1. It would have been handy to have something like category.id/topic.id/post-in-topic.id there instead of the 98.

That’s fair enough, perhaps I will re-introduce topic ID as well, since outbound_message_id is indeed set once and not changed (based on whether the post is generated in the Discourse web UI or via an incoming email).

@cameron-simpson

That would also be sufficient. This is just convenience on the debugging headers side.

Probably won’t need this now since I am adding the topic ID into the Message-ID once more.

[cameron-simpson]

@martin

That is still the old code you are looking at there, you only need to look at add_experimental_identification_field_headers . But your point still stands.

Ah, should have seen this I guess:

 most_recent_post = referenced_posts.first
      most_recent_post_message_id = Email::MessageIdService.generate_or_use_existing(most_recent_post)
      @message.header["In-Reply-To"] = most_recent_post_message_id

Anyway, the existing stuff is already valid. Entirely your call.

[martin (Martin Brennan)]

@cameron-simpson

@martin

@cameron-simpson

web msg-ids seem to be forum post.id, would be nicer if topic.id/in-topic.id for easier tracing in headers

The problem with topic ID is that it is too brittle and not specific/unique enough, also it will seem a little confusing when a post gets moved between topics.

Given that the Message-ID is effectively opaque/set-once I don’t view that as a problem unless there’s scope for reissuing the same message-id - if all your counters are strictly monotonic I would not expect that to happen. I just found it very tedious to match up post.id eg 98 to topic/post eg 59/1. It would have been handy to have something like category.id/topic.id/post-in-topic.id there instead of the 98.

I’m actually not sure about this, I’ve got a gut feeling we don’t really want the topic ID in those Message-IDs, just having the post ID makes it super simple. Will try this instead:

@martin

Maybe we can include a custom header in the email, e.g. X-Discourse-Topic-ID or something (if that is allowed) to allow for easier visual tracking?

[martin (Martin Brennan)]

Actually we have them already and remove them here:

discourse/discourse/blob/d135d0613f44812566b739e77537225f9e7d5c90/lib/email/sender.rb#L179-L181

We can just keep them in, doesn’t hurt and helps with visual debugging!

[cameron-simpson]

That’s fine by me. I just found it particularly hard to tie email messages headers to posts because the post.id isn’t evident in the UI or the URL for the particular post on the web. Having that present in an additional context header would be just as good.

Thanks,
Cameron

[cameron-simpson]

Just doing a quick ping - the PR seems to have been quiet for quite a while. - Cameron

[martin (Martin Brennan)]

Hi Cameron, I’ve been at our company meetup then on vacation for the last 2 weeks, just getting back into the swing of things today. If it’s not merged this week then it will definitely be merged next week. Apologies for the delays!

[cameron-simpson]

By Martin Brennan via Discourse Meta at 20Sep2022 00:17:

Hi Cameron, I’ve been at our company meetup then on vacation for the
last 2 weeks, just getting back into the swing of things today. If it’s
not merged this week then it will definitely be merged next week.
Apologies for the delays!

No need for apologies. I knew you’d been away, wasn’t sure about your
timing or return. And I guess it might still be Monday where you are

Thank you,
Cameron Simpson cs@cskk.id.au

[martin (Martin Brennan)]

I just merged the PR, I will try to get the Python forum deployed later today so you can start using it in earnest.

[cameron-simpson]

By Martin Brennan via Discourse Meta at 25Sep2022 23:29:

I just merged the PR, I will try to get the Python forum deployed later
today so you can start using it in earnest.

That would be amazing. Thanks, Cameron

[martin (Martin Brennan)]

That’s been done now, please let me know here if you experience any issues

[cameron-simpson]

Thank you. I’ll let you know how it looks. Cameron

[cameron-simpson]

It’s immediately looking better - Cameron

[martin (Martin Brennan)]

Fantastic, great to hear. Thanks for all your excellent help and advice Cameron

[martin (Martin Brennan)]

I’d just thought I would check in, @cameron-simpson has everything still been flowing along nicely with email threading?

[cameron-simpson]

Yes, it’s been really good!

I thought I’ve seen the odd glitch (two distinct threads on the one
topic, but I cannot see any examples at present), but I haven’t
characterised any of them yet, or eliminated my local mail folder as
being involved So I’ve no problems to report yet.

Thank you,
Cameron Simpson cs@cskk.id.au

[martin (Martin Brennan)]

Thanks Cameron, great to hear! I’ll schedule this topic to close after another month, if it is closed and another issue comes up feel free to open another one