E-Mail Threading in ML-Mode does not work in Thunderbird

feature
1

I have one of these “dinosaur”-Settings, with the wish to replicate a mailing list.
However, the threading in thunderbird does not work the way it should (with the latest Docker installation).
Assume “A” and “B” are users and they send mails that should result in following thread:

A: message1
  B: reply1 to message1
    A: reply1 to reply1 to message1
      B: reply1 to reply1 to reply1 to message1
  A: reply2 to message1

How I see them in my thunderbird is the following

A: message1
  B: reply1 to message1
  A: reply1 to reply1 to message1
  B: reply1 to reply1 to reply1 to message1
  A: reply2 to message1

Its flat, like in only one level deep.
I wonder a bit that nobody has these issues - probably its related to the fact that we do not use reply-key mail-addresses and tough maillinglist-setting everywhere possible.

I compared the Headers (in-reply-to and references are the relevant ones) to threads that fold well and found that I can “fix” the issue manually in my inbox by manipulating the headers such that the references is sorted such that the reply-to - message-id is last.

That would mean the bug lies in ordering of the references-header. This is found here:
https://github.com/discourse/discourse/blob/e8559f222c856455f38bf3edb62985b464581acb/lib/email/sender.rb#L112

Unfortunately I will not soon find the time to setup a dev-environment and test these carefully. I suspect that the uniq kicks out the actual topic_message_id, so maybe something along these lines would fix it:

[referenced_post_message_ids - topic_message_id].flatten.compact.uniq + [topic_message_id]

Although it looks quiet ugly, but you get the point.

5 Likes
2

Interesting :thinking: and good :male_detective: work. I’ll fix it.

I wish email threading was documented so we could stop guessing and write headers the way they’re actually used…

5 Likes
4

Wouldn’t that still put the topic message ID last? You’d want it to be first, instead right?

5

Nope. The references in mails that are displayed well in my thunderbird traverse “from the root to the last message”. So, original “post” first, then the first answer (in that branch), the answer to that answer, … and so on.

From:

https://www.jwz.org/doc/threading.html

[…] it asserts that this header contain a list of Message-IDs listing the parent, grandparent, great-grandparent, and so on, of this message, oldest first. That is, the direct parent of this message will be the last element of the References header.

6

That’s not how topics work in Discourse, they are mostly flat with some light, one level only attachment of replies … so I am afraid this is an irreconcilable difference.

This is a correct representation of the discussion from Discourse’s standpoint:

A: message1
  B: reply1 to message1
  A: reply1 to reply1 to message1
  B: reply1 to reply1 to reply1 to message1
  A: reply2 to message1
7

@codinghorror: ? Sometimes here on the discourse frontend I see something like “3 Replies” after a reply and can unfold that (which is superhandy btw). Are these only on the first level?
Anyways, the change is cheap, the benefit huge - I find threaded mail views enormously helpful, especially when the original “discussion” branches into several topics.

8

Correct, only one level. There is no way to expand grandchildren or great-grandchildren etc.

1 Like
9

If you understand “code speak”, the field reply_to_post_number is an integer, not an array. So even if some type of recursive iterator was coded up, the deeper level threading would be limited to one per post. i.e. when a post quotes (refers to) more than one other post, only one of them is stored in the table.

10

Sometimes I speak code better than english :wink:

Thats totally fine from the point of view of algorithmic thread calculation, from my interpretation of

https://www.jwz.org/doc/threading.html

Apparently, the real length of the References-Array does not matter. From my experiments however the order matters. The last referenced ("`reply-to’") post should be the last.

It seems to me that there is some doubt of whether it makes sense to take advantage of email-readers ability to reflect “hierarchies” and the (simplified) tree-nature of replies visually (which can often be switched off by the email-client anyway). I dont have that doubts :slight_smile:

@zogstrip wanted to look into this and I would be glad if we help the email-clients to visualize what happened - especially when reply-by-mail is offered imho it makes sense to give the users the possibility to see to which post/mail was replied to. Again, the change should be extremely cheap code-wise and results in an increased usabilty. And users who dislike to see deeper threads in their mail-client (if such users exist in a larger number that would be a surprise for me) can usually switch threaded-views off.

3 Likes
11

So, I repeat. Wouldn’t something more like this be more correct?

[referenced_post_message_ids, topic_message_id].flatten.compact.uniq.reverse

12

@riking I see I have to dig in the code and check where the stuff from - I cannot tell you exactly what the “topic message” is. But if you consider the thread - tree

  MSG1
    MSG1R1
    MSG1R2
      MSG1R2R1
        MSG1R2R1R1
    MSG1R3

then for MSG1R2R1R1 the reply-to should be MSG1R2R1 and the references would list the path from the root node to itself (in that order, old-to-new), so
MSG1, MSG1R2, MSG1R2R1. In my impression, the “later” references are the more important ones (so, the parent is more important than the grand-grand-grand-grand-parent).

13

topic_message_id is MSG1 in your diagram.

Note that Discourse’s view of the thread is more like this:

  MSG1
    MSG2 (← ø)
    MSG3 (← MSG1)
      MSG4 (← MSG3)
        MSG6 (← MSG4, MSG5)
    MSG5 (← ø)
14

Also in Discourse’s view a single post can be in reply to three other upstream posts. Just quote three different posts above you, in a single reply…

15

Actually

[topic_message_id, referenced_post_message_ids].flatten.compact.uniq.reverse

solves the issue. I prepare a PR on github .
@zogstrip, @riking, @codinghorror : How are the chances of that PR (will link it from here) getting merged upstream?

2 Likes
16

Very high since it’s what I was thinking of doing :wink:

1 Like
17

Cool. Here you go :slight_smile:

https://github.com/discourse/discourse/pull/5567

I am not sure whether further optimizations are needed or possible, but the referenced change fixes the issue for me afaics.

6 Likes