Moving chat messages to a new channel jumbled them all up

JammyDodger · May 17, 2022, 10:33am

I moved 28 messages from one channel to a new one and they all went out of order:

JammyDodger

#How can chat seed topic discussions?

Hmmm. Is it just me, or do these seem out of order now?

JammyDodger

#The Chat Feedback topic

I've had another look at the out-of-order posts in https://meta.discourse.org/chat/channel/147/chat?messageId=2644, and it seems they all have the same timestamp.

martin · May 23, 2022, 1:57am

Thanks for reporting this, I did try to take this into account with the feature but I think I got a little too lucky with the testing I did So we do indeed set all the timestamps to the same thing here:

github.com

discourse/discourse-chat/blob/82c0ee7f1069bd04509c8eb15870d9165eab101a/lib/message_mover.rb#L87-L94

      
        
            def create_destination_messages_in_channel(destination_channel)
              moved_message_ids = DB.query_single(<<~SQL, message_ids: @ordered_source_message_ids, destination_channel_id: destination_channel.id)
                INSERT INTO chat_messages(chat_channel_id, user_id, message, cooked, cooked_version, created_at, updated_at)
                SELECT :destination_channel_id, user_id, message, cooked, cooked_version, CLOCK_TIMESTAMP(), CLOCK_TIMESTAMP()
                FROM chat_messages
                WHERE id IN (:message_ids)
                RETURNING id
              SQL

The problem is that we do not want to interleave the moved messages between the existing channel messages, and it gets harder and harder the more messages your move.

A question before I dive too deep on this – can you remember and identify which messages are out of place? Is there just a couple, or are they completely out of order? I think what has possibly caused the discrepancy is that when we are fetching messages for a channel, we do it by ID order (which we order DESC and then reverse in most cases):

github.com

discourse/discourse-chat/blob/82c0ee7f1069bd04509c8eb15870d9165eab101a/app/controllers/chat_controller.rb#L177-L187

      
        
            messages = preloaded_chat_message_query.where(chat_channel: @chat_channel)
            
            
order_direction = :desc
            if params[:before_message_id]
              messages = messages.where("id < ?", params[:before_message_id])
            end
            
            
if params[:after_message_id]
              messages = messages.where("id > ?", params[:after_message_id])
              order_direction = :asc
            end

github.com

discourse/discourse-chat/blob/82c0ee7f1069bd04509c8eb15870d9165eab101a/app/controllers/chat_controller.rb#L201-L203

      
        
            chat_view = ChatView.new(
              chat_channel: @chat_channel,
              chat_messages: params[:after_message_id] ? messages : messages.to_a.reverse,

Whereas in the message mover I am ordering by created_at to maintain order, which could cause some small discrepancies:

github.com

discourse/discourse-chat/blob/82c0ee7f1069bd04509c8eb15870d9165eab101a/lib/message_mover.rb#L59-L61

      
        
            def find_messages(message_ids, channel)
              ChatMessage.where(id: message_ids, chat_channel_id: channel.id).order(:created_at)
            end

I have some ideas on how we might be able to address this (maybe changing the message mover to order by ID or changing the controller to order by created_at would be enough, preferring the latter because I think it would make more sense), but I would like to hear just how messed up the order is if it’s possible to tell.

JammyDodger · May 23, 2022, 6:02am

I un-deleted them from the original channel after I noticed they were jumbled in the new one. I should be able to quote them in order here:

Original order

Originally sent in The Chat Feedback topic

JammyDodger

I'm looking for ways we can capture the essence of the new chat flow as a demonstration of how chat can be the seeds of bigger discussion

Has anyone got any ideas on how we can do that from where we are now with the chat-testing here on Meta?

The feedback points are great, and I think they will be calved off into their own proper topics shortly too, but I was hoping for something that would work as a great exemplar for anyone just joining the @chat-testers. Something that people can just look at and go 'Aaaah. Yes, I wasn't sure at first, but I see how can chat can be the pre-cursor to in depth discussion'

I may be asking too much

RGJ

TBH I don't see how chat can be the pre-cursor to in depth discussion

But that might be because I'm old

JammyDodger

So we're looking for an example that will help change @RGJ's mind too

Moin

I like the example here. But that example does not fit here. Maybe the topic could be about a feature you miss just now. Something you wouldn't start a topic because for example you don’t have time to check whether there is a similar feature request or you think no one else will be interested into that.

JammyDodger

@Moin, your search skills are always a blessing

I was looking for one about seeds and trees, but couldn't find it on here

But, yes. Some kind of relaxed/friendly/informal chat in which an idea forms through the course of the easy back-and-forth, and then inspires a proper discussion topic

oshyan

This is 100% my interest and use-case for chat. But can you clarify exactly what you mean by an example of this. Are you interested in, say, samples of a chat (regardless of whether it happened in Discourse chat) that could/should/would have (obviously) led to deeper discussion and/or did lead to deeper discussion but in an arguably less ideal medium for it (chat vs. forum)? If that's the case I may need a little to find some, but I absolutely have good examples from my productivity community. If you're looking for examples in Discourse chat, specifically, that's going to be harder to find. But I absolutely think this is a big value of chat in Discourse, and depending on the community, it will play a bigger or smaller role.

Jonathan_Poyer

I think showcase a discussion around a new feature is actually a nice way to demo it at least during the first moment of that idea. For some people, this spark of discussion is during the time of development or about to begin. There are always more things to discuss about and referring to a topic (or multiple) makes sense.

oshyan

As a perhaps more conceptual example of a situation where chat can (and should) quickly turn into a topic, even mid-way through a discussion, this is something that often happens in software dev management communities I'm part of, as well as my productivity community:

New person joins chat and asks a seemingly simple or innocuous question
Responses from highly informed and/or impassioned regulars quickly grow to 10s of lines of text, paragraph breaks start happening, and that chat channel becomes saturated only with discussion on this one question (topic)
Everything becomes difficult to respond to as each "message" includes lots of points and ideas and no select-to-quote/reply feature makes parsing and responding to each thing difficult
These conversations are also often valuable debates that would quickly disappear in the subsequent flow of a chat, so even retroactively moving them to a Topic is potentially very valuable

JammyDodger

I think I was initially looking for something we could provide as an example topic/chat here on Meta to demonstrate to people new to Discourse Chat of how it could gel nicely with pre-existing ideas of the 'long-form paragraph' view of Discourse

So even something that we created to neatly demonstrate the principle

Though it sounds like you have lots of examples that would make a great discussion topic

I think anything that can help people easily visualise where in their forum structure chat can slot into would be useful. I am open to all ideas

agungor

I feel that this conversation is becoming such an example per se.

Jonathan_Poyer

Feels like something that would become a thread on other platforms would either need to turn into another chat or a split off topic. But at the same time, topic feel also more like long term discussion vs one-off as here?

jimkleiber

Maybe one way is to quote the initial chat message that started the idea into a topic: How can chat seed topic discussions?

This can give visibility to people who aren't in the chat, especially if the topic starts with a question

but ironically I'm answering here lol

Hmm, I just tried to quote my replies here into that same topic but I think I only have the option to quote into a new topic, not an existing one

JammyDodger

I was just mulling that over. I was working out if I could make a topic with each person's chat as a reply somehow, as there were no suitable chat channels to slide them to. But now you've made it I can see that it could have its own chat channel, and we could move this conversation there

jimkleiber

Ah yes, almost like creating the topic that will create the new chat channel

and then the topic can be filled with just quotes from the chat, pulling the highlights

Jumbled

Originally sent in How can chat seed topic discussions?

oshyan

This is 100% my interest and use-case for chat. But can you clarify exactly what you mean by an example of this. Are you interested in, say, samples of a chat (regardless of whether it happened in Discourse chat) that could/should/would have (obviously) led to deeper discussion and/or did lead to deeper discussion but in an arguably less ideal medium for it (chat vs. forum)? If that's the case I may need a little to find some, but I absolutely have good examples from my productivity community. If you're looking for examples in Discourse chat, specifically, that's going to be harder to find. But I absolutely think this is a big value of chat in Discourse, and depending on the community, it will play a bigger or smaller role.

Jonathan_Poyer

I think showcase a discussion around a new feature is actually a nice way to demo it at least during the first moment of that idea. For some people, this spark of discussion is during the time of development or about to begin. There are always more things to discuss about and referring to a topic (or multiple) makes sense.

oshyan

As a perhaps more conceptual example of a situation where chat can (and should) quickly turn into a topic, even mid-way through a discussion, this is something that often happens in software dev management communities I'm part of, as well as my productivity community:

New person joins chat and asks a seemingly simple or innocuous question
Responses from highly informed and/or impassioned regulars quickly grow to 10s of lines of text, paragraph breaks start happening, and that chat channel becomes saturated only with discussion on this one question (topic)
Everything becomes difficult to respond to as each "message" includes lots of points and ideas and no select-to-quote/reply feature makes parsing and responding to each thing difficult
These conversations are also often valuable debates that would quickly disappear in the subsequent flow of a chat, so even retroactively moving them to a Topic is potentially very valuable

JammyDodger

I think I was initially looking for something we could provide as an example topic/chat here on Meta to demonstrate to people new to Discourse Chat of how it could gel nicely with pre-existing ideas of the 'long-form paragraph' view of Discourse

So even something that we created to neatly demonstrate the principle

I think anything that can help people easily visualise where in their forum structure chat can slot into would be useful. I am open to all ideas

I'm looking for ways we can capture the essence of the new chat flow as a demonstration of how chat can be the seeds of bigger discussion

Though it sounds like you have lots of examples that would make a great discussion topic

Has anyone got any ideas on how we can do that from where we are now with the chat-testing here on Meta?

RGJ

TBH I don't see how chat can be the pre-cursor to in depth discussion

agungor

I feel that this conversation is becoming such an example per se.

Jonathan_Poyer

Feels like something that would become a thread on other platforms would either need to turn into another chat or a split off topic. But at the same time, topic feel also more like long term discussion vs one-off as here?

JammyDodger

The feedback points are great, and I think they will be calved off into their own proper topics shortly too, but I was hoping for something that would work as a great exemplar for anyone just joining the @chat-testers. Something that people can just look at and go 'Aaaah. Yes, I wasn't sure at first, but I see how can chat can be the pre-cursor to in depth discussion'

jimkleiber

Maybe one way is to quote the initial chat message that started the idea into a topic: How can chat seed topic discussions?

This can give visibility to people who aren't in the chat, especially if the topic starts with a question

JammyDodger

I think, at least. I'll double-check my thinking before I do anything

I may be asking too much

jimkleiber

but ironically I'm answering here lol

RGJ

But that might be because I'm old

JammyDodger

I was just mulling that over. I was working out if I could make a topic with each person's chat as a reply somehow, as there were no suitable chat channels to slide them to. But now you've made it I can see that it could have its own chat channel, and we could move this conversation there

jimkleiber

Ah yes, almost like creating the topic that will create the new chat channel

JammyDodger

So we're looking for an example that will help change @RGJ's mind too

jimkleiber

and then the topic can be filled with just quotes from the chat, pulling the highlights

Moin

I like the example here. But that example does not fit here. Maybe the topic could be about a feature you miss just now. Something you wouldn't start a topic because for example you don’t have time to check whether there is a similar feature request or you think no one else will be interested into that.

JammyDodger

@Moin, your search skills are always a blessing

I was looking for one about seeds and trees, but couldn't find it on here

But, yes. Some kind of relaxed/friendly/informal chat in which an idea forms through the course of the easy back-and-forth, and then inspires a proper discussion topic

mcwumbly · May 23, 2022, 4:50pm

I wonder when it’s preferable to move messages rather than quote them. Perhaps it depends whether there is an existing topic or not? Not sure. In what scenarios would one of the following be better to nudge people towards?

Quote chat messages in existing topic
Move chat messages to an existing topic
Quote chat messages to a new topic
Move chat messages to a new topic

Since strings of chat messages are, erm, more “chatty” than topics, I have some sense that we may want to encourage quoting more than moving, in general.

Are there cases folks have observed or have in mind where you’re like, “nope, quoting would not be good here. definitely need to move them instead.”?

Moin · May 23, 2022, 4:53pm

If you just quote the discussion can be continued in two places.

mcwumbly · May 23, 2022, 5:14pm

@Moin are you suggesting that moving messages would be preferable when you really want to avoid that?

martin · May 23, 2022, 11:29pm

Thanks for doing that – that’s completely jumbled up! I will have to do some local testing on larger message sets. I think at the very least this will be needed:

I am generally uneasy about ordering by ID though because of weird inconsistencies, I think ordering messages by created_at would be better generally for the channels. @joffreyjaffeux or @mcwumbly what are your thoughts on this? If we decide to do that then the message mover might need to artificially space out the created_at values by 10ms each or so for consistent ordering.

I think in general if they are totally irrelevant to the current channel it would be better to move to a more appropriate channel. We used this many times previously internally when we used Mattermost. For example, a bunch of incident response in the general channel that should be moved into the incident channel for better record keeping. Or, idle chatter in a channel that is better to be in the random channel.

I don’t think there is any value in those cases of quoting and leaving old cruft behind, and as Moin says, things can then be confusing, where the discussion is continued in two different places.

Keep in mind these two options do not currently exist. We removed “Move to Topic” because in the initial implementation it was making one post per chat message, and also not deleting the original messages in the channel. If in future we want to make this again it needs to:

a) quote batches of messages together (say 100 per post) using the chat quote feature and
b) delete the originals in the channel to avoid duplication.

mcwumbly · May 24, 2022, 1:14am

I’ll abstain from commenting on the implementation for ordering posts and let @joffreyjaffeux comment on that aspect of things.

Ah, yeah. I wasn’t asking about moving chat messages within chat, but I can see how that can be useful, and it doesn’t have the problem of trying to convert short-form into ling-form (or vice versa) “in post”.

Makes sense. I like the general form of quoting as more of a “transcript” like this because I think it’s likely going to read that way anyway. In the past, when I used the Slack transcript feature, I often found myself wrapping it in [details] as well, and summarizing things in the main post body.

Another thought I had along those lines might be to have a fancier “expand context” feature, so you could quote a single message, but then load additional messages inline on demand to view more context from chat without leaving the topic.

I’m skeptical about this part being necessary or valuable when referencing discussions across the slow lane/fast lane boundary.

martin · May 24, 2022, 2:02am

It’s only if you’re choosing Move to Topic that this would happen, why keep things around in the channel if your intention was to move it? We have had some discussions around this internally already. Sure just a normal quote of messages into a topic would not delete anything.

Some trivia for you, the class that generates the quotes is actually called ChatTranscriptService

This is interesting, we actually do have something similar to this with our topic quotes (you probably already have seen that). It would probably be useful to get a little more context without having to actually visit the channel.

sam · May 24, 2022, 3:11am

I would say the use case for moving is:

We have a channel dedicated to discussion about “Whales”
A bunch of people start having a an involved discussion about “Penguins” cause they forgot to click “#penguin” and stuff got heated
Mod steps in and kicks the Penguin talk to the Penguin channel.

I guess the fundemental thing here is the re-sequencing.

I would say “fudge created_at” is about the only sane solution here cause you want everything moved in one block? Plus it technically is created at the point of time it is moved.

mcwumbly · May 24, 2022, 3:11am

Yeah, I think I’m wondering whether it’s needed, or if quote/transcribe is the thing to focus on making work really well.

martin · May 24, 2022, 3:51am

Yes I would 100% do this if our normal GET messages route for a channel was ordered by created_at, that’s what I want to sort out, was just wondering if Joffrey had some historical knowledge about that. If not will change both things at once.

joffreyjaffeux · May 24, 2022, 9:32am

Yes Im 100% with sam and you😁 Moving everything in one go and giving it the created_at of the time of the move is the only sane approach IMO. Otherwise it opens a gigantic can of worms… how do I know where to find it? receiving unread notifications for things created before my last read? nope nope nope

martin · May 25, 2022, 12:15am

Nice, I will tweak the move tool to the moved messages be in the future with a small spaced out increment, and make the chat messages order by created_at instead of ID in the general controller

martin · June 1, 2022, 12:31am

Just merged this to hopefully address the issue:

github.com/discourse/discourse-chat

FIX: Message order consistency

discourse:main ← discourse:issue/fix-message-ordering-and-message-mover

opened 03:13AM - 30 May 22 UTC

martin-brennan

+27 -15

In https://meta.discourse.org/t/moving-chat-messages-to-a-new-channel-jumbled-th…em-all-up/227301 we were having issues with moved messages between channels being jumbled up. During investigations it was found that the ChatController /messages route ordered messages by ID but everything else, including the message mover, ordered by created_at. This commit makes things consistent by changing the /messages route to also order by created_at, then ID. This should make it so the order is always consistent in the UI and also for moved messages.

I didn’t do anything to artificially space out created_at for now into the future, so let’s see how this goes first.

martin · June 12, 2022, 10:00pm

This topic was automatically closed after 11 days. New replies are no longer allowed.

Topic		Replies	Views
Ability to move chat discussion to a topic? Feature chat	23	1321	May 16, 2023
Discourse email messages are incorrectly threaded Bug	86	4324	January 12, 2023
Inviting existing users to topics UX	32	2861	June 13, 2022
Introducing chat threads! Announcements chat , new-feature	33	6169	July 8, 2024
Could a List of Recommended Topics Be Added to the Top Menu? Feature ai	21	1261	June 26, 2024

Moving chat messages to a new channel jumbled them all up

Related topics