Continuing the discussion from Very poor performance in topics with a large number of posts:
Just wondering, what will you guys be doing that will improve the performance?
1 Like
As I understand it every post id in the topic is sent down at the time of topic request. Not sure when we started doing that, but it obviously becomes crippling on large topics.
Not a huge priority at the moment as there are only imported topics large enough to be a problem. No other “natural” real world examples of enormous topics yet.
It actually happens in two spots.
- When we initially grab the topic we grab every post_id in the topic
- When we grab any additional posts in the topic (eg: you scroll up or down and load a new chunk) we grab every post_id in the topic.
Largest topic we have that we host is: Did you ever want to play questions? - games - Boing Boing BBS
1 Like
This has been in place for quite some time, probably years. I knew when I did it that it would be a problem for really long topics but it bought us a lot of stability.
Consider this technical problem: “Best of” is on and you’ve filtered to three distinct users. There are more than 20 posts and the user scrolls down. If the client knows all the filtered post ids it can just say, hey give me posts 1, 2, 3, 4, 5 and that’s super fast for the server to reply with, without the server having to calculate the entire list on every request to get the next X items.
Now that we’re feeling the growing pains it should probably be revisited to accommodate huge topics. There are a couple of approaches we could do here. It will just take some time to do it properly.
8 Likes
Any progress on this issue? We have a topic with ~500 replies and it’s quite painful to scroll through.
Yes, this was addressed in Discourse 1.5.
Ah we are on 1.7 and the scroll behavior is still quite laggy. It appears to load replies in batches of 20, with a significant wait time between each batch. With 500 replies it can take minutes to scroll all the way through. Are we missing a configuration parameter that might eliminate these pauses?
Why are you scrolling without reading?
-
If you need to jump to the bottom, click the bottom date on the timeline at the right… or press end on your keyboard.
-
If you need to enter the topic at the bottom, click the last post date in the topic list.
4 Likes
I noticed a couple of months ago that our major off-topic/spam thread was becoming slower than other threads. Yesterday a user finally complained about it. At 188.7k replies it consistently takes 4 seconds or longer to perform any navigation in the thread, while other threads take less than a second.
We’re used to shutting these threads down and creating news ones, as this was a problem on vBulletin as well. I’m not a forum architect so I’m not here to say it can (or should) be fixed, and I don’t know how many other forums have topics with so many replies, but I figured I’d put the information here. This is obviously nonessential, but if you ever achieve all of your hopes for Discourse and want one last thing to work on, I guess there’s this. 
1 Like
Ah, we did. I guess we didn’t consider that there might be consequences. We have three other threads that are longer than 10k posts at the moment, so I guess it’d be good to keep an eye on them.
When auto-closing, is there any sort of thread re-creation that ties the new thread back to the old one? Just curious, as we might re-enable that limit if so. Otherwise we’ll just handle long threads ourselves since they don’t crop up too often.
Links between topics always connect them, this has been true of Discourse since beta. So just add a link to a post pointing to the new topic.
2 Likes
Whats the performance impact of long threads?
I have one that has escalated to 4.5K very rapidly and will keep escalating for at least 36 hours. Expecting thousands of messages, server is screaming.
Will it help anything to close the thread and start a new one, a “part 2” to continue the discussion? Ping @codinghorror, @sam
I would recommend keeping topics below the 10k count for now. Worth closing and doing a part2 when it reaches that point.
2 Likes
Yeah, I am guessing it will be below 10K and cools down tomorrow midnight (trade deadline). Just wondering if it gives me any benefit to cut it as we have our annual super peak.
We keep deferring this work, the bandage to stop the bleeding was a new default that auto-closes topics at 10k replies… but the underlying issue is still present.
1 Like
Only ignorant brainstorming. If the bottleneck is a large number of posts belonging to a topic, would there be a way to insert a middle layer something like
topic has many blocks
block has many posts
similar to changing a large array into a multidimensional array but for database queries.
I stopped this bleeding in
https://github.com/discourse/discourse/commit/bad6a5142cd103b95a64250b3b6e89539187e279
While I was looking through the client side code, I noticed that the client only cares about the posts in the response payload and doesn’t do anything with the stream (all the post ids).
2 Likes
Maybe only the timeline does, and @sam already optimized that yesterday (by removing date from the timeline), which should also be covered in this topic I think.
From what I’m seeing the client do with the response I’m 99% sure that we don’t need to send down all the post ids when scrolling.
Maybe @eviltrout can review?
Yup we removed the date in
https://github.com/discourse/discourse/commit/9c925a66ff2b8bb19dd0d587606910e87b47e488
However, it didn’t actually affect the performance of the query much when we tried it out. The root cause is due to the fact that we’re over selecting the ids here.
Instead of having the client send the post ids that it wants to the server while scrolling, I plan to have the client send the current position and tell the server whether it is looking up/down the stream. I’ve got it working locally but I’ll need to sort out all the other edge cases.
5 Likes