# Performance improvements on long topics?
**URL:** https://meta.discourse.org/t/performance-improvements-on-long-topics/30187
**Category:** Support
**Created:** [June 19, 2015, 1:05am UTC](https://meta.discourse.org/t/performance-improvements-on-long-topics/30187 "2015-06-19T01:05:44Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@nahtnam](https://meta.discourse.org/u/nahtnam)
#### Post date: [June 19, 2015, 1:05am UTC](https://meta.discourse.org/t/performance-improvements-on-long-topics/30187/1 "2015-06-19T01:05:45Z")
Continuing the discussion from [Very poor performance in topics with a large number of posts](https://meta.discourse.org/t/very-poor-performance-in-topics-with-a-large-number-of-posts/30170/4):
> [@sam](#):
>
> This is a known issue and is slotted to be fixed in our next major release.
>
> Its a rather complex fix which will take at least a few days of work and will probably be done by @eviltrout
Just wondering, what will you guys be doing that will improve the performance?
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [June 19, 2015, 2:01am UTC](https://meta.discourse.org/t/performance-improvements-on-long-topics/30187/2 "2015-06-19T02:01:13Z")
As I understand it every post id in the topic is sent down at the time of topic request. Not sure when we started doing that, but it obviously becomes crippling on large topics.
Not a huge priority at the moment as there are only imported topics large enough to be a problem. No other “natural” real world examples of enormous topics yet.
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [June 19, 2015, 2:08am UTC](https://meta.discourse.org/t/performance-improvements-on-long-topics/30187/3 "2015-06-19T02:08:09Z")
> [@codinghorror](#):
>
> As I understand it every post id in the topic is sent down at the time of topic request
It actually happens in two spots.
1. When we initially grab the topic we grab every post\_id in the topic
2. When we grab any additional posts in the topic (eg: you scroll up or down and load a new chunk) we grab every post\_id in the topic.
Largest topic we have that we host is: [Did you ever want to play questions? - games - Boing Boing BBS](http://bbs.boingboing.net/t/do-you-want-to-play-questions/54417)
---
### Author:  [@eviltrout](https://meta.discourse.org/u/eviltrout)
#### Post date: [June 19, 2015, 2:48pm UTC](https://meta.discourse.org/t/performance-improvements-on-long-topics/30187/4 "2015-06-19T14:48:55Z")
> [@codinghorror](#):
>
> Not sure when we started doing that, but it obviously becomes crippling on large topics.
This has been in place for quite some time, probably **years**. I knew when I did it that it would be a problem for really long topics but it bought us a lot of stability.
Consider this technical problem: “Best of” is on and you’ve filtered to three distinct users. There are more than 20 posts and the user scrolls down. If the client knows all the filtered post ids it can just say, hey give me posts 1, 2, 3, 4, 5 and that’s super fast for the server to reply with, without the server having to calculate the entire list on every request to get the next X items.
Now that we’re feeling the growing pains it should probably be revisited to accommodate huge topics. There are a couple of approaches we could do here. It will just take some time to do it properly.
---
### Author:  [@hammer](https://meta.discourse.org/u/hammer)
#### Post date: [November 8, 2016, 9:43pm UTC](https://meta.discourse.org/t/performance-improvements-on-long-topics/30187/5 "2016-11-08T21:43:17Z")
Any progress on this issue? We have a topic with ~500 replies and it’s quite painful to scroll through.
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [November 8, 2016, 9:51pm UTC](https://meta.discourse.org/t/performance-improvements-on-long-topics/30187/6 "2016-11-08T21:51:09Z")
Yes, this was addressed in Discourse 1.5.
---
### Author:  [@hammer](https://meta.discourse.org/u/hammer)
#### Post date: [November 8, 2016, 10:05pm UTC](https://meta.discourse.org/t/performance-improvements-on-long-topics/30187/7 "2016-11-08T22:05:31Z")
Ah we are on 1.7 and the scroll behavior is still quite laggy. It appears to load replies in batches of 20, with a significant wait time between each batch. With 500 replies it can take minutes to scroll all the way through. Are we missing a configuration parameter that might eliminate these pauses?
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [November 8, 2016, 11:01pm UTC](https://meta.discourse.org/t/performance-improvements-on-long-topics/30187/8 "2016-11-08T23:01:34Z")
Why are you scrolling without reading?
- If you need to jump to the bottom, click the bottom date on the timeline at the right… or press end on your keyboard.
- If you need to enter the topic at the bottom, click the last post date in the topic list.
---
### Author:  [@Tsirist](https://meta.discourse.org/u/Tsirist)
#### Post date: [January 9, 2017, 11:59pm UTC](https://meta.discourse.org/t/performance-improvements-on-long-topics/30187/9 "2017-01-09T23:59:42Z")
I noticed a couple of months ago that our major off-topic/spam thread was becoming slower than other threads. Yesterday a user finally complained about it. At 188.7k replies it consistently takes 4 seconds or longer to perform any navigation in the thread, while other threads take less than a second.
We’re used to shutting these threads down and creating news ones, as this was a problem on vBulletin as well. I’m not a forum architect so I’m not here to say it can (or should) be fixed, and I don’t know how many other forums have topics with so many replies, but I figured I’d put the information here. This is obviously nonessential, but if you ever achieve all of your hopes for Discourse and want one last thing to work on, I guess there’s this. 😂
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [January 10, 2017, 12:36am UTC](https://meta.discourse.org/t/performance-improvements-on-long-topics/30187/10 "2017-01-10T00:36:42Z")
By default we auto-close topics that reach 10k replies. So did you bypass that default?
---
### Author:  [@Tsirist](https://meta.discourse.org/u/Tsirist)
#### Post date: [January 10, 2017, 1:32am UTC](https://meta.discourse.org/t/performance-improvements-on-long-topics/30187/11 "2017-01-10T01:32:30Z")
Ah, we did. I guess we didn’t consider that there might be consequences. We have three other threads that are longer than 10k posts at the moment, so I guess it’d be good to keep an eye on them.
When auto-closing, is there any sort of thread re-creation that ties the new thread back to the old one? Just curious, as we might re-enable that limit if so. Otherwise we’ll just handle long threads ourselves since they don’t crop up too often.
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [January 10, 2017, 2:17am UTC](https://meta.discourse.org/t/performance-improvements-on-long-topics/30187/12 "2017-01-10T02:17:41Z")
Links between topics always connect them, this has been true of Discourse since beta. So just add a link to a post pointing to the new topic.
---
### Author:  [@ljpp](https://meta.discourse.org/u/ljpp)
#### Post date: [February 14, 2017, 1:43pm UTC](https://meta.discourse.org/t/performance-improvements-on-long-topics/30187/13 "2017-02-14T13:43:17Z")
Whats the performance impact of **long** threads?
I have one that has escalated to 4.5K very rapidly and will keep escalating for at least 36 hours. Expecting thousands of messages, server is screaming.
Will it help anything to close the thread and start a new one, a “part 2” to continue the discussion? Ping @codinghorror, @sam
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [February 14, 2017, 1:53pm UTC](https://meta.discourse.org/t/performance-improvements-on-long-topics/30187/14 "2017-02-14T13:53:50Z")
> [@ljpp](#):
>
> Will it help anything to close the thread and start a new one, a “part 2” to continue the discussion? Ping @codinghorror, @sam
I would recommend keeping topics below the 10k count for now. Worth closing and doing a part2 when it reaches that point.
---
### Author:  [@ljpp](https://meta.discourse.org/u/ljpp)
#### Post date: [February 14, 2017, 1:57pm UTC](https://meta.discourse.org/t/performance-improvements-on-long-topics/30187/15 "2017-02-14T13:57:52Z")
Yeah, I am guessing it will be below 10K and cools down tomorrow midnight (trade deadline). Just wondering if it gives me **any benefit** to cut it as we have our annual super peak.
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [February 25, 2018, 10:19am UTC](https://meta.discourse.org/t/performance-improvements-on-long-topics/30187/16 "2018-02-25T10:19:22Z")
We keep deferring this work, the bandage to stop the bleeding was a new default that auto-closes topics at 10k replies.. but the underlying issue is still present.
---
### Author:  [@Mittineague](https://meta.discourse.org/u/Mittineague)
#### Post date: [February 25, 2018, 10:31am UTC](https://meta.discourse.org/t/performance-improvements-on-long-topics/30187/17 "2018-02-25T10:31:23Z")
Only ignorant brainstorming. If the bottleneck is a large number of posts belonging to a topic, would there be a way to insert a middle layer something like
topic has many blocks
block has many posts
similar to changing a large array into a multidimensional array but for database queries.
---
### Author:  [@tgxworld](https://meta.discourse.org/u/tgxworld)
#### Post date: [June 22, 2018, 9:17am UTC](https://meta.discourse.org/t/performance-improvements-on-long-topics/30187/18 "2018-06-22T09:17:37Z")
> [@sam](#):
>
> When we grab any additional posts in the topic (eg: you scroll up or down and load a new chunk) we grab every post\_id in the topic.
I stopped this bleeding in
[https://github.com/discourse/discourse/commit/bad6a5142cd103b95a64250b3b6e89539187e279](https://github.com/discourse/discourse/commit/bad6a5142cd103b95a64250b3b6e89539187e279)
While I was looking through the client side code, I noticed that the client only cares about the posts in the response payload and doesn’t do anything with the `stream (all the post ids)`.
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [June 22, 2018, 9:32am UTC](https://meta.discourse.org/t/performance-improvements-on-long-topics/30187/19 "2018-06-22T09:32:48Z")
> [@tgxworld](#):
>
> I noticed that the client only cares about the posts in the response payload and doesn’t do anything with the `stream (all the post ids)` .
Maybe only the timeline does, and @sam already optimized that yesterday (by removing date from the timeline), which should also be covered in this topic I think.
---
### Author:  [@tgxworld](https://meta.discourse.org/u/tgxworld)
#### Post date: [June 22, 2018, 9:53am UTC](https://meta.discourse.org/t/performance-improvements-on-long-topics/30187/20 "2018-06-22T09:53:15Z")
> [@codinghorror](#):
>
> Maybe only the timeline does
From what I’m seeing the client do with the response I’m 99% sure that we don’t need to send down all the post ids when scrolling.
>
Maybe @eviltrout can review?
> [@codinghorror](#):
>
> @sam already optimized that yesterday (by removing date from the timeline), which should also be covered in this topic I think.
Yup we removed the date in
[https://github.com/discourse/discourse/commit/9c925a66ff2b8bb19dd0d587606910e87b47e488](https://github.com/discourse/discourse/commit/9c925a66ff2b8bb19dd0d587606910e87b47e488)
However, it didn’t actually affect the performance of the query much when we tried it out. The root cause is due to the fact that we’re over selecting the ids here.
> [@eviltrout](#):
>
> Consider this technical problem: “Best of” is on and you’ve filtered to three distinct users. There are more than 20 posts and the user scrolls down. If the client knows all the filtered post ids it can just say, hey give me posts 1, 2, 3, 4, 5 and that’s super fast for the server to reply with, without the server having to calculate the entire list on every request to get the next X items.
Instead of having the client send the post ids that it wants to the server while scrolling, I plan to have the client send the current position and tell the server whether it is looking up/down the stream. I’ve got it working locally but I’ll need to sort out all the other edge cases.
[Next page](https://meta.discourse.org/t/performance-improvements-on-long-topics/30187.md?page=2)