Major slowdown for staff in large topics

I have a forum with many large threads (16k+ posts). After installing 2.3.0.beta10 +14, I notice a severe slowdown in large threads. The following actions take multiple seconds now:

  • Entering thread
  • Jumping inside thread
  • Scrolling inside thread

When any of these requests is running, the corresponding unicorn process uses 100% of one CPU core/thread for the duration of the request.

In a currently active thread with ~19700 posts, these actions take ~4.5s each. Smaller threads take less time, but the amount of posts has no reliable relation to the amount of time—a thread with 15600 posts take 244ms while another thread (recently active) with 11000 posts takes 2.6s.

Regular users are unaffected by this issue. Moderators and administrators are affected.

Unfortunately, the forum is not publicly accessible.

3 Likes

Hi Daniel, this problems should be expected with megatopics, see The MEGATOPIC: public good, or public menace?

We recommend instead that you make more topics linked to each other instead of just gigantic one.

4 Likes

I realize big topics are discouraged. Still, I strongly disagree. Since this is a regression, it is not to be expected. Also, because there is no direct relation between post count and response time, something else is the deciding factor, like maybe different users in a thread. On a forum with more active users, the problem may manifest a lot earlier.

I didn’t have any performance problems whatsoever since deploying Discourse more than two years ago, even in topics with nearly 30000 posts. I firmly believe this issue can be resolved.

2 Likes

Is it possible there was a regression here @tgxworld?

I have, anecdotally, noticed that the larger threads in my forum seem to be loading slower recently. For a bit a couple weeks ago it was quite rough @ 2s to load a 2K post topic, but it’s settled down to around 1s now.

I assumed this was isolated to my installation (picked up a bad neighbor in Digital Ocean?), and would note that I’m currently only on 2.3.0beta9+593, so perhaps different. But, so, presents anecdata.

I will note that this seems to hold true for me:

At the very least, on the same computer/browser, loading a larger topic as a guest or regular user is significantly faster than on my current (admin) account.

4 Likes

cc @sam @tgxworld make sure you note this part

Can you run mini profiler? You will need to be a developer for that per: Description of various user states in Discourse

What queries are running slow?

1 Like

UPDATE I just found what could be the root cause of the slowdown, the new reviewable system is not windowing. We will get it sorted.

Can you confirm that mega topics are fine if you are anonymous?

3 Likes

Anonymous mode appears to work fine on my end, nice snappy topic loads.

1 Like

Until the following query, everything is fine:

SELECT "reviewables".* FROM "reviewables" WHERE "reviewables"."target_type" = 'Post' AND "reviewables"."target_id" IN (<all the post ids>)

The query itself is fast, too (Reader: 19.6 ms), if I understand correctly. Firefox is close to crashing from displaying the backtrace though. :smiley:

The next query is:

SELECT  "posts"."id" FROM "posts" WHERE "posts"."topic_id" = 1466 AND (posts.user_id = 1 OR post_type IN (1,2,3,4)) AND (posts.deleted_at IS NULL OR posts.post_number = 1) ORDER BY "posts"."sort_order" ASC LIMIT 1; 

…which is already delayed by an additional 4200ms on the timeline.


Also, yes, if I use a non-admin, non-mod user, everything is as fast as it should be.

2 Likes

Sorry about this, I will get it fixed today!

5 Likes

Here’s a fix:

https://github.com/discourse/discourse/commit/f86a5bd5a9e603a71c1b95e71805b0f2044c85d0

14 Likes

My 1-2s loads are back down to <200ms, so I’d say that worked pretty well. :+1:

8 Likes