Posting is taking multiple seconds

We have 5M posts and searching is pretty snappy. Hosting on a 6 shared vcores of a 2.8Ghz Xeon E5-2680 with 16GB RAM and SSD storage. That said I don’t have metrics on how many simultaneous posts our users initiate, there might be locking issues.

Posting, on the other hand, is often quite slow. It can take 6-7 seconds for a post to go through. Discourse is non-blocking when it chugs away so that doesn’t kill the user experience, though.

1 Like

This is… not normal.

I think this is most likely the case on any topic with 100s or 1000s of posts. On short topics this should not be the case.

I recall @Wingtip’s forum has tons of very long topics.

2 Likes

Yes, we definitely do.

Are the 6-7 second post times specific to the mega topics? If you reply to a topic with 100 replies, does it also take 6-7 seconds?

I was unable to reproduce just now-- slow posting is intermittent. It doesn’t seem to correlate with post count in the thread. Maybe it’s a locking issue with someone else posting simultaneously or something.

7 posts = very fast, roughly the same as here on meta
500 posts = maybe slightly slower, not too bad. Maybe 1.5 second?
16k posts = seems about the same as with 500

1 Like

And I believe that performance is also affected if a bunch of people are accessing a mega topic, which slows down the whole PG server with queued requests.

That makes sense, to prioritize the experience of many people viewing a thread over one or two trying to post.

How brave are you feeling? Do you mind enabling mini_profiler for a bit? It will give you timings on the side, then we can isolate which query is playing up for you!

My post here took 576ms

The worst query is:

And @riking is working on a PR that will fix this :confetti_ball:

6 Likes

How much does enabling it impact overall performance? And can it be enabled without a site rebuild? I haven’t been able to get a rebuild to succeed since the postgres 12 change, even by setting it to stick with pg10.

You need to set your email address as the developer email in the app.yml file. You can apply it without a rebuild by destroying and restarting the container. It you have made changes to the container, by upgrading with discourse docker they will be lost.

You can also edit config/discourse conf inside the container and then

  sv restart unicorn

Given that rebuilds are failing, my concern is if I destroy the container I’ll be completely out of luck. Had so many errors with attemping the pg12 upgrade yesterday that I’m really not comfortable making changes until I have confidence I won’t take the site down for an extended period.

1 Like

Well, free advice is worth what you pay for it, but it’s pretty safe to

cd /var/discourse
./launcher enter app
vi config/discourse.conf
# in vi, edit the developer_emails line to have your email address and quit vi
sv unicorn restart
# panic for the 30-90 seconds it takes the web server to restart and begin serving pages

You can still break things this way, though. I think it’s possible to muck up that config file such that discourse won’t start.

sv restart unicorn worked, sv unicorn restart did not.

I get that backward about 1/3 of the time. :wink:

This is an improvement, though, as I used to get it wrong half the time.

1 Like

Is this what you needed? This is posting to a thread with 16k posts. This seemed like a normal post time, not particularly slow.

Here’s the full output:
https://paste.mozilla.org/f5nJrPag

2 Likes

It’s a 2s query coming from app/models/user_stat.rb:159:in `update_topic_reply_count’.

So it’s the exact thing @riking is working on.

4 Likes

Great! Any substantial improvements to post time will really make our users happy, this is the most complained-about thing at the moment. Although I have no doubt they’ll find something else to whine about shortly thereafter.

3 Likes

If it makes you feel any better, I complained about that query as well, because I have so much read state here on meta. We’re doing pointless work to count stuff on every reply…

1 Like

In our test import from another software, we have a topic with a bit over 200,000 posts and I was able to post to it just fine. It must be because it’s not the topic size that matters, but rather the user who is doing the posting.

1 Like