Our Discourse site has more than 10k users and about 700 daily engaged users. users sending an average of 10k posts per day. our community has more than 160k pageviews/day including crawlers and anonymous users. Almost all of our users are connected to us via mobile devices.
We ran community in standalone setup mode on a single VPS with 16 CPU cores and 24gb of ram and config app.yml with these values:
With the above config, some users says site load slowly for them, and some times while submitting a post, the screen goes blank (The header is still shown). Also, sometimes during peak hours, site performance slows down.
Please explain if we did the wrong configuration or need more resources.
Thanks a lot
10k posts per day is pretty high, relative to the amount of pageviews. I can imagine that you would be hitting some resource limits here given your configuration and I suspect it will be the database. You could try to move to a multi container setup, effectively offloading the unicorn workers from the main server.
It could very well be, but I would first try to scale anything that can scale horizontally, horizontally. It will also give you a much better idea of where your bottleneck is.
Most performance issues can be solved by simply throwing more resources against the problem. The hard part is doing that in a smart way so you can save some bucks (or potentially a lot of money).
Thanks for your expertise and friendly advice, I will definitely read about how to implement this. another question I have is what settings should be applied in the app for the specifications I mentioned above. (24 CPU cores with 32gb of ram)
Are the current settings appropriate or is it better to increase the values?
Hard to say without inspecting the system and see what is going on.
Since you said the most issues happen when submitting a post, the issue is probably with database writes and I don’t think increasing shared buffers any further will be of much help, but you could try. I’ve seen it cranked up to over 50% of memory (against all advice) so you could try to gradually raise it to up to 12GB.
If you’re not seeing 502’s then there is no use in increasing UNICORN_WORKERS either.
You don’t mention using it so that’s why I think, the first thing I would do is adding a CDN this would reduce a lot of weight from the VPS as the bigger requests would not touch the server.
Adding to the CDN, I would also go with S3-like storage that would allow you to scale differently the storage and the VPS resources (if your community is uploads heavy).
This recommendations help a lot to reduce load and the price increase is a lot lower than a bigger VPS.
Thanks @marianord, Unfortunately, we do not use CDN. The upload rate in our forum is not very high. Most of the time, users talk about various subjects. For example, in the last year, we had about 2.8 million posts and 2.7 million likes, but only 25gb of files are uploaded.
Do you think that according to the information I wrote, using the CDN like s3 will reduce our server load?
I disagree with @marianord. I don’t think that a CDN would make a noticeable difference with respect to the load on your server.
These are just static files and they are not heavy to serve at all.
S3 offloads the files and backups to another server that is managed by a cloud provider (really high level summary)
CDN caches the statics files of your server (images, JS, CSS) to server from multiple servers (PoP) around the globe to speed up the loading of this assets.
At least that is my experience, you’re reducing the amount of requests that get to your server therefore reducing the load. It’s a lot easier to serve only 10 JSON requests per user that to serve 100 requests per user.
Maybe this will not solve all the issues @nildarar is facing but will reduce the heavier load of the server removing all the static requests (the cached ones) from the Discourse server.
A request for a static file does not have a large impact on the overall server load. The requests for dynamic content are the heavy ones.
In general, a json request is not a static asset that will be cached by a CDN. It is dynamic content which is generated on the fly. Why are you talking about json files in a CDN context?
static requests != heavier load.
I am sorry but this is really bad advice.
This is an example from a 6-CPU machine (so CPU adds up to 600%) running Discourse without a CDN or S3.
You can see that nginx is only responsible for 6.7% (so that is 1/100 of the capacity). Only a part of that is used for static assets.
If we were to offload the static assets to S3 and/or a CDN it would reduce the overall server load by less than one percent.
True, but Discourse has a few exceptions, like stylesheets that are served by ruby, so having a caching CDN means those requests doesn’t use up the unicorn processes.
Regarding the OP problem, the first thing needed is having someone knowledgeable do some performance analysis during peak hours and identifying what is the current bottleneck.
Thanks for your guidance, until a few months ago we used the Cloudflare CDN service and made good changes to the static content through the page rules. After it, I read somewhere that using proxies like Cloudflare drastically reduces the performance of the discourse, so we disabled it.
Yesterday we increased the CPU cores from 16 to 24 and made the following changes in the app.yml
With these changes, our problem was temporarily solved, but I think we should make a fundamental change for the next few months from now.
So according to your recommendations, using CDN for serving static contents and splitting the discourse into two separate containers has a higher priority in performance improvements.
This may be older information, but I do recall reading that discourse prefers a lower number of more powerful CPU’s vs a higher number of lower powered ones… even if you update the number of unicorn workers.
@codinghorror, can you confirm if this information is still accurate?
Using something like Prometheus + Grafana can help you just to get the historical of the data, instead of seeing it live and then do some deeper analysis of what is happening.
Hello again
Following your tips, we installed Prometheus and monitored the community performance for a while. Please see the reports below and check with the values you see in different installations.
