My discourse speed is very slow

ahmed_khan1 · March 3, 2021, 4:06pm

i already have run discourse-setup that didn’t make any difference in speed

yes we have more than 27000 topics on our site

lhkjacky · March 3, 2021, 4:30pm

In my experience, Topic List Previews (Plugin) slow down discourse significantly. You may try switch to Topic List Previews (TC), it load much faster than Plugin.

If you disable the Topic List Previews completely, Discourse will speed up several times.

Faizan_Zahid · March 3, 2021, 5:54pm

We are using a custom plugin to allow topics to still show the cdn images shared on topics , in meta OG if we share them on social media. That cdn support plugin is developed by @fzngagan , faizan, do you have any way to move your written plugin to TLP TC so that site can load faster?

@pfaffman Do you have any specific tech suggestions whatexactly do we need to do to speed up our database and website, I m assuming on howtodiscuss.com it has over 50K+ topics and it’s a high traffic website.

My questions :

Is there any multi threading or parallel CPU or cores configuration we can set up in discourse ? nodebb provides this to cope up with high traffic.
Any server side caching support in discourse , where and how to configure it?
How can we achieve 100% Google page speedscore for https://howtodiscuss.com
what changes can we do in app.yaml file? we r on 4 Vcpu, 8GB digital ocean droplet.
we r using cloudflare to serve our site. any optimizations we can do there? when I turn on rocket loader of cloudflare, my discourse doesnt work properly and gives errors in js console client side.
any way to auto minify css, js, html, and enable gzip compression on server side?
Please share insights on what optimization you guys have done on meta.discourse.org to cope up with higher traffic, we can try the same.
Please share insights on optimizations which other high traffic sites on discourse have tried?

Once we are able to speed up our website, we are happy for you to post our site also in successful discourse communities list. Thanks

pfaffman · March 3, 2021, 5:59pm

Well, the number of topics isn’t the number of posts or the size of the database, but that isn’t so big.

The site is down now, so I can’t see the number of posts.

I recommend that you edit app.yml by hand and look at the tuning information there and rebuild. There are several topics that discuss ways to tune the database.

You might also switch to one of the new Digital Ocean faster CPU droplets.

dadberg · March 3, 2021, 6:23pm

Hey, @ahmed_khan1 Don’t risk any downtime, even few minutes down will cost you precious reputation in Google eyes.

Just pay someone professional to handle this for you, once and forever. As I can see your site got a sizeable traffic, so follow the sanity and never go for cheap.

I recommend someone like @pfaffman and @angus.

merefield · March 3, 2021, 9:40pm

I’m running a 20,000 Topic site on a 2 Core, 2GB machine with no issues (but may not be as active as yours …)

But yeah, if it’s an issue with running a lot of javascript, then the back end performance is irrelevant.

Have you set this to zero?

That will help a little.

You can move to the TC to speed TLP up, but it does have a rendering limitation in Chromium based browsers (hence why I’ve not decommissioned the plugin so far, Im waiting for LayoutNG to upgrade Grid CSS).

ahmed_khan1 · March 4, 2021, 8:16am

we have a new digitalocean droplet. just purchase it on 1
here’s app.yml

## this is the all-in-one, standalone Discourse Docker container template

##

## After making changes to this file, you MUST rebuild

## /var/discourse/launcher rebuild app

##

## BE *VERY* CAREFUL WHEN EDITING!

## YAML FILES ARE SUPER SUPER SENSITIVE TO MISTAKES IN WHITESPACE OR ALIGNMENT!

## visit http://www.yamllint.com/ to validate this file as needed

templates:

  - "templates/postgres.template.yml"

  - "templates/redis.template.yml"

  - "templates/web.template.yml"

  - "templates/web.ratelimited.template.yml"

## Uncomment these two lines if you wish to add Lets Encrypt (https)

#- "templates/web.ssl.template.yml"

#- "templates/web.letsencrypt.ssl.template.yml"

## which TCP/IP ports should this container expose?

## If you want Discourse to share a port with another webserver like Apache or nginx,

## see https://meta.discourse.org/t/17247 for details

expose:

  - "2045:80" # http

#  - "443:443" # https

params:

  db_default_text_search_config: "pg_catalog.english"

  ## Set db_shared_buffers to a max of 25% of the total memory.

  ## will be set automatically by bootstrap based on detected RAM, or you can override

  db_shared_buffers: "2048MB"

  ## can improve sorting performance, but adds memory usage per-connection

  #db_work_mem: "40MB"

  ## Which Git revision should this container use? (default: tests-passed)

  #version: tests-passed

env:

  LANG: en_US.UTF-8

  # DISCOURSE_DEFAULT_LOCALE: en

  ## How many concurrent web requests are supported? Depends on memory and CPU cores.

  ## will be set automatically by bootstrap based on detected CPUs, or you can override

  UNICORN_WORKERS: 8

  ## TODO: The domain name this Discourse instance will respond to

  ## Required. Discourse will not work with a bare IP number.

  DISCOURSE_HOSTNAME: "howtodiscuss.com"

  ## Uncomment if you want the container to be started with the same

  ## hostname (-h option) as specified above (default "$hostname-$config")

  #DOCKER_USE_HOSTNAME: true

  ## TODO: List of comma delimited emails that will be made admin and developer

  ## on initial signup example 'user1@example.com,user2@example.com'

  DISCOURSE_DEVELOPER_EMAILS: "admin@gmail.com"

  ## TODO: The SMTP mail server used to validate new accounts and send notifications

  # SMTP ADDRESS, username, and password are required

  # WARNING the char '#' in SMTP password can cause problems!

  DISCOURSE_SMTP_ADDRESS: smtp-relay.smtp.com

  DISCOURSE_SMTP_PORT: 587

  DISCOURSE_SMTP_USER_NAME: admin@gmail.com

  DISCOURSE_SMTP_PASSWORD: smtp_password

  #DISCOURSE_SMTP_DOMAIN: discourse.example.com    # (required by some providers)

  #DISCOURSE_NOTIFICATION_EMAIL: nobody@discourse.example.com    # (address to send notifications from)

  #DISCOURSE_SMTP_ENABLE_START_TLS: true           # (optional, default true)

  ## If you added the Lets Encrypt template, uncomment below to get a free SSL certificate

  #LETSENCRYPT_ACCOUNT_EMAIL: me@example.com

  #DISCOURSE_MAXMIND_LICENSE_KEY: 1234567890123456

  ## The http or https CDN address for this Discourse instance (configured to pull)

  ## see https://meta.discourse.org/t/14857 for details

  #DISCOURSE_CDN_URL: https://discourse-cdn.example.com

## The Docker container is stateless; all data is stored in /shared

volumes:

  - volume:

      host: /var/discourse/shared/standalone

      guest: /shared

  - volume:

      host: /var/discourse/shared/standalone/log/var-log

      guest: /var/log

## Plugins go here

## see https://meta.discourse.org/t/19157 for details

hooks:

  after_code:

    - exec:

        cd: $home/plugins

        cmd:

          - git clone https://github.com/discourse/docker_manager.git

          - git clone https://github.com/discourse/discourse-sitemap.git

          - git clone https://github.com/discourse/discourse-math.git # custom plugin for math JAX support

          - git clone https://github.com/discourse/discourse-solved.git # custom plugin for question and answers and choosing best answer

          - git clone https://github.com/discourse/discourse-voting.git # custom plugin for voting on topics only

          - git clone https://github.com/discourse/discourse-yearly-review.git # custom plugin for automatically posting a yearly topic of forum activity

          - git clone https://github.com/discourse/discourse-user-notes.git # custom plugin to allow staff to add notes about a user

          - git clone https://github.com/paviliondev/discourse-question-answer.git # custom non official plugin for SO style QnA voting on posts

          - git clone https://github.com/discourse/discourse-adplugin.git # custom official plugin for ads

          - git clone https://github.com/discourse/discourse-whos-online.git # custom plugin to see who is online

          - git clone https://github.com/paviliondev/discourse-topic-previews.git # custom plugin for showing topics previews nicely

          - git clone https://github.com/paviliondev/discourse-ratings.git # custom plugin to allow user rating on topics

          - git clone https://github.com/discourse/discourse-cakeday.git # official plugin for birthdays and join date celebrations

          - git clone https://github.com/discourse/discourse-saved-searches.git # official plugin for saved search notifications

          - git clone https://github.com/paviliondev/discourse-follow.git # custom plugin to allow users to follow each other

          - git clone https://github.com/FaizanZahid/discourse-amp-htd-plugin.git # custom build AMP support for HTD

          - git clone https://github.com/discourse/discourse-assign.git # custom for assigning topics to staff

          - git clone https://github.com/paviliondev/x-discourse-tlp-thumbnail.git # custom plugin for showing thumbnails from CDN images

## Any custom commands to run after building

run:

  - exec: echo "Beginning of custom commands"

  ## If you want to set the 'From' email address for your first registration, uncomment and change:

  ## After getting the first signup email, re-comment the line. It only needs to run once.

  - exec: rails r "SiteSetting.notification_email='noreply@howtodiscuss.com'"

  - exec: echo "End of custom commands"

fzngagan · March 4, 2021, 11:27am

I think you’re running multiple sites on this single instance. Not sure how that would affect the performance issues.

ahmed_khan1 · March 4, 2021, 1:35pm

we are running only this site on this server.

pfaffman · March 4, 2021, 1:48pm

You might bump db_work_mem to 100.

You might have issues with your reverse proxy that is slowing things down.

What does the mini profiler show?

You can also install the prometheus plugin to get some more data.

renato · March 4, 2021, 2:19pm

Have you tried disabling the Who’s Online plugin?

ahmed_khan1 · March 4, 2021, 4:29pm

I have tried but don’t see a big improvement
@pfaffman
here are CPU and memory usage

i have changed unicorn workers to 24
and memory buffers to 4096MB

merefield · March 4, 2021, 4:33pm

Have you looked at the browser end?

Granted this is a snapshot but your CPU utilisation looks tiny. I think you are over speccing.

Memory use is only high because of the number of unicorns. Reduce those if it starts to hit swap badly.

I would definitely look at the data retrieval and JavaScript latencies on the development tools of the browser.

Perceived slowness may be down to the amount of work the browser is doing.

gerhard · March 4, 2021, 4:56pm

From the HTTP headers it looks like you are using Ezoic and Cloudflare. Is that correct? I’m not sure what it does, but that might be another factor for perceived slowness.

My recommendation: Start with a simple setup without any proxies or putting your Discourse behind Cloudflare. Make sure it’s reasonable fast and only then start optimizing by following Enable a CDN for your Discourse or, if you really want to use Cloudflare, Full site CDN acceleration for Discourse.

Faizan_Zahid · March 4, 2021, 8:01pm

That’s correct, only 1 site, which is https://howtodiscuss.com is being served on this server via NGINX. Do you have any ways to boost its speed ?

PaulMw · March 4, 2021, 8:19pm

Try it with ezoic switched off, it can be a real bottleneck.

Faizan_Zahid · March 4, 2021, 8:19pm

@gerhard

Yes we are using Cloudflare CF, but i assume people use CF to speed up the website so thats why we are using it.
And yes, we linked our CF account with Ezoic but we are still not using Ezoic and not serving ads from them. I did not find any way to remove Ezoic integration with my own CF account.

Right now, we are using Digital ocean single droplet, and only this website is hosted on the entire server. Website is served using NGINX which looks something like this, made some values dummy: We r using NGINX because we are also serving AMP pages via PHP on https://howtodiscuss.com/t/amazon-erc-number/29795/?amp=1

server {
listen httpsPORT ssl http2;

ssl on;
ssl_certificate         /path/to/pem;
ssl_certificate_key     /path/to/key;
server_name howtodiscuss.com www.howtodiscuss.com;
location / {
    proxy_ssl_server_name       on;
    proxy_pass       http://localhost:port;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    proxy_set_header X-NginX-Proxy true;

    proxy_redirect off;

    # Socket.IO Support
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_ssl_protocols         TLSv1 TLSv1.1 TLSv1.2;
        




}

And our website is already served via Cloudflare, so that should still speed up things but its not seem to speeding it up.

And you mentioned about these, we will try, is this all free to deploy and run, like hosting my entire forum on full Cloudflare CDN, is this 100% free? Can you please tell me exact steps and what way i should follow these to make best ROI?

Also, i tried turning on CF ROCKET loader, but then discourse doesnt work as expected, and also i think if i turn on CF minification of css, js, html files, the emberJs scripts again dont work well and site doesn’t behave as expected. What else can i tweak in my CF account to speed site up without compromising performance and usability of site?

How can we enable GZIP compression and resources compression and minification on discourse and enable server side /client side caching?

Another bug we recently found is that when we goto https://howtodiscuss.com/logs it as admin, it redirects us to https://howtodiscuss.com/logs/logs/ is this a discourse bug in latest version?

We @ahmed_khan1 will soon install the https://github.com/discourse/discourse-prometheus plugin and share stats with you guys.

Also, looking at our current setups, if you have any more suggestions and optimisation advises, please do share. Thanks

pfaffman · March 4, 2021, 8:21pm

Why not just let the internal NGINX do the job?

Faizan_Zahid · March 4, 2021, 8:24pm

As i mentioned, we are serving AMP pages, we are doing the AMP routing in NGINX but i did not share the AMP code in the code snippet of NGINX.

Our AMP pages can be accessed on

etc we are serving these via PHP and using discourse API to build these pages on AMP. So thats why we need NGINX to handle the AMP pages routing for us, as discourse does not provide this AMP routing out of the box.

If anyone can help us achieve 100% score on Google PageSpeed & GTMetrix for our website, and grow bigger, please reach us on howtodiscuss.com@gmail.com

https://developers.google.com/speed/pagespeed/insights/?url=https%3A%2F%2Fhowtodiscuss.com%2Ft%2Fsarahs-discovery-of-weight-loss%2F28583&tab=mobile

gerhard · March 4, 2021, 8:41pm

If I were you I wouldn’t try to make a slow site faster. That’s kinda pointless. Start with a site that is fast enough and try to make it even faster. So, first step is finding out why it is so slow. I would do the following:

Create a new droplet (for testing purposes) and follow the Discourse setup instructions
Don’t add any plugins yet. And no Cloudflare, Nginx etc
Restore a backup of your production site
Is it fast enough? Good.
Add plugins
Still fast? Good. If not, try to find the plugin that’s slowing it down.
Add Nginx in front of it for all the PHP stuff
Still fast? Good. If not, try something else, like
- try finding out why Nginx makes it slow
- or running Nginx on a private port for the PHP stuff and trying to put HAProxy in TCP mode in front of Discourse and your Nginx
Start using a CDN for static assets or full site CDN – see links in my last post and read those topics carefully. Cloudflare needs special treatment (a template in app.yml etc.) and don’t use Rocket Loader or anything like that.