High-CPU / High memory on fresh discourse install

Installation
1

I have been attempting to install the official version of discourse on my docker host. I have followed the instructions for setting up a standalone server, with the modification to run behind a proxy in order to share the ports with other sites I have (non-discourse). However, after installing and building, the resulting site is taking over the resources of the box, and nearly freezes all other processes, rendering the box inoperable, until I manage to forcefully kill the container.

This is the app.yml I am currently using:

## this is the all-in-one, standalone Discourse Docker container template
##
## After making changes to this file, you MUST rebuild
## /var/discourse/launcher rebuild app
##
## BE *VERY* CAREFUL WHEN EDITING!
## YAML FILES ARE SUPER SUPER SENSITIVE TO MISTAKES IN WHITESPACE OR ALIGNMENT!
## visit http://www.yamllint.com/ to validate this file as needed

templates:
  - "templates/postgres.template.yml"
  - "templates/redis.template.yml"
  - "templates/web.template.yml"
  - "templates/web.ratelimited.template.yml"
## Uncomment these two lines if you wish to add Lets Encrypt (https)
  #- "templates/web.ssl.template.yml"
  #- "templates/web.letsencrypt.ssl.template.yml"

## which TCP/IP ports should this container expose?
## If you want Discourse to share a port with another webserver like Apache or nginx,
## see https://meta.discourse.org/t/17247 for details
expose:
  - "80"

params:
  db_default_text_search_config: "pg_catalog.english"

  ## Set db_shared_buffers to a max of 25% of the total memory.
  ## will be set automatically by bootstrap based on detected RAM, or you can override
  db_shared_buffers: "1024MB"

  ## can improve sorting performance, but adds memory usage per-connection
  #db_work_mem: "40MB"

  ## Which Git revision should this container use? (default: tests-passed)
  #version: tests-passed

env:
  LANG: en_US.UTF-8
  # DISCOURSE_DEFAULT_LOCALE: en

  ## How many concurrent web requests are supported? Depends on memory and CPU cores.
  ## will be set automatically by bootstrap based on detected CPUs, or you can override
  UNICORN_WORKERS: 8

  ## TODO: The domain name this Discourse instance will respond to
  ## Required. Discourse will not work with a bare IP number.
  DISCOURSE_HOSTNAME: discourse.belmankraul.com
  VIRTUAL_HOST: 'discourse.belmankraul.com'

  ## Uncomment if you want the container to be started with the same
  ## hostname (-h option) as specified above (default "$hostname-$config")
  #DOCKER_USE_HOSTNAME: true

  ## TODO: List of comma delimited emails that will be made admin and developer
  ## on initial signup example 'user1@example.com,user2@example.com'
  DISCOURSE_DEVELOPER_EMAILS: '[my developer email]'

  ## TODO: The SMTP mail server used to validate new accounts and send notifications
  # SMTP ADDRESS, username, and password are required
  # WARNING the char '#' in SMTP password can cause problems!
  DISCOURSE_SMTP_ADDRESS: [my smtp server]
  DISCOURSE_SMTP_PORT: 587
  DISCOURSE_SMTP_USER_NAME: [my smtp username]
  DISCOURSE_SMTP_PASSWORD: "******"
  #DISCOURSE_SMTP_ENABLE_START_TLS: true           # (optional, default true)

  ## If you added the Lets Encrypt template, uncomment below to get a free SSL certificate
  #LETSENCRYPT_ACCOUNT_EMAIL: me@example.com

  ## The CDN address for this Discourse instance (configured to pull)
  ## see https://meta.discourse.org/t/14857 for details
  #DISCOURSE_CDN_URL: //discourse-cdn.example.com

## The Docker container is stateless; all data is stored in /shared
volumes:
  - volume:
      host: /root/docker-persist/discourse/shared/standalone
      guest: /shared
  - volume:
      host: /root/docker-persist/discourse/shared/standalone/log/var-log
      guest: /var/log

## Plugins go here
## see https://meta.discourse.org/t/19157 for details
hooks:
  after_code:
    - exec:
        cd: $home/plugins
        cmd:
          - git clone https://github.com/discourse/docker_manager.git

## Any custom commands to run after building
run:
  - exec: echo "Beginning of custom commands"
  ## If you want to set the 'From' email address for your first registration, uncomment and change:
  ## After getting the first signup email, re-comment the line. It only needs to run once.
  #- exec: rails r "SiteSetting.notification_email='info@unconfigured.discourse.org'"
  - exec: echo "End of custom commands"

docker_args: "--network reverse-proxy"

It was copied straight off of standalone.yml in samples, with the modifications for it to work under a reverse proxy (using jwilder/nginx-proxy). In order to achieve this, I have a separate network, called reverse-proxy. All of that works perfectly after the site manages to build (the 6-8mins). The setup wizard comes up, I am able to enter all of my data. But when I finish it, the site just locks up, processor and memory usage spike, and the whole host box slows to a crawl. I have to then kill the container, in order to bring back stability to the box. My initial setup was a 4VCPU, 4GB of RAM, 60GB. I have now increased RAM to 8GB, to no avail. Any idea what is going on?

I must confess I am one of the ones who would very much rather prefer a standard docker / docker-compose implementation. I have seen the arguments for and against. I am currently having to resort to the bitnami implementation of discourse, which supports docker-compose nicely, but is behind in updates from master, until I can overcome the official implementation problem.

Thanks in advance for the help.

2

I have a few questions!

  1. Why do you need a reverse proxy network when there are more efficient ways of doing it?
  2. I’ve never heard of the variable VIRTUAL_HOST in regards to discourse

With that, I’d say that there is something wrong with your configuration, Please try installing discourse on a separate machine if possible . (of course, on the same provider) to check if it is discourse or something else on your existing server that is causing the resource spike.

3

You said this server hosts other sites. Figure out your baseline load on CPU and memory. Subtract used from available to gauge sane settings on your unicorns, db_worker & shared_buffers.

You might want to first try dropping your unicorns down to 3 and the shared buffers to something like 384MB. If that does it you might just have a fairly well loaded server (or noisy neighbors).

3 Likes
5

Bhanu:

  1. I am using a reverse proxy network because it is recommended as optimal for using the proxy container that I run all of my solutions with.

  2. As mentioned in my original post, I am using jwilder/reverse proxy, which uses the VIRTUAL_HOST env variable to proxy the web traffic based on host name. This has been extensively talked about here at meta, and it works quite well. I have no issues with that part. More about that here:

https://meta.discourse.org/t/using-letsencrypt-ssl-in-multi-site-setup/50235/10

One of the biggest questions I have about having to run Discourse on a separate machine is, why docker, then? The other thing… The minimal resource requirement is 1GB. I can’t for the life of me understand why 4GB or even 8GB would not be enough, for operation in a shared container environment. This is an on-premises ESXi vm setup, sure, I could create a separate virtual machine for this… But the question remains…why docker, then? I am assuming that by “my configuration” you mean my docker version, host OS etc (since I posted my app.yml config already). It is as follows:

Ubuntu 18.04.1 LTS x64

Docker-ce 18.06.1-ce

6

You just set the unicorn workers and shared_buffers to high for a shared machine.

Try setting to the values @boredomdenied suggested and rebuilding.

4 Likes
7

Rafael, you and @boredomdenied were correct. After setting my worker and shared_buffers values to what was recommended, everything is working perfectly. Thank you all for your help. I am marking @boredom’s reply as the solution.

4 Likes