Scaling Discourse involves breaking down services into separate Docker containers and placing them behind a load balancer. tkiblin asked how to set up a PostgreSQL (PGSQL) cluster with web front-ends talking to the cluster, storing data on a shared NFS, and using HAProxy for load balancing. codinghorror pointed to the advanced install instructions, which cover this process.
rubydoob outlined the components involved in scaling Discourse: multiple servers for Nginx/Rails, scaling Redis through replication or partitioning, creating multiple Sidekiq instances, and setting up a cluster of PostgreSQL servers.
Folks, I’ve seen the question a few times, but really haven’t seen any good answers…
How does one scale and make Discourse highly available?
Like, how can I setup a PGSQL cluster, and have a bunch of web “front-ends” talking to the cluster, and storing data on a shared NFS for the web front-ends? Using haproxy in front for the load balancers, docker containers for the web side, and a PG cluster on the backend?
Also, does Discourse have the ability to use read nodes and write nodes on the db side like Wordpress, Xenfo, and some others do?
This is covered in the advanced install instructions. Basically a lot more Docker containers, broken down per service (web/sql/redis), and in front of a load balancer.
You have rails on the front-end, redis, sidekiq, and postgres.
nginx/rails can be hosted on multiple servers.
With redis you would have to scale the server out, or create multiple instances with replication. This would require some archectual changes to discourse but you could also partition the redis ‘schema’ so you are spreading the load between different redis instances based on the task etc.
sidekiq is creating multiple instances, and round-robin between them or partitioning the usage like #2 (reference: Redirecting to Google Groups)