Discourse HA setup in network isolated env

Support Self-hosting

AI-generated summary

Veeru_Badavath is planning to set up Discourse with high availability in a production environment with 3 application servers, 2 Postgres DB servers, and HAProxy load balancer. The servers do not have internet connectivity, but they can pull DockerHub images, and they have a Nexus repo for artifact storage.

Falco suggested that Veeru_Badavath needs an external Redis server and a load balancer. Veeru_Badavath confirmed that they are using HAProxy and Nexus repo.

Veeru_Badavath had issues running the previously bootstrapped container image in the production server due to Unicorn server issues. Falco asked for clarification on the steps Veeru_Badavath took, and Veeru_Badavath explained that they pushed the image to the Nexus repo and pulled it in the production server.

Veeru_Badavath shared the error log when starting the image in production, and Falco asked for the log. Veeru_Badavath provided the log, and Falco suggested that the issue might be due to the lack of access to external PostgreSQL, Redis, and Object Storage.

Veeru_Badavath explained that they were testing a standalone setup, and Falco stated that this approach would not work. Veeru_Badavath asked about object storage, and Falco explained that object storage is necessary for handling multiple app servers and user uploads.

Veeru_Badavath confirmed that they are using external Postgres server and Redis cluster, and Falco suggested using shared dir/storage for the Discourse build dir.

Finally, Veeru_Badavath asked about the impact of the rebuild step on existing data in the Postgres DB, and Falco explained that there are migrations that will change both data and structure, and provided guidance on handling this in an HA environment.

1

Thanks in advance.
I’m planning setup discourse with high availability in production environment. The below are my design plan and some environment condition.

  1. Setup with 3 application servers and 2 Postgres DB servers. Always one PG server on write mode and another in read only mode.

  2. These 3 application servers will points to same DB server.

  3. DB each instances should server read and write only operations respectively.

  4. Production servers doesn’t have internet connectivity but I can pull dockerHub images.

  5. We have our own GitLab server.

  6. Is it possible to bootstrap docker image and that image on multiple servers?

Please anyone help for doing this setup. If any links or suggestions please PM me.

2

After running ./launcher bootstrap app somewhere you will need to save that resulting container image (usually done by pushing it to a registry) and then download and run it on your three app servers.

You will also need a central Redis server (and potentially replicas for it). You are also missing a load balancer to direct requests for those different application servers.

3

Thanks for the reply @Falco

We are using HAProxy load-balancer and Nexus repo for artifacts storing.

4

Hi @Falco
In my production env, I dont have internet access, so what I am planning is performing bootstrap on Internet accessible machine taking that bootstrap image to production servers. While doing this on production VM container is not coming up because Unicorn server looking some parent process ID so it is not running.

Please help me here, whether do I need copy bootstrap performed /var/discourse dir to production server?

5

What is “this” ? The bootstrap or the running the previously bootstrapped container image?

6

When I’m running the previously bootstrapped container image

7

How did you save and exported the image to the production server?

How, exactly, are you trying to run this saved image in production?

8

previously bootstrapped image pushing to Nexus repo and from Nexus repo pulling in production server.

docker run command generating from ./launcher start-cmd app and running same command in production server.

9

Can you share the error log when starting it on production?

10 
run-parts: executing /etc/runit/1.d/00-ensure-links
run-parts: executing /etc/runit/1.d/00-fix-var-logs
run-parts: executing /etc/runit/1.d/01-cleanup-web-pids
run-parts: executing /etc/runit/1.d/anacron
run-parts: executing /etc/runit/1.d/cleanup-pids
Cleaning stale PID files
run-parts: executing /etc/runit/1.d/copy-env
Started runsvdir, PID is 42
chgrp: invalid group: ‘syslog’
ok: run: redis: (pid 51) 0s
ok: run: postgres: (pid 56) 0s
supervisor pid: 53 unicorn pid: 75
config/unicorn_launcher: line 71: kill: (75) - No such process
config/unicorn_launcher: line 15: kill: (75) - No such process
(53) exiting
ok: run: redis: (pid 51) 7s
ok: run: postgres: (pid 101) 0s
supervisor pid: 96 unicorn pid: 103
config/unicorn_launcher: line 71: kill: (103) - No such process
config/unicorn_launcher: line 15: kill: (103) - No such process
(96) exiting
ok: run: redis: (pid 51) 14s
ok: run: postgres: (pid 127) 0s
supervisor pid: 120 unicorn pid: 129
config/unicorn_launcher: line 71: kill: (129) - No such process
config/unicorn_launcher: line 15: kill: (129) - No such process
(120) exiting
ok: run: redis: (pid 51) 22s
timeout: down: postgres: 0s, normally up, want up
ok: run: redis: (pid 51) 30s
timeout: down: postgres: 1s, normally up, want up
ok: run: redis: (pid 51) 37s
ok: run: postgres: (pid 174) 0s
supervisor pid: 165 unicorn pid: 176
config/unicorn_launcher: line 71: kill: (176) - No such process
config/unicorn_launcher: line 15: kill: (176) - No such process
(165) exiting
ok: run: redis: (pid 51) 48s
ok: run: postgres: (pid 196) 1s
supervisor pid: 191 unicorn pid: 198
config/unicorn_launcher: line 71: kill: (198) - No such process
config/unicorn_launcher: line 15: kill: (198) - No such process
(191) exiting
ok: run: redis: (pid 51) 54s
timeout: down: postgres: 1s, normally up, want up
11

Are you using an external PostgreSQL, Redis and Object Storage? That is expected when doing HA, and both your production servers and the build servers need to have access to those external services.

12

Just testing the scenario, bootstrap performing on one server and running bootstrapped container image on another server with standalone setup.

13

That’s a sure way that will not work.

14

Any reason for mentioning object storage, can’t we use server internal storage?

15

In HA setup do we need to copy bootstrap performed discourse directory in all app server?

16

How are you going to handle multiple app servers and user uploads? A shared network drive between all servers? That may work, but our official solution for that is Object Storage using the S3 API.

No need for that, as long as you

19

Yes, I’m using external Postgres server and redis cluster

3 app servers using for Redis cluster along with application.

So I need to use shared dir/storage for discourse build dir. Thank you @Falco now it is clear for me.

20

sorry for troubling you @Falco, I have one last query.

When performing rebuild step, will it impact existing Data in Postgres DB? if yes then how to handle it?

21

Yes, there are migrations that will be ran during a rebuild, that will change both data and structure.

In an HA environment you will need to follow the guidance we have here: Introducing Post Deployment Migration