Running a separate data container


(Jeffrey C Witt) #1

Ok, I am working on learning how to run a separate data container, with the ultimate goal of being able to run multiple databases to support multi site. So I’m starting from the host directory var/docker/sample/data.yml and I wonder if I can asks some general questions to see if I’m on track.

First, my understanding is that I should copy this file into my /var/docker/containers folder, and then, when it is ready, I should run ./launcher rebuild data

If that’s correct I move on to understanding the file itself.

The first part of the file calls a number of templates:

templates:
  - "templates/cron.template.yml"
  - "templates/postgres.template.yml"
  - "templates/redis.template.yml"
  - "templates/sshd.template.yml"

I’m assuming templates/postgres.template.yml is going to set up the initial database. Am I correct so far?

Then jumping down to the hooks in the same data.yml file

hooks:
  after_postgres:
    - exec:
        stdin: |
          alter user discourse with password 'SOME_SECRET';
        cmd: sudo -u postgres psql discourse
        raise_on_fail: false

Here I’m assuming that after the postgres db has been created, we are going to change the password for user ‘discourse’ to ‘SOME_SECRET’ Does that sound correct?

If so, then I imagine I can add the further commands about creating a second database directly after this, so that the whole after_postgres looks like this:

hooks:
      after_postgres:
        - exec:
            stdin: |
              alter user discourse with password 'SOME_SECRET';
            cmd: sudo -u postgres psql discourse
            raise_on_fail: false
     - exec: sudo -u postgres createdb b_discourse || exit 0
     - exec:
          stdin: |
            grant all privileges on database b_discourse to discourse;
          cmd: sudo -u postgres psql b_discourse
          raise_on_fail: false

     - exec: /bin/bash -c 'sudo -u postgres psql b_discourse <<< "alter schema public owner to discourse;"'
     - exec: /bin/bash -c 'sudo -u postgres psql b_discourse <<< "create extension if not exists hstore;"'
     - exec: /bin/bash -c 'sudo -u postgres psql b_discourse <<< "create extension if not exists pg_trgm;"'

Is there anything else I need to do before running this to set up two databases “discourse” and “b_discourse”?

If my file is ready, should I run ./launcher start data or first ./launcher bootstrap data and then ./launcher start data (I think probably the latter).

Thanks for your help.
jw


How to move from standalone setup to separate containers?
Multisite configuration with Docker
(Kane York) #2

Yep, this is correct. You also need the samples/web_only.yml file (this is the one you might make multiple copies of).

You can also use ./launcher bootstrap <container name> to do just that part of the rebuild (rebuild is stop, destroy, bootstrap, start).

Yep, and you need to put that password in the web_only.yml file(s) too.

Set the password for the b_discourse as well.

And yes, there is one more thing. The web_only should end up looking like this:

env:
  DISCOURSE_DEVELOPER_EMAILS: '' # same as standalone
  DISCOURSE_DB_SOCKET: '' # this is for using /shared to do db access
  DISCOURSE_DB_PASSWORD: SOME_SECRET # db password
  DISCOURSE_DB_HOST: YOUR_IP_ADDRESS # <-- ip of data container (e.g 172.17.0.243)
  DISCOURSE_DB_NAME: b_discourse # <-- database name
  DISCOURSE_HOSTNAME: 'www.example.com' # same as standalone
  DISCOURSE_REDIS_HOST: YOUR_IP_ADDRESS  # <-- ip of data container

For the IP addresses, note that all Docker containers are connected together with a single network bridge (so they can talk to each other). You may also want to investigate how to get Docker to give static IPs to your containers.


(Kane York) #3

Another thing: You should probably edit the paths of the volumes section.

So, data.yml:

volumes:
  - volume:
        host: /var/docker/shared/data
        guest: /shared

web_a.yml:

volumes:
  - volume:
        host: /var/docker/shared/web_a
        guest: /shared

web_b.yml:

volumes:
  - volume:
        host: /var/docker/shared/web_b
        guest: /shared

(Jeffrey C Witt) #4

Just so I’m clear, DISCOURSE_DB_SOCKET should be left blank or as a blank string, like so… ''


(Brahn) #5

Any hints on where to start for that? The only other option I found was to use 172.17.42.1 but then the only way I could get it to work was to set an iptables INPUT policy for the docker0 adapter which doesn’t feel right.


(Jake Jackson) #6

Could we get some more information on running separate data and web containers?

There’s a few unanswered questions here, like:

  1. Should the DISCOURSE_DB_SOCKET be left blank, or do we need to enter a value?
  2. When configuring the web container how do we get the IP address of the data container?
  3. The documentation for this mentions we can bootstrap a new web process and switch in the new image when done, but it doesn’t make it explicitly clear how this is done.
  4. Is there any way to migrate from a single container to a split data/web container?

Any help would be much appreciated.


(Jason Nall) #7

I second question 4. I am currently planning one of these moves and I’m going to do a detailed blog post on the process, but appreciate any advice or precedent material that I can look at.


(Iolo) #8

I notice that you suggest separate folders for the web volumes but do you need to keep these in sync somehow?


(Iolo) #9

My plan involved butchering the advanced backup and restore tutorial at Advanced, manual method of manually creating and restoring Discourse backups

A better solution would be appreciated.


(Sigurður Guðbrandsson) #10

The actual data is outside of the containers in a “shared” folder (on the main server).

root@forum:/var/discourse# ls
README.md  bin  cids  containers  image  launcher  samples  scripts  shared  templates
root@forum:/var/discourse# ls shared/
standalone
root@forum:/var/discourse# ls shared/standalone/
backups  log  postgres_backup  postgres_data  postgres_run  redis_data  state  uploads  vendor_bundle

If you want to split the standalone container into data and web, you need to create two shared folders for the containers called data and web-only

root@forum:/var/discourse# mkdir shared/data && mkdir shared/web-only
root@forum:/var/discourse# ls shared
data  standalone  web-only

Now you move the database stuff to the data shared folder and the rest to web-only

root@forum:/var/discourse# mv shared/standalone/postgres_* shared/data/
root@forum:/var/discourse# mv shared/standalone/* shared/web-only/
root@forum:/var/discourse# ls shared/*
shared/data:
postgres_backup  postgres_data  postgres_run

shared/standalone:

shared/web-only:
backups  log  redis_data  state  uploads  vendor_bundle

Now copy the data.yml and web_only.yml from the samples folder to the containers folder, configure them and run the containers as mentioned above in the thread.

root@forum:/var/discourse# cp samples/data.yml containers/
root@forum:/var/discourse# cp samples/web_only.yml containers/

(Iolo) #11

What if I want 2 web_only containers? It’s mainly how to handle the uploads folder that I’m having trouble deciphering at this stage.


(Sigurður Guðbrandsson) #12

Let me ask you first - what are you trying to accomplish? (not the technical stuff - the end results)


(Iolo) #13

The ideal aim would be high availability for discourse but we would settle for a cold standby implementation with a recovery time of under an hour. Preventing user data loss is also more important than high availability.

What we are currently planning on having is 2 servers, both servers will have a standard postgres database (in sync via streaming replication), a redis container and a web_only container, all sitting behind a haproxy for SSL termination and load balancing/failover.

We may also need to be able to replicate the database out to other departments such as data analytics or reports generation. This is also our reasoning behind choosing the standard postgres installation vs a data container.

Edit: I should say, I have most of this done, I just don’t understand how web containers work with each other without issues


(Sander Datema) #14

I also don’t see where the double web_only containers come in…

Even if it’s about upgrading the one, while the other keeps running, they both use the same database. Meaning: after a migration the old web container might fail or screw up.