How to backup and restore a whole /var/discourse app folder?

[adrelanos]

How to backup and restore a whole /var/discourse folder?

Due to issues with the usual backup and restoration process, I was wondering if I could backup the whole /var/discourse folder and re-use it on another server. This is what I did…

On production server:

rsync_opts="\
   --recursive \
   --links \
   --hard-links \
   --safe-links \
   --owner \
   --group \
   --perms \
   --times \
   --delete \
   --sparse \
   --compress \
   --partial \
   --rsh=ssh
"

dir=/var/discourse

rsync  $rsync_opts "$dir/" root@xx.xx.xx.xx:"/var/production-backup/$dir"

On stage server:

Install docker.

rsync --recursive --links --hard-links --safe-links --owner --group --perms --times --delete --sparse --compress --partial /var/production-backup/var/discourse/ /var/discourse

But getting 502 Bad Gateway.

Trying to investigate.

cd /var/discourse

./launcher start app

root@whonix-app:/var/www/discourse# service postgresql status
12/main (port 5432): down

root@whonix-app:/var/www/discourse# service postgresql start
[....] Starting PostgreSQL 12 database server: main[....] Error: The cluster is owned [FAILoup id 116 which does not exist ... failed!
 failed!

Guess some fixes:

chown -R postgres.postgres /etc/postgresql
chown -R postgres.postgres /shared/postgres_*
chown -R postgres.postgres /var/lib/postgresql
chown -R postgres.postgres /var/log/postgresql
chown -R redis.redis /etc/redis/redis.conf
chown -R redis.redis /shared/redis_data
chown -R redis.redis /var/run/redis
chown -R redis.redis /var/lib/redis
chown -R redis.redis /var/log/redis
chgrp -R ssl-cert /etc/ssl/private

But did not help.

root@whonix-app:/var/www/discourse# service postgresql start
[....] Starting PostgreSQL 12 database server: main[....] Error: /usr/lib/postgresql/12/bin/pg_ctl /usr/lib/postgresql/12/bin/pg_ctl start -D /shared/postgres_data -l /var/log/postgresql/postgresql-12-main.log -s -o -c config_file="/etc/postgresql/12/main/postgresql.conf" exited with status 1: 2020-05-25 10:20:10.501 UTC [603] FATAL: database files are incompatible with server 2020-05-25 10:20:10.501 UTC [603] DETAIL: The data directory was initialized by PostgreSQL version 10, which is not compatible with this version 12.2 (Debian 12.2-2.pgdg100+1). pg_ctl: could not start server Examine the lo[FAILput. ... failed!
 failed!

Why are these file permission issues introduced?

Why is postgres updated from version 10 to version 12 by simply copying the whole folder from one server to another? I must be doing something wrong.

Could you please kindly share instructions on how to backup the whole discourse app on one server and move it to another server?

[RGJ (Richard - Communiteq)]

@adrelanos

the whole phabricator app

Discourse does not use phabricator ?

[adrelanos]

Typo. I meant discourse. Typo fixed. Original question remains.

[gerhard (Gerhard Schlager)]

Move your Discourse Instance to a Different Server

In this guide we’ll move an existing Discourse instance on Linode (source) to a new Discourse instance on Digital Ocean (destination), but of course these steps will work on any other cloud providers that support Docker. If you are using the same domain name and moving servers, you should set the TTL (time to live) on your DNS entry to a very low value in advance of the change. If you are using different domain names, this won’t matter. Log In and Update Only admins can perform backu…

@techAPJ

What you want to avoid is restoring a newer backup on an older version (the same goes for the Postgres version).

[adrelanos]

That doesn’t move the whole /var/discourse folder. I am aware of these instructions these don’t work for me. Therefore I wanted a more “complete” 1to1 “hardcopy” way to backup.

[pfaffman (Jay Pfaffman)]

You can shut down the container and copy the whole thing over to a new server, excluding the tmp, backup, and cache directories (and I think another one? ). That should work. I have some this recently, I think for a similar reason.

But you still really need to get the broken index problem solved.

[adrelanos]

I think docker version is introducing differences. (Which then lead to failure.)

original server
docker-engine 17.05.0~ce-0~debian-stretch

vs newer (stage) server
docker-engine 17.05.0~ce-0~debian-stretch

That results in original server having postgres version 10 while newer (stage) server has postgres 12 already.

@pfaffman

excluding the tmp, backup, and cache directories (and I think another one? ).

Is that mandatory? Any easier way? Why would it be required to not copy all as is and restore?

@pfaffman

copy the whole thing

Leading to permission issues which I cannot explain. Should be possible to copy without breaking permissions. And I am also not sure I fully fixed all permission issues.

@pfaffman

But you still really need to get the broken index problem solved.

Yes but I think it would be much safer to only proceed with that once I can at least reproduce whatever is still working now.

[neounix (Dark Matter)]

@adrelanos

That doesn’t move the whole /var/discourse folder. I am aware of these instructions these don’t work for me. Therefore I wanted a more “complete” 1to1 “hardcopy” way to backup.

You cannot simply “tar up” the /var/discourse directory and move it to another machine, untar it, and start the discourse app.

One of the main reasons is that when you build / bootstrap Discourse, it (launcher as I recall off the top of my head) will check if a base Discourse container (image) exists and pull down the base discourse Docker image (if not) and start up that base docker image into a container.

After that base git pull, the build process will build another docker image (the app)

Both of these docker images (the base image and the app image) do not exist inside /var/discourse, so tarring up /var/discourse is only a partial “solution” (using this term loosely).

These Discourse docker images are built as docker images and as a part of Docker; they don’t “live” in /var/discourse but are built there and then moved to docker as docker images.

It might be possible to edit your container yml file and rebuild from scratch, but the more conventional way is to just save your:

• container yml file(s)
• your full backup with uploads

and then edit your container yammy, clone the discourse-docker repo and rebuild.

Then, restore your full backup, including uploads (from the command line in the container).

Using github as the repo is a cleaner solution that the older unix'y way of “tarring the whole darn thing up” and “moving the whole darn thing” over to another server. However, even with the “older unix way” this method often does not provide a complete solution because there are often shared libs in system, shared lib user directories, and more which are not a part of the distribution directory, and there are etc files which are not in the distro root directory, etc.

So, even on most modern linux systems, we use apt (on Ubuntu, for example) to pull the repo. In the case of Discourse docker, you are pulling (and building) discourse-docker to set up the base container and another discourse repo to build the app. So, /var/discourse is a “place to build” (the images) and a “place to share” (the data, the backups, the public static files,etc).

Hope this summary was useful in some small way.

[pfaffman (Jay Pfaffman)]

Sure, your can copy it all with rsync -rav.

You might have more luck if you change you app to use the postgres 10 template. But it sounds like the easiest thing might be to just fix your database in place.

[Stephen]

You can move the folder, it works fine. It’s just not the preferred route as it sidesteps discourse-setup and any of the tuning/tests it runs along the way.

[adrelanos]

@Stephen

You can move the folder, it works fine. It’s just not the preferred route as it sidesteps discourse-setup and any of the tuning/tests it runs along the way.

It didn’t work in my case as the upgraded docker resulted in a newer postgres version inside the docker container which then resulted in an unusable forum due to postgres migration issues. Had to change from postres to postgres 10 template.

How to backup and restore a whole /var/discourse app folder? explains the details well.

I guess I would have to backup and restore the /var/docker folder as well. But even that has a chance to fail because of this:

@neounix

owever, even with the “older unix way” this method often does not provide a complete solution because there are often shared libs in system, shared lib user directories, and more which are not a part of the distribution directory, and there are etc files which are not in the distro root directory, etc.

[RGJ (Richard - Communiteq)]

You’re going down a rabbit hole.
If I were you I would focus on solving your original problem with backup/restore.

[neounix (Dark Matter)]

@RGJ

You’re going down a rabbit hole.

Maybe even a hole.

@RGJ

If I were you I would focus on solving your original problem with backup/restore.

Agreed… for sure…

@adrelanos

There are no “issues” with the backup-restore process. See this “praise” yours truly @neounix penned a few months back on this topic:

The Brilliant Discourse Restore Process

In the last month, we have performed countless migration tests in various scenarios and docker containers. In addition, I have performed even more Discourse DB restores from one server to another. Based on extensive recovery testing, the Discourse restore process is brilliant and has worked flawlessly in every situation from the command line. For example, we use a very primitive script for this, which I store in my /shared/neo/bin directory: #!/bin/bash echo "cd /var/www/discourse" cd /var/…

[neounix (Dark Matter)]

@adrelanos

How to backup and restore a whole /var/discourse folder?

Dear @adrelanos,

Back to your original question in post #1 above, being curious by nature, I was not really satisfied with my reply earlier and did some testing today.

In a nutshell, I just confirmed that we can use docker save (for the base and the app containers) and tar for the /var/discourse directory, and completely save, transfer (backup), and restore the app this way.

I’m nearly certain (99.99%) that this method is not officially supported but you deserve an answer, so I tested this for you:

Basically, here are the steps, in summary:

1. Save your containers with docker save.

For example, if you are running a standalone app, you can save the base and the app container, like this (based on your configuration):

docker save -o /tmp/my.discourse.docker.app.tar  discourse/base:2.0.20200512-1735

and also:

docker save -o /tmp/my.discourse.docker.app.tar local_discourse/app:latest  

2. You can also tar up the /var/discourse directory , as you mentioned:

cd /var/
tar -cvzf /tmp/my.var.discourse.tar.gz discourse

and then gzip your docker tar files if you wish and archive them:

gzip /tmp/my.discourse.docker*.tar

3. … and you can move these files to another server, archive them on the same server, whatever you wish, reverse the steps, and start the Discourse app without an issue.

I just confirmed this by “doing it”, and deleting all container images and the /var/discourse directory. Basically, I wiped out everything and restarted (no need to rebuild since the domain did not change, etc) from the backups.

For example, to restore, you can take the saved docker images above and load the images, for example:

gzip -d /tmp/my.discourse.docker.app.tar.gz
docker load -i /tmp/my.discourse.docker.app.tar

gzip -d /tmp/my.discourse.docker.base.tar.gz
docker load -i /tmp/my.discourse.docker.base.tar

4. Then, untar your original /var/discourse directory

cd /var
tar -xvzf /tmp/my.var.discourse.tar.gz

5. Next, you need to look at your images to make sure they are properly labeled:

docker images

6. and if the images are not properly labeled, make sure you tag them correctly, for example for your app image:

docker tag 58ffc74989af local_discourse/app:latest

7. Then, you just do this:

cd /var/discourse
./launcher start app

and it works just fine. I just tested it (twice).

Hope this helps.

FWIW: I tried this method two different ways, doing the backup method above, wiping out all docker containers, images, and the /var/discourse directory (totally destroying all, each time).

In each case, I was able to load my saved docker images, untar the /var/discourse directory, run ./launcher start app and Discourse started up flawlessly and to prove it, I could do a normal backup from the UI, proving all was good.

Not sure if this answers your question (and have not been participating in the Postgres 10 to 12 upgrade or discussions); but regarding your question of just tarring up the app as a backup, and restoring, the answer is yes but you must use not only archive the /var/discourse directory, but you must also docker save your images.

The main “gotcha” is to keep your image repo name and tags correct, and you should be good to go.

Hope this answers your question, in some small way, on:

How to backup and restore a whole /var/discourse app folder?

The answer is that you must archive both your folder and your docker images (like in the example above) using docker save for the images (to backup), and docker load to restore.

Keep in mind that this method is not officially supported; but out of curiosity, I wanted to see how to do it from the sys admin perspective, and found out it was easier than my replies earlier indicated.

Note1:

You might want to move all the backups from your backups/default directory (out of the directory tree) before you tar everything up in /var/discourse/ and keep those backups (seperately) since those files are so huge…

Note2:

This type of backup is not supported and so it is not recommended, for most Discourse sys admins. I recommend users follow the recommended (and officially supported) Discourse backup and recovery method.

Stay Curious!

Take care.

---

For more details including screen shots, kindly see my full post here:

How to Do a Full Backup of a Discourse App with tar and docker save

[adrelanos]

This is an excellent approach! Thank you!

One issue on restoration server.

./launcher logs app

2020-06-18 13:33:56.434 UTC [127] FATAL: data directory “/shared/postgres_data” has wrong ownership
2020-06-18 13:33:56.434 UTC [127] HINT: The server must be started by the user that owns the data directory.
./run: 3: echo: echo: I/O error
2020-06-18 13:33:57.448 GMT [128] LOG: skipping missing configuration file “/shared/postgres_data/postgresql.auto.conf”

---

That could be due to some missing tar options? Added -p and -s during extraction but didn’t help.

original server (outside docker):

ls -la /var/discourse/shared/standalone/postgres_data/

drwx------ 7 messagebus messagebus 4096 May 25 13:16 base

original server (inside docker (./launcher enter app)):

ls -la /var/lib/postgresql/10/main/

drwx------ 5 root postgres 4096 May 25 23:28 base

---

restoration server outside docker:

ls -la /var/discourse/shared/standalone/postgres_data/

drwx------ 7 messagebus messagebus 71 May 25 11:16 base

restoration server inside docker:

drwx------ 5 root postgres 41 May 25 23:28 base

---

./launcher rebuild app would fix it but that’s besides the point.

Any idea?

[lucasbasquerotto]

@neounix

docker save -o /tmp/my.discourse.docker.app.tar discourse/base:2.0.20200512-1735

I think you meant docker save -o /tmp/my.discourse.docker.base.tar discourse/base:2.0.20200512-1735, based on your restore process. In any case, good explanation!

But, as you said, I don’t think this is an officially supported approach (but I also don’t think there’s anything else that could cause errors, unless the discourse team start to use more than 1 base image in the rebuild process).

[lucasbasquerotto]

@adrelanos

FATAL: data directory “/shared/postgres_data” has wrong ownership

It seems there is the same issue at:

https://meta.discourse.org/t/postgresql-12-update/151236/298?u=lucasbasquerotto

There’s not an answer to this specific issue tough, in the FAQ, but maybe the discourse team add a solution to that, considering that more than 1 person faced it. There is a FAQ about The source cluster was not shut down cleanly, that could be related to your issue.

[D8vid]

A method I used that does not involve a docker save or a wholesale tar+untar of /var/discourse:

Self-hosting migration example — VPS to VPS

I’m sharing my recent experience of migrating a self-hosted Discourse installation from one VPS to another, as a way of saying ‘thanks!’ to the creators of Discourse and its community of administrators. This migration preserves LetsEncrypt SSL & email configurations. The original installation was on a Google Cloud Platform (GCP) g1-small instance type (1 vCPU, 1.7 GB RAM), using the official guide (INSTALL-cloud.md) & tracking the tests-passed branch. The new installation is on an Evolution Ho…