I’m desperately trying to work out how to create a duplicate of our production system to allow testing of upgrades etc before taking it live. I’m hoping there is an easily reproducible set of steps to “export” the production system and then “import” into a new clear server.
I’ve looked at a number of different threads but nothing seem to cover this.
Is there a set of steps to duplicate a production system onto another machine with all data/attachements/etc in place ?
Regards
Henti
Download a backup from production and restore on staging.
It’s pretty simple. Off the top of my head:
Create your sandbox, install discourse.
Decide if you want mail (you don’t really need it if you’re just testing how to do “stuff”)
Fudge app.yml so mail won’t go out.
Download a backup from live, restore.
Restart.
Check it works.
Clean up the .db (https://meta.discourse.org/t/disk-space-usage-by-postgres/61137/4)
Have at it.
EDIT: My sandbox is just for checking software and such, so I don’t have outgoing mail. In fact I deliberately mangled all the entries in app.yml so that it would never send mail (and I disabled all outgoing).
I think you may want to run a remap domain in there too. That way all of the links do not go out to Production.
Hi Andrew,
I originally thought that was the correct way, but I could not get past step 1. My installation of discourse fails and unfortunately, nobody seems to be able to assist with that.
https://meta.discourse.org/t/failure-when-trying-to-install-older-version-of-discourse/61722
I was hoping it’s the wrong process and there is a different way to do it.
Thank you for the information.
Regards
Henti
I think you could do it by changing git head, but I wasn’t smart enough for that. In my case I was close enough to the live site that the restore ran.
Have you seen this:
That is the process I tried.
I’ll continue trying to get that working today. Thanks again for all the assistance.
Resolved the problem.
I had the failed data in /var/discourse that I didn’t know I needed to clean up.
Just to close the loop. Steps I followed.
Get latest backup from /var/discourse/shared/standalone/backups/default
mkdir /srv/discourse
git clone https://github.com/discourse/discourse_docker.git /srv/discourse
cd /srv/discourse
# If you need a specific commit, do:
# git reset --hard git_commit_hash_goes_here
Copy the app.yml from your production server to new/dev instance. Update app.yml
new_hostname=$(hostname -f)
sudo sed -i "s/^ DISCOURSE_HOSTNAME:.*$/ DISCOURSE_HOSTNAME: $new_hostname/g" containers/app.yml
# If you need to duplicate a specific version of discourse, use the git commit hash, otherwise use the branch name. Below we are using stable branch
sudo sed -i 's/^ #version: .*$/ version: stable/g' containers/app.yml
sudo sed -i "s/^ DISCOURSE_DEVELOPER_EMAILS: .*$/ DISCOURSE_DEVELOPER_EMAILS: 'sysadmins@yourcompany.com'/g" containers/app.yml
Bootstrap, start the app container
sudo ./launcher bootstrap app
sudo ./launcher start app
Copy your backup to restore location
sudo mkdir /var/discourse/shared/standalone/backups/default -p
sudo cp /tmp/discourse-backup/lgtm* /var/discourse/shared/standalone/backups/default
Now we run commands in the container to prepare, and restore the backup, and fix the permissions.
backup=$(basename /var/discourse/shared/standalone/backups/default/*)
sudo /usr/bin/docker exec -i app discourse restore $backup
sudo /usr/bin/docker exec -i app chown discourse.www-data /var/www/discourse/public/backups/default/ -R
If you get an error about SiteSettings not allowing restore. (I think it’s older version of discourse that needs this)
sudo /usr/bin/docker exec -i app rails runner "SiteSetting.allow_restore = true"
I tried using the Advanced, manual method of manually creating and restoring Discourse backups method for my restore, but it failed with rake db:migrate creating an empty db. Might work for you, I did not test further.
Slight update.
Replacing the DISCOURSE_HOSTNAME with $(hostname) bit me in the arse … I didn’t realise it’s used for url generation, so make sure it’s the url external users will be seeing if you use proxy/load balancers.
H
I went about this in a different, although slightly uglier way - I cloned my entire droplet on DigitalOcean, then logged into the droplet and disabled all of the ssl/https settings in the app.yml. This at least gave me a clone of my community content to screw with settings on.
I have no elegant way to migrate changes from my test server to the live server except by manually repeating, so this is still total amateur hour, but it at least let’s me explore without breaking the live server.
Took your approach. Thanks. A quick question: I’ve backed up a production instance and restored it to a local Mac instance for dev, but resolving to localhost:3000 is incomplete. e.g. My logo sits at http://localhost:3000/uploads but Discourse is looking in http://localhost/uploads. I see talk of editing my app.yml, but don’t see one inside my discourse directory. Is there something I’m missing here?
Try running the below in a rails console (probably RAILS_ENV=development bundle exec rails c or bundle exec rails c)
SiteSetting.port = "3000"
It’s a hidden site setting if I remember correctly which is why it doesn’t show up in the settings panel.
That worked. Thanks.
I’ve set everything up with a local Rails install vs a local Docker container. Looking at so many examples of local Docker containers I’m wondering, did I make the wrong choice? Does it matter at this point, or will I eventually wish I’m in a Docker container to better mirror my DigitalOcean install?
Personally I’d go for the development environment but create a VM with a production style install (if you have the resources to do so).
That way you can test out your changes and work in a development environment. You can then use the production style container to test out your changes in a prod like environment (sometimes the local Rails install suppresses some errors or causes strange behaviour) and get the best of both worlds.
Thanks for that feedback. Makes sense.
Are all options for hosting a Docker container paid options, regardless of traffic? I’m used to using Heroku to stage Rails sites at no cost, but the Docker containers are new to me.
Looks as if Heroku supports Docker, but seems the Community recommends Digital Ocean, which I assume does not have a free tier.
Theoretically if you don’t mind having a private dev instance (i.e. it doesn’t allow access from the internet but can be accessed from the network your PC lives at) just run a VM using virtualbox or somthing similar with a bridged IP. Install ubuntu, set up remote root login and you should be OK to follow the production install guide.
The only thing you’d need to pay for is the internet connection that you probably already have and the power for the computer used. There are no server costs related to running your own Ubuntu Server in a VM that you control on your PC.
Now I’m wondering if we should just add a Droplet to our DigitalOcean account for a staging instance at $5 / month. Is the cleanest, simplest approach just a matter of spinning up another Droplet inside the same DigitalOcean account which can be accessed at staging.ourdomain.org? I see posts about running two instances inside a single Droplet. Is there any reason to do that other than cost?
This is a pretty important post, but could we also get some input and directions for processes that exclude a working backup system? I’m finding that the integrated backup doesn’t appear to have any way to function when you use an external database server. In my particular use case I’d like to take a production Discourse installation with a separated database server and then deploy it to a staging/test server where the database is INTEGRATED into the Dockerized setup.
I’ll post my findings once I get this working and what steps I took, but I figured this is a pretty important consideration for this topic.
I don’t think that your issue belongs here particularly when it is an unlikely scenario for almost every developer, .i.e. most developers will not need to duplicate any production environment beyond an officially supported install which is the gold-standard for reproducing bugs.
Also, you’re effectively spamming this forum with your problem because you will be duplicating the solution to your issue at Pg_dump backup failure for remote pgsql - port & version differentials; what options exist?