نقل موقع Discourse إلى VPS آخر باستخدام rsync

[SystemZ (Michał Frąckiewicz)]

This method is different than restoring file in UI.
Maybe there is a more downtime but it’s simpler for linux folks without Discourse knowledge and it’s easy to automate.

Prepare new VPS

First, prepare our new Current Ubuntu LTS VPS with empty Discourse dir, OS upgrade and Docker install

sudo mkdir -p /var/discourse
sudo apt-get update
sudo apt-get dist-upgrade
wget -qO- https://get.docker.com/ | sh

Then, make sure that we are on latest installed kernel and other libraries by just restarting

sudo reboot

Copy files to new VPS

Let’s copy all Discourse files, this first sync pass will reduce our downtime later.

On our new VPS let’s run

rsync -rvz example.com:/var/discourse /var

where example.com is our old VPS with forum files.
This may take a while so grab some favourite drink.

Turn off current forum and sync again

After first sync pass, turn off forum on old VPS via

cd /var/discourse/
./launcher stop app

Now when forum is stopped, let’s run sync on new VPS again

rsync -rvz --delete example.com:/var/discourse /var

--delete makes sure that will be 1:1 copy

Start forum on new host

After final sync we can finally run Discourse.

Rebuild and start container.
This may take a few minutes.

./launcher rebuild app

Done!
All settings, posts, threads, backups were copied to new location, no UI interaction was needed.

[mrjn (Manish R Jain)]

@SystemZ

rsync -rvz example.com:/var/discourse /var/discourse

This should really be:
rsync -rvz example:com:/var/discourse /var

Otherwise, you’ll end up with a discourse directory under /var/discourse.

Update: After doing this, all images in Assets for Site Design are now broken. Any ideas, what’s going on? @codinghorror Here’s our site: https://discuss.dgraph.io/

[SystemZ (Michał Frąckiewicz)]

@mrjn

This should really be:rsync -rvz example:com:/var/discourse /var

Otherwise, you’ll end up with a discourse directory under /var/discourse.

True, my mistake, forgot to add slashes,
Edited and fixed

@mrjn

Update: After doing this, all images in Assets for Site Design are now broken. Any ideas, what’s going on? @codinghorror Here’s our site: https://discuss.dgraph.io/

Did you modified links config somehow?

[mrjn (Manish R Jain)]

No. Didn’t modify any configs. All the previous assets don’t load. But any new ones that we add work just fine. The files are definitely there.

[SystemZ (Michał Frąckiewicz)]

Check permissions and owner of assets.

[mrjn (Manish R Jain)]

@SystemZ

Check permissions and owner of assets.

So, the permissions are okay, owner ubuntu and group www-data. The real problem I just identified was that I think I must have changed the settings for the static resources to upload to Amazon S3. So, all the new resources are going there, and that’s where discourse is probably reading them from.

So, it’s no longer looking at the stored data in /var/discourse/shared/standalone/uploads.

Btw, so 2 things:

• Is using S3 for static resources not recommended?. I understand it’s not deprecated, but checking to ensure we’re following the recommended setup for discourse.
• If it is recommended, what’re my options? I suppose I could just upload all the existing assets on disk to S3. Then, we should be able to retrieve them just like before. Right?

[SystemZ (Michał Frąckiewicz)]

@mrjn

So, the permissions are okay, owner ubuntu and group www-data. The real problem I just identified was that I think I must have changed the settings for the static resources to upload to Amazon S3. So, all the new resources are going there, and that’s where discourse is probably reading them from.

So, it’s no longer looking at the stored data in /var/discourse/shared/standalone/uploads.

You have an answer then

@mrjn

Btw, so 2 things:

Is using S3 for static resources not recommended?. I understand it’s not deprecated, but checking to ensure we’re following the recommended setup for discourse.
If it is recommended, what’re my options? I suppose I could just upload all the existing assets on disk to S3. Then, we should be able to retrieve them just like before. Right?

I think this questions belong to other topics.

[schungx (Stephen Chung)]

OK, not sure if the instructions here are out-dated or anything.

I have a forum that I mucked up the O/S. So, like a good cloud citizen, I spun up a brand new shiny Ubuntu 17.10 VM and followed the above instructions.

First thing that I found out immediately is that I need to be root to rsync the files. The rsync command came back with a pile of permission denied errors without copying anything.

Turns out that /var/discourse is owned by root (thanks to sudo mkdir -p /var/discourse in the instructions). Nothing a little hack won’t fix, so I did sudo chown 777 /var/discourse and did the rsync again.

This time it copied most of the files (but unfortunately all owned by me instead of root… probably needs a chown somewhere down the line).

The PostgreSQL files won’t copy and came back as permission denied. I checked the directories on the remote machine and lo and behold those directories are accessible only by root, not by mere mortals.

Using sudo on the rsync didn’t work either because it then asked for the password for root on the remote machine, which is also Ubuntu and thus root doesn’t have a password.

At the end, I gave up and installed a brand new Discourse installation then restored from backup.

I think the above instructions need serious rework especially wrt file ownerships and permissions.

[notriddle (Michael Howell)]

To perform an rsync as root on the sender and as ubuntu on the receiver, put ubuntu@ I’m front of the host, like this:

rsync -rvz ubuntu@example.com:/var/discourse /var

[schungx (Stephen Chung)]

Will that copy the entire tree under the same owner as well as the same permissions?

I suppose rsync preserves permissions, but will it work if all the files are copied under root? I can see that the PostgreSQL data files are owned by some other user etc. And obviously the uploads and backups directories are owned by the admin user, not root.

[notriddle (Michael Howell)]

It won’t be able to preserve the permissions in this case, because ubuntu can’t just change the files it be owned by root.

[schungx (Stephen Chung)]

In this case I wonder whether, after copying the directory tree, it will just work?

Next time I muck up something and have to move to a fresh VM, I’ll try it out!

[JagWaugh (Andrew Waugh)]

I’m not exactly an rsync guru, but why not copy from as root and to as root, and use the -p switch to preserve permissions?

[schungx (Stephen Chung)]

Apparently, rsync with -g and -o flags, while running as root, can preserve owners and groups. -p with rsync preserves permissions.

So technically speaking… we’ll want rsync -rvzgop ?

[notriddle (Michael Howell)]

The VM is set up to not allow you to SSH in as root directly.

[JagWaugh (Andrew Waugh)]

Both of them? i.e. can you “pull” as root from the node which you can’t login directly to as root?

(Like I wrote, I’m not the sharpest knife in the drawer.)

[schungx (Stephen Chung)]

Apparently it may be possible:

Rsync over ssh with root access on both sides

[scottfsmith (Scott Smith)]

I have a very active site I wanted to quickly move over so I decided to use this rsync method rather than the backup/restore approach which is the standard recommended one. (I am also out of disk space on the old node for backups…) There were a few glitches in the above guide so I wrote out my own steps for future reference and thought others might be interested.

The main things I changed from the above is to always do root-root ssh (both of my trees are owned by root), and to use -avz as the rsync parameters — the -a is the archive mode and will preserve dates on files as well. Without -a the second rsync to only update changed files will have to sync all the files again anyway, saving no time at all. With -a in both runs the --delete run of rsync should only take a few minutes.

So here is the plan

1. Set up the new node as above or however you do it. Make sure you run discourse-setup to generate a default app.yml file which has good parameters for your new machine. (Make sure to set this file aside so step 5. and 8. below don’t overwrite it.)

2. (24 hours in advance) On your DNS provider page set DNS TTL to a small value on all IPs that will move. The default is often 24 hours, set to 5 minutes. This will let users see the new site sooner. Set up new.example.com as a public name for the new site IP for easy access.

3. enable root login on new site — make sure line
   # PermitRootLogin no
   in /etc/ssh/sshd_config is commented out on new, if not comment and re-start sshd -
   service ssh restart

4. Set up ssh key for old-root to new-root login via the usual method (ssh-keygen in the home dir of root user on both sides, move old public key to authorized_keys on new, etc). You can of course use password login instead but it is less secure.

5. Perform initial rsync while logged in to old (old forum can still be running):

   sudo rsync -avz /var/discourse new.example.com:/var/

6. Post a message on board that example.com migration is starting in X minutes and site will be down for an hour or so. You could suggest they could temporarily use new.example.com if the site continues to not load. (The latter suggestion is for the cases where their DNS is not getting refreshed like it was supposed to). Note they will need to re-log-in on the new site so you may want to also mention that.

7. sudo ./launcher stop app
   on old site to shut it down (permanently we hope).

8. Followup rsync --delete run on old to get any changes since previous rsync:

   sudo rsync -avz --delete /var/discourse new.example.com:/var/

9. edit app.yml for configuration changes on the new site. This is basically a merge of your old app.yml and the default one you made in step 1. above, which sets the cache, workers, etc parameters based on your new machine.

10. sudo ./launcher rebuild app on new site to fire it up

11. If it works, change DNS for all *.example.com (for me I also had mg.example.com for mailgun) to the new IP.

12. (If it failed, sudo ./launcher start app on old site and try again later.)

13. restore TTL on DNS.

14. disable root login on new.

Note I am not running a CDN or https and there are may be other steps needed for those.

[pfaffman (Jay Pfaffman)]

@scottfsmith

Note I am not running a CDN or https and there are probably other steps needed for those.

Unless CDN is using an ip number rather than the hostname there is nothing to change there.

There no reason not to enable let’s encrypt.

If the new host has different ram and cpu resources, discourse-setup will update the memory settings.

[scottfsmith (Scott Smith)]

On my old machine I was running three different web servers, thats the reason https was not enabled. Yes I know I could do it but it is a lot harder with proxy servers in the middle. Its also one reason I did the move, now I have a completely vanilla install so I can easily enable https. Vanilla is a good flavor

Good point about discourse-setup updating various settings … I merged the new machine parameters into my old app.yml during step 9 above. The default app.yml also was slightly revised in other minor ways, and I could also incorporate those changes into my config. I did this before step 7 and put the file aside and plopped it in for step 9, avoiding the need to be contemplating settings while the site was down. Will edit the above to mention this.

The whole changeover (step 7 to having new site up) took 20 minutes and DNS was working for me 10 more minutes later.