Setting up Discourse import environment on DigitalOcean droplet

import

(Arpit Jalan) #1

Want to setup a Discourse import environment? Let’s get started!

Setup a Discourse instance on DigitalOcean

Setup a Discourse instance on DigitalOcean using our official guide. Email setup is not required for migration. I accessed the instance using DigitalOcean allotted IP instead of setting up a domain name.

I forked the discourse_docker repository instead of cloning the original one. The reason will be explained in further steps.

Download the import data on server

cd /var/discourse/shared/
mkdir import_data

Download the import data (using curl, wget, sftp, etc) and unzip the files if required. Let’s assume the downloaded files are vanilla_mysql.sql (MySQL database dump) and import_uploads (uploads/attachments folder).

Move the SQL dump and uploads to shared docker storage

Create /import/data/ folder in /var/discourse/shared/standalone/.

cd /var/discourse/shared/standalone/
mkdir import
cd import/
mkdir data

Move the downloaded data to the folder created above.

cd /var/discourse/shared/import_data
mv vanilla_mysql.sql /var/discourse/shared/standalone/import/data/
mv import_uploads/ /var/discourse/shared/standalone/import/data/

Prepare and build import container

Copy the existing app.yml to create new import.yml file.

cd /var/discourse
cp containers/app.yml containers/import.yml

In the first step I mentioned that I forked discourse_docker. That is because we need to create a new template file specific to our import requirements.

In my case I created a new import template file for Vanilla bulk import script and pushed it to a new branch import.

Note that the template file will need changes as per your import requirements. In my case I edited it to install MariaDB (for MySQL) and import the database dump.

Let’s fetch that custom branch now.

git fetch origin import
git reset --hard origin/import

Edit containers/import.yml to include the custom import template file - "templates/import/vanilla.template.yml". Afterwards it should look something like this:

templates:
  - "templates/postgres.template.yml"
  - "templates/redis.template.yml"
  - "templates/web.template.yml"
  - "templates/web.ratelimited.template.yml"
  - "templates/import/vanilla.template.yml"

That’s it. Now build the import container.

/var/discourse/launcher stop app
/var/discourse/launcher rebuild import

Enter into import container and verify/cleanup database

:bulb: Starting here I recommend using tmux so that the import can run in background and you can shutdown your computer if needed.

Enter into the import container.

/var/discourse/launcher enter import

Now to import the database dump run the script import_vanilla.sh script you created in the custom template.

import_vanilla.sh

The above script will import the database for you. Now let’s verify the MySQL database.

mysql -uroot
use vanilla;
show tables;

Now is the time to do database cleanup if needed. In my case I had to delete the duplicate users in the GDN_User table, so I ran this command from mariadb console:

DELETE t1 FROM GDN_User t1 INNER JOIN GDN_User t2 WHERE t1.UserID < t2.UserID AND t1.Email = t2.Email;

Start the import process

Now the source code inside the import container will be pulled from the latest Discourse repo. To make custom/specific changes to the import script you are using, I recommend pushing them to your own Discourse repo and then fetching them from inside the container.

git remote add import https://github.com/techAPJ/discourse.git
git fetch import vanilla
git reset --hard import/vanilla
rake db:migrate

Now start the import process:

su discourse -c 'bundle exec ruby script/bulk_import/vanilla.rb'

Depending on database and uploads size the time for import process to complete will vary. If you used tmux as I suggested above simply detach from the session and the import will continue running in background. You can attach at any time to view import progress.

Verify the import data and take a backup

After the import process is complete let’s verify the data. Exit from the import container and start app container.

/var/discourse/launcher stop import
/var/discourse/launcher start app

Create an Admin account for yourself.

/var/discourse/launcher enter app
rake admin:create

Visit the DigitalOcean server IP and login. Now make site setting changes as per your requirement. I recommend enabling setting login required to prevent data from leaking in public.

Now is the time to run any custom rake task or perform any post import process.

If everything looks good take a backup of the data and download it on your computer.

scp root@{DROPLET_IP_ADDRESS}:/var/discourse/shared/standalone/backups/default/backup.tar.gz .

:tada: Voila!

That’s it the downloaded backup file is now ready to be restored on Discourse servers.


Credits

This import setup is inspired from @gerhard’s excellent Importing from phpBB3 guide.