Want to setup a Discourse import environment? Let’s get started!
Setup a Discourse instance on DigitalOcean
Setup a Discourse instance on DigitalOcean using our official guide. Email setup is not required for migration. I accessed the instance using DigitalOcean allotted IP instead of setting up a domain name.
Download the import data on server
cd /var/discourse/shared/
mkdir import_data
Download the import data (using curl, wget, sftp, etc) and unzip the files if required. Let’s assume the downloaded files are vanilla_mysql.sql (MySQL database dump) and import_uploads (uploads/attachments folder).
Move the SQL dump and uploads to shared docker storage
Create /import/data/ folder in /var/discourse/shared/standalone/.
cd /var/discourse/shared/standalone/
mkdir import
cd import/
mkdir data
Move the downloaded data to the folder created above.
cd /var/discourse/shared/import_data
mv vanilla_mysql.sql /var/discourse/shared/standalone/import/data/
mv import_uploads/ /var/discourse/shared/standalone/import/data/
Prepare and build import container
Copy the existing app.yml to create new import.yml file.
cd /var/discourse
cp containers/app.yml containers/import.yml
The discourse_docker repository ships with several import templates in templates/import/, including one for Vanilla. You can use it directly without needing to fork discourse_docker.
Available import templates include:
vanilla.template.yml,phpbb3.template.yml,mysql-dep.template.yml,mbox.template.yml,mssql-dep.template.yml, andchrome-dep.template.yml. Checktemplates/import/for the full list.
Vanilla Template (templates/import/vanilla.template.yml)
# This template installs MariaDB and all dependencies needed for importing from vanilla.
env:
UNICORN_SIDEKIQS: 0
params:
home: /var/www/discourse
hooks:
after_web_config:
- exec:
cd: /etc/runit/3.d
cmd:
- rm 01-nginx
- rm 02-unicorn
- file:
path: /etc/mysql/conf.d/import.cnf
contents: |
[mysqld]
# disable InnoDB since it is extremely slow in Docker container
default-storage-engine=MyISAM
default-tmp-storage-engine=MyISAM
innodb=OFF
sql_mode=NO_AUTO_CREATE_USER
datadir=/shared/import/mysql/data
skip-host-cache
skip-name-resolve
- exec:
cmd:
- mkdir -p /shared/import/mysql/data
- apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y nano libmariadb-dev mariadb-server
- sed -Ei 's/^log/#&/' /etc/mysql/my.cnf
- file:
path: /etc/service/mysql/run
chmod: "+x"
contents: |
#!/bin/bash
cd /
umask 077
# Make sure the datadir exists, is accessible and contains all system tables
mkdir -p /shared/import/mysql/data
chown mysql -R /shared/import/mysql/data
/usr/bin/mysql_install_db --user=mysql
# Shamelessly copied from http://smarden.org/runit1/runscripts.html#mysql
MYSQLADMIN='/usr/bin/mysqladmin --defaults-extra-file=/etc/mysql/debian.cnf'
trap "$MYSQLADMIN shutdown" 0
trap 'exit 2' 1 2 3 15
/usr/bin/mysqld_safe & wait
- file:
path: /etc/runit/3.d/99-mysql
chmod: "+x"
contents: |
#!/bin/bash
sv stop mysql
- file:
path: /usr/local/bin/import_vanilla.sh
chmod: "+x"
contents: |
#!/bin/bash
set -e
chown discourse -R /shared/import/data
# Allow connection as root user without password
mysql -uroot -e "UPDATE mysql.user SET plugin = 'mysql_native_password' WHERE user = 'root' AND plugin = 'unix_socket'"
mysql -uroot -e "FLUSH PRIVILEGES"
if [ -f "/shared/import/data/vanilla_mysql.sql" ]; then
if [ -f "/shared/import/mysql/imported" ] && ! sha256sum --check /shared/import/mysql/imported &>/dev/null ; then
echo "Checksum of database dump changed..."
rm /shared/import/mysql/imported
fi
if [ ! -f "/shared/import/mysql/imported" ]; then
echo "Loading database dump into MySQL..."
mysql -uroot -e "DROP DATABASE IF EXISTS vanilla"
mysql -uroot -e "CREATE DATABASE vanilla"
mysql -uroot --default-character-set=utf8 --database=vanilla < /shared/import/data/vanilla_mysql.sql
sha256sum /shared/import/data/vanilla_mysql.sql > /shared/import/mysql/imported
fi
else
sv stop mysql
fi
echo "import_vanilla.sh completed"
- exec:
cd: $home
cmd:
- mkdir -p /shared/import/data
- chown discourse -R /shared/import
before_code:
- exec:
cd: $home
cmd:
# Add your discourse core fork and pull custom code
- su discourse -c 'git remote set-url origin https://github.com/{github_username}/discourse.git'
after_bundle_exec:
- exec:
cd: $home
cmd:
# Add the gems used in the script
- echo "gem 'mysql2'" >> Gemfile
- echo "gem 'ruby-bbcode-to-md', :github => 'nlalonde/ruby-bbcode-to-md'" >> Gemfile
- su discourse -c 'bundle config unset deployment'
- su discourse -c 'bundle install --no-deployment --path vendor/bundle --jobs $(($(nproc) - 1)) --without test development'
Note that the template file will need changes as per your import requirements. If you need to customize the import script in discourse core, fork the discourse repository (not discourse_docker), push your changes to a branch, and update the before_code hook in the template with your GitHub username.
Edit containers/import.yml to include the import template file - "templates/import/vanilla.template.yml". Afterwards it should look something like this:
templates:
- "templates/postgres.template.yml"
- "templates/redis.template.yml"
- "templates/web.template.yml"
- "templates/web.ratelimited.template.yml"
- "templates/import/vanilla.template.yml"
That’s it. Now build the import container.
/var/discourse/launcher stop app
/var/discourse/launcher rebuild import
Enter into import container and verify/cleanup database
Starting here I recommend using
tmuxso that the import can run in background and you can shutdown your computer if needed.
Enter into the import container.
/var/discourse/launcher enter import
Now to import the database dump run the import_vanilla.sh script created by the template.
import_vanilla.sh
#!/bin/bash
set -e
chown discourse -R /shared/import/data
# Allow connection as root user without password
mysql -uroot -e "UPDATE mysql.user SET plugin = 'mysql_native_password' WHERE user = 'root' AND plugin = 'unix_socket'"
mysql -uroot -e "FLUSH PRIVILEGES"
if [ -f "/shared/import/data/vanilla_mysql.sql" ]; then
if [ -f "/shared/import/mysql/imported" ] && ! sha256sum --check /shared/import/mysql/imported &>/dev/null ; then
echo "Checksum of database dump changed..."
rm /shared/import/mysql/imported
fi
if [ ! -f "/shared/import/mysql/imported" ]; then
echo "Loading database dump into MySQL..."
mysql -uroot -e "DROP DATABASE IF EXISTS vanilla"
mysql -uroot -e "CREATE DATABASE vanilla"
mysql -uroot --default-character-set=utf8 --database=vanilla < /shared/import/data/vanilla_mysql.sql
sha256sum /shared/import/data/vanilla_mysql.sql > /shared/import/mysql/imported
fi
else
sv stop mysql
fi
echo "import_vanilla.sh completed"
The above script will import the database for you. Now let’s verify the MySQL database.
mysql -uroot
use vanilla;
show tables;
Now is the time to do database cleanup if needed. In my case I had to delete the duplicate users in the GDN_User table, so I ran this command from mariadb console:
DELETE t1 FROM GDN_User t1 INNER JOIN GDN_User t2 WHERE t1.UserID < t2.UserID AND t1.Email = t2.Email;
Start the import process
Now the source code inside the import container will be pulled from the latest Discourse repo (or your fork, if configured in the before_code hook). To make custom/specific changes to the import script you are using, push them to your own Discourse fork and update the template accordingly.
Now start the import process:
su discourse -c 'bundle exec ruby script/bulk_import/vanilla.rb'
Depending on database and uploads size the time for import process to complete will vary. If you used tmux as I suggested above simply detach from the session and the import will continue running in background. You can attach at any time to view import progress.
Verify the import data and take a backup
After the import process is complete let’s verify the data. Exit from the import container and start app container.
/var/discourse/launcher stop import
/var/discourse/launcher start app
Create an Admin account for yourself.
/var/discourse/launcher enter app
rake admin:create
Visit the DigitalOcean server IP and login. Now make site setting changes as per your requirement. I recommend enabling setting login required to prevent data from leaking in public.
Now is the time to run any custom rake task or perform any post import process, like a rebake.
If everything looks good take a backup of the data and download it on your computer.
scp root@{DROPLET_IP_ADDRESS}:/var/discourse/shared/standalone/backups/default/backup.tar.gz .
Voila!
That’s it the downloaded backup file is now ready to be restored on Discourse servers.
Credits
This import setup is inspired from @gerhard’s excellent Importing from phpBB3 guide.
Last Reviewed by @cocococosti on 2023-08-09T04:00:00Z
Last edited by @JammyDodger 2024-05-27T14:52:56Z
Check document
Perform check on document: