# Set up an environment to migrate another forum to Discourse

**URL:** https://meta.discourse.org/t/set-up-an-environment-to-migrate-another-forum-to-discourse/88252
**Category:** Migrating to Discourse
**Tags:** how-to
**Created:** [23.Май.2018 13:53:40 UTC](https://meta.discourse.org/t/set-up-an-environment-to-migrate-another-forum-to-discourse/88252 "2018-05-23T13:53:40Z")
**Posts on this page:** 1
**Showing post:** 1

<div class="post-metadata">

### Author: ![Discourse](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/discourse/32/148734_2.png) [@Discourse](https://meta.discourse.org/u/Discourse)
#### Post date: [23.Май.2018 13:53:40 UTC](https://meta.discourse.org/t/set-up-an-environment-to-migrate-another-forum-to-discourse/88252/1 "2018-05-23T13:53:40Z")

</div>

Want to setup a Discourse import environment? Let’s get started!

### Setup a Discourse instance on DigitalOcean

Setup a Discourse instance on DigitalOcean using our [official guide](https://github.com/discourse/discourse/blob/main/docs/INSTALL-cloud.md). Email setup is not required for migration. I accessed the instance using DigitalOcean allotted IP instead of setting up a domain name.

### Download the import data on server

```plaintext
cd /var/discourse/shared/
mkdir import_data

```

Download the import data (using curl, wget, sftp, etc) and unzip the files if required. Let’s assume the downloaded files are `vanilla_mysql.sql` (MySQL database dump) and `import_uploads` (uploads/attachments folder).

### Move the SQL dump and uploads to shared docker storage

Create `/import/data/` folder in `/var/discourse/shared/standalone/`.

```plaintext
cd /var/discourse/shared/standalone/
mkdir import
cd import/
mkdir data

```

Move the downloaded data to the folder created above.

```plaintext
cd /var/discourse/shared/import_data
mv vanilla_mysql.sql /var/discourse/shared/standalone/import/data/
mv import_uploads/ /var/discourse/shared/standalone/import/data/

```

### Prepare and build import container

Copy the existing `app.yml` to create new `import.yml` file.

```plaintext
cd /var/discourse
cp containers/app.yml containers/import.yml

```

The `discourse_docker` repository ships with several import templates in `templates/import/`, including one for Vanilla. You can use it directly without needing to fork `discourse_docker`.

> 💡 Available import templates include: `vanilla.template.yml`, `phpbb3.template.yml`, `mysql-dep.template.yml`, `mbox.template.yml`, `mssql-dep.template.yml`, and `chrome-dep.template.yml`. Check `templates/import/` for the full list.

> **Vanilla Template (templates/import/vanilla.template.yml)**
>
> ```plaintext
> # This template installs MariaDB and all dependencies needed for importing from vanilla.
> env:
> UNICORN_SIDEKIQS: 0
> 
> params:
> home: /var/www/discourse
> 
> hooks:
> after_web_config:
> - exec:
> cd: /etc/runit/3.d
> cmd:
> - rm 01-nginx
> - rm 02-unicorn
> 
> - file:
> path: /etc/mysql/conf.d/import.cnf
> contents: |
> [mysqld]
> # disable InnoDB since it is extremely slow in Docker container
> default-storage-engine=MyISAM
> default-tmp-storage-engine=MyISAM
> innodb=OFF
> sql_mode=NO_AUTO_CREATE_USER
> 
> datadir=/shared/import/mysql/data
> 
> skip-host-cache
> skip-name-resolve
> 
> - exec:
> cmd:
> - mkdir -p /shared/import/mysql/data
> - apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y nano libmariadb-dev mariadb-server
> - sed -Ei 's/^log/#&/' /etc/mysql/my.cnf
> 
> - file:
> path: /etc/service/mysql/run
> chmod: "+x"
> contents: |
> #!/bin/bash
> cd /
> umask 077
> 
> # Make sure the datadir exists, is accessible and contains all system tables
> mkdir -p /shared/import/mysql/data
> chown mysql -R /shared/import/mysql/data
> /usr/bin/mysql_install_db --user=mysql
> 
> # Shamelessly copied from http://smarden.org/runit1/runscripts.html#mysql
> MYSQLADMIN='/usr/bin/mysqladmin --defaults-extra-file=/etc/mysql/debian.cnf'
> trap "$MYSQLADMIN shutdown" 0
> trap 'exit 2' 1 2 3 15
> /usr/bin/mysqld_safe & wait
> 
> - file:
> path: /etc/runit/3.d/99-mysql
> chmod: "+x"
> contents: |
> #!/bin/bash
> sv stop mysql
> 
> - file:
> path: /usr/local/bin/import_vanilla.sh
> chmod: "+x"
> contents: |
> #!/bin/bash
> set -e
> 
> chown discourse -R /shared/import/data
> 
> # Allow connection as root user without password
> mysql -uroot -e "UPDATE mysql.user SET plugin = 'mysql_native_password' WHERE user = 'root' AND plugin = 'unix_socket'"
> mysql -uroot -e "FLUSH PRIVILEGES"
> 
> if [-f "/shared/import/data/vanilla_mysql.sql"]; then
> if [-f "/shared/import/mysql/imported"] && ! sha256sum --check /shared/import/mysql/imported &>/dev/null ; then
> echo "Checksum of database dump changed..."
> rm /shared/import/mysql/imported
> fi
> 
> if [! -f "/shared/import/mysql/imported"]; then
> echo "Loading database dump into MySQL..."
> mysql -uroot -e "DROP DATABASE IF EXISTS vanilla"
> mysql -uroot -e "CREATE DATABASE vanilla"
> mysql -uroot --default-character-set=utf8 --database=vanilla < /shared/import/data/vanilla_mysql.sql
> sha256sum /shared/import/data/vanilla_mysql.sql > /shared/import/mysql/imported
> fi
> else
> sv stop mysql
> fi
> 
> echo "import_vanilla.sh completed"
> 
> - exec:
> cd: $home
> cmd:
> - mkdir -p /shared/import/data
> - chown discourse -R /shared/import
> 
> before_code:
> - exec:
> cd: $home
> cmd:
> # Add your discourse core fork and pull custom code
> - su discourse -c 'git remote set-url origin https://github.com/{github_username}/discourse.git'
> 
> after_bundle_exec:
> - exec:
> cd: $home
> cmd:
> # Add the gems used in the script
> - echo "gem 'mysql2'" >> Gemfile
> - echo "gem 'ruby-bbcode-to-md', :github => 'nlalonde/ruby-bbcode-to-md'" >> Gemfile
> - su discourse -c 'bundle config unset deployment'
> - su discourse -c 'bundle install --no-deployment --path vendor/bundle --jobs $(($(nproc) - 1)) --without test development'
> 
> ```

Note that the template file will need changes as per your import requirements. If you need to customize the import script in `discourse` core, fork the **[discourse](https://github.com/discourse/discourse)** repository (not `discourse_docker`), push your changes to a branch, and update the `before_code` hook in the template with your GitHub username.

Edit `containers/import.yml` to include the import template file `- "templates/import/vanilla.template.yml"`. Afterwards it should look something like this:

```plaintext
templates:
  - "templates/postgres.template.yml"
  - "templates/redis.template.yml"
  - "templates/web.template.yml"
  - "templates/web.ratelimited.template.yml"
  - "templates/import/vanilla.template.yml"

```

That’s it. Now build the import container.

```plaintext
/var/discourse/launcher stop app
/var/discourse/launcher rebuild import

```

### Enter into import container and verify/cleanup database

> 💡 Starting here I recommend using [**`tmux`**](https://danielmiessler.com/study/tmux/) so that the import can run in background and you can shutdown your computer if needed.

Enter into the import container.

```plaintext
/var/discourse/launcher enter import

```

Now to import the database dump run the `import_vanilla.sh` script created by the template.

> **import\_vanilla.sh**
>
> ```plaintext
> #!/bin/bash
> set -e
> 
> chown discourse -R /shared/import/data
> 
> # Allow connection as root user without password
> mysql -uroot -e "UPDATE mysql.user SET plugin = 'mysql_native_password' WHERE user = 'root' AND plugin = 'unix_socket'"
> mysql -uroot -e "FLUSH PRIVILEGES"
> 
> if [-f "/shared/import/data/vanilla_mysql.sql"]; then
> if [-f "/shared/import/mysql/imported"] && ! sha256sum --check /shared/import/mysql/imported &>/dev/null ; then
> echo "Checksum of database dump changed..."
> rm /shared/import/mysql/imported
> fi
> 
> if [! -f "/shared/import/mysql/imported"]; then
> echo "Loading database dump into MySQL..."
> mysql -uroot -e "DROP DATABASE IF EXISTS vanilla"
> mysql -uroot -e "CREATE DATABASE vanilla"
> mysql -uroot --default-character-set=utf8 --database=vanilla < /shared/import/data/vanilla_mysql.sql
> sha256sum /shared/import/data/vanilla_mysql.sql > /shared/import/mysql/imported
> fi
> else
> sv stop mysql
> fi
> 
> echo "import_vanilla.sh completed"
> 
> ```

The above script will import the database for you. Now let’s verify the MySQL database.

```plaintext
mysql -uroot
use vanilla;
show tables;

```

Now is the time to do database cleanup if needed. In my case I had to delete the duplicate users in the `GDN_User` table, so I ran this command from mariadb console:

```plaintext
DELETE t1 FROM GDN_User t1 INNER JOIN GDN_User t2 WHERE t1.UserID < t2.UserID AND t1.Email = t2.Email;

```

### Start the import process

Now the source code inside the `import` container will be pulled from the latest Discourse repo (or your fork, if configured in the `before_code` hook). To make custom/specific changes to the import script you are using, push them to your own Discourse fork and update the template accordingly.

Now start the import process:

```plaintext
su discourse -c 'bundle exec ruby script/bulk_import/vanilla.rb'

```

Depending on database and uploads size the time for import process to complete will vary. If you used `tmux` as I suggested above simply detach from the session and the import will continue running in background. You can attach at any time to view import progress.

### Verify the import data and take a backup

After the import process is complete let’s verify the data. Exit from the import container and start app container.

```plaintext
/var/discourse/launcher stop import
/var/discourse/launcher start app

```

Create an Admin account for yourself.

```plaintext
/var/discourse/launcher enter app
rake admin:create

```

Visit the DigitalOcean server IP and login. Now make site setting changes as per your requirement. I recommend enabling setting `login required` to prevent data from leaking in public.

Now is the time to run any custom rake task or perform any post import process, like a [rebake](https://meta.discourse.org/t/rebake-all-posts-matching-a-pattern/48713).

If everything looks good take a backup of the data and download it on your computer.

```plaintext
scp root@{DROPLET_IP_ADDRESS}:/var/discourse/shared/standalone/backups/default/backup.tar.gz .

```

### 🎉 Voila!

That’s it the downloaded backup file is now ready to be restored on Discourse servers.

* * *

### Credits

This import setup is inspired from @gerhard’s excellent [Importing from phpBB3](https://meta.discourse.org/t/importing-from-phpbb3/30810) guide.

* * *

_Last Reviewed by @cocococosti on 2023-08-09T04:00:00Z_

> Last edited by @JammyDodger 2024-05-27T14:52:56Z
> 
> > **Check document**
> >
> > Perform check on document:

---

_[View the full topic](https://meta.discourse.org/t/set-up-an-environment-to-migrate-another-forum-to-discourse/88252)._