Importing NodeBB (MongoDB) to Discourse

As you properly know, NodeBB supports two DB backends, Redis and MongoDB. Discourse importer script supports them both. In this tutorial, we will learn how to migrate NodeBB with MongoDB as DB backend. We will be using NodeBB Importer with mongo adapter. If your NodeBB forum uses Redis as backend then please follow this tutorial which demonstrate the redis adapter.

The plan

  • Preparing the development environment.

  • Export database from production environment.

  • Import the production DB to a Discourse instance.

  • Run the importer script.

What can be migrated

  • Groups
  • Attachments
  • Categories
    • Root Category => Root Category
    • Sub Category & Sub-Sub-Category => Sub-Category
  • Topics & Posts
    • pinned topic => pinned topic
    • locked topic => closed topic
    • topic views
    • upvoted_by
    • styles, mentions, emoji, and attachments.
  • Users (with the following attributes)
    • profile background
    • avatars
    • banned status
    • username
    • name
    • email
    • admin
    • bio
    • group
    • website
    • location
    • joining at

Preparing Local Development Environment

As mentioned in our plan we first need to prepare our development environment. Follow one of these guides to install Discourse itself:

:bulb: Please, consult this guide if you have any problems setting up Discourse server.

Then install MongoDB database server.

Ubuntu-18-04:

$ wget -qO - https://www.mongodb.org/static/pgp/server-4.0.asc | sudo apt-key add -
$ echo "deb [ arch=amd64 ] https://repo.mongodb.org/apt/ubuntu bionic/mongodb-org/4.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.0.list
$ sudo apt-get update
$ sudo apt-get install -y mongodb-org
$ sudo service mongod status

For more details, refer to the official guide

Mac OS:

$ brew tap mongodb/brew
$ brew install mongodb-community@4.0
$ brew services start mongodb-community@4.0
$ brew services status mongodb-community@4.0

For more details, refer to the official guide

Windows 10:

Download the installer and install MongoDB as a Windows service:

https://fastdl.mongodb.org/win32/mongodb-win32-x86_64-2008plus-ssl-4.0.12-signed.msi

Run the cmd with Administrative privileges to check if the mongo server is working properly:

"C:\Program Files\MongoDB\Server\4.0\bin\mongo.exe"

For more details, refer to the official guide.

This environment will be our Discourse server.

Exporting Production Database Dump:

Shut down your NodeBB forum (production server).

$ cd /path_to_nodebb
$ ./nodebb stop

Shut down your database:

$ sudo service mongodb stop

Backup your DB:

$ mongodump --out ~/my_backup_path/

The output of the previous command will be something like this:

2019-08-16T15:17:09.845+0300 done dumping admin.system.users (1 document)
2019-08-16T15:17:09.846+0300 done dumping admin.system.version (2 documents)
2019-08-16T15:17:09.849+0300 done dumping nodebb.sessions (10 documents)
2019-08-16T15:17:09.850+0300 done dumping nodebb.socket.io (215 documents)
2019-08-16T15:17:09.854+0300 done dumping nodebb.objects (1488 documents)

Note that the backup is actually a directory, not just a file.

:bulb: You can check the size of your DB by running use myDatabase then db.stats().dataSize; inside mongo CLI. The returned value will be in bytes.

Backup your forum assets:

$ cd /path_to_nodebb_root_directory/
$ tar -czf ./uploads.tar.gz ./public/uploads

After you have your DB and assets forum you should copy them to Discourse server.

Importing The Database

Now that we have our database we can import it in our local MongoDB instance:

$ mongorestore ~/path_of_my_backup_directory/

For more options, refer to official guide.

Next, you need to extract the uploads.tar.gz so the importer can import the assets:

$ tar xvzf uploads.tar.gz

Running The Importer Script

Now that our database and assets in place we are ready to run our importer script. Before that, we need to edit the NodeBB importer script to fit our needs.

This is the path of your NodeBB uploads folder:

ATTACHMENT_DIR = '/absolute_path/uploads'

Next, we need to tell the importer to use the mongo adapter instead of redis adapter:

adapter = NodeBB::Mongo
@client = adapter.new('mongodb://127.0.0.1:27017/nodebb')

# adapter = NodeBB::Redis
# @client = adapter.new(
# host: "localhost",
# port: "6379",
# db: 14
# )

Run the importer with clean Discourse and mongo gem support:

$ cd ~/discourse
$ echo "gem 'mongo'" >> Gemfile
$ bundle install
$ bundle exec rake db:drop db:create db:migrate
$ bundle exec ruby script/import_scripts/nodebb/nodebb.rb

The importer will connect to the MongoDB instance and migrates everything to Discourse.

After the importer finishes, start Discourse:

$ bundle exec rails server

Start Sidekiq to process the migrated data:

$ bundle exec sidekiq

You can monitor the progress at http://localhost:3000/sidekiq/queues.

Perform a Discourse backup and upload it to your Discourse production server by following this tutorial.

:tada:

If you have any questions about the process I am happy to help.

Happy migration :grinning:

7 Likes

Great work really. While I was trying, I am getting below error. Any idea what I might be doing wrong here?

importing groups
   10 / 10 (100.0%)  [474765 items/min]    
importing top level categories...
    8 / 8 (100.0%)  [437896 items/min]    
importing child categories...
   68 / 68 (100.0%)  [774048 items/min]  
importing users
 5534 / 5534 (100.0%)  [355520 items/min]    
adding users to groups...

importing topics...
Traceback (most recent call last):
	12: from script/import_scripts/nodebb/nodebb.rb:532:in `<main>'
	11: from /home/fsuzer/discourse/script/import_scripts/base.rb:47:in `perform'
	10: from script/import_scripts/nodebb/nodebb.rb:52:in `execute'
	 9: from script/import_scripts/nodebb/nodebb.rb:336:in `import_topics'
	 8: from /home/fsuzer/discourse/script/import_scripts/base.rb:877:in `batches'
	 7: from /home/fsuzer/discourse/script/import_scripts/base.rb:877:in `loop'
	 6: from /home/fsuzer/discourse/script/import_scripts/base.rb:878:in `block in batches'
	 5: from script/import_scripts/nodebb/nodebb.rb:337:in `block in import_topics'
	 4: from /home/fsuzer/discourse/script/import_scripts/nodebb/mongo.rb:71:in `topics'
	 3: from /home/fsuzer/discourse/script/import_scripts/nodebb/mongo.rb:71:in `map'
	 2: from /home/fsuzer/discourse/script/import_scripts/nodebb/mongo.rb:71:in `block in topics'
	 1: from /home/fsuzer/discourse/script/import_scripts/nodebb/mongo.rb:79:in `topic'
/home/fsuzer/discourse/script/import_scripts/nodebb/mongo.rb:100:in `post': undefined method `[]' for nil:NilClass (NoMethodError)

Some field from the topic is empty. Or perhaps the topic query returned nothing.

1 Like