Migrate a Vanilla forum to Discourse

I know there are a lot of posts about importing contents from a vanilla forum, but theres a lack of comprehensive guide!

Not everyone can afford to hire a developer to migrate their forums and this is my way of giving back to this community

So here it goes:

You Need:

1. Terminal (if mac) or alternative for windows
2. A mind strong enough to understand and follow this guide!
3. Patience
4. A cup of coffee

Step 1: Getting the Importer file!

Use this addon to get the porter file: Vanilla Porter Addon

This will give you a .gz file. Extract it.
Now rename the newly extracted file to something short and easy and change the extension to .txt.

For example: “export_blabla_fdata_20150312_052004.txt” becomes “export.txt”

(For clarity in this guide, and less typing, rename it to export.txt.)

Step 2: Uploading the Importer file!

Now you need to upload this file to your server: (A lot of people ask me how to do this on Fiverr :o )
On Terminal (MAC) use this command:

scp path/to/exporter.txt root@[IP Address of your droplet]:/var/discourse/shared/standalone

Here’s an example:

scp user/desktop/exporter.txt root@104.236.198.206:/var/discourse/shared/standalone

Hit enter and then input the root password of your droplet!

Voila, your importer file is now on the server!

For Windows users, you can use WinSCP or FileZilla if you already have it. Connect with port 22.

Step 3: Make sure import file has correct permissions

First SSH into your server

ssh root@[ip-address of your server/droplet]

It will then ask you the root password. Enter it!

Now that you are in the server.
Navigate to the place where you uploaded the txt file.
To do so:
Run this command:

cd /var/discourse/shared/standalone/uploads

Now lets assign 666 chmod to the exporter file we just uploaded:

chmod 666 export.txt

Hit enter and you are done! On to the next step

Step 4: Setting the mood for some migration

Run these commands:

cd /var/discourse/
./launcher enter app
su discourse
cd /var/www/discourse

If you are curious. These commands are just first switching you to the discourse folder, then entering the app, then changing the user to discourse etc. Im sleepy so wont elaborate.

Step 5: Running the importer script!

RAILS_ENV=production ruby script/import_scripts/vanilla.rb /shared/export.txt

Had you been following this guide correctly and obtained a “correct” export file from the vanilla forum.
Everything should work out and Congrats!

You just migrated your forum from Vanilla to Discourse! Congrats!

Now if you have any questions, Or if you are stuck somewhere; Please feel free to post your issue below! I would try to help!
If you spot a mistake or wish to suggest an improvement to this course do let me know! (How to turn this into wiki post?)

Hey Abdul,

Thanks for the explanation, but I was trying your process and I am getting CSV parsing error. Do you know how I can get it resolved.

So… I’m trying to import a forum from YAF. I’ve passed through vanilla and I’m trying to follow these instructions.
I’m running on a bitnami dicourse vm (bitnami-discourse-1.5.3-1-r28-linux-ubuntu-14.04-x86_64).
I’ve come to step 5 :
RAILS_ENV=production ruby script/import_scripts/vanilla.rb /shared/export.txt

I get the following error:
You are running an old version of bundler, please upgrade bundler ASAP, if you are using Discourse docker, rebuild your container.
and
script/import_scripts/vanilla.rb:82:in import_users': undefined method’ for nil:NilClass (NoMethodError)
from script/import_scripts/vanilla.rb:19:in execute' from /opt/bitnami/apps/discourse/htdocs/script/import_scripts/base.rb:45:inperform’
from script/import_scripts/vanilla.rb:254:in `’
The bundler version installed in the vm is Bundler version 1.10.4

I don’t get the impression that the bundler error is fatal, but the final error is confusing me. any help here would be appreciated.

Update – if I setup an development environment like described here : Discourse as Your First Rails App
I stil get the same fatal error …
script/import_scripts/vanilla.rb:82:in import_users': undefined method’ for nil:NilClass (NoMethodError)
from script/import_scripts/vanilla.rb:19:in execute' from /opt/bitnami/apps/discourse/htdocs/script/import_scripts/base.rb:45:inperform’
from script/import_scripts/vanilla.rb:254:in `’

That’s an old version of Discourse,and an unsupported configuration.

You should upgrade to a supported os and docker.

It seems like the script doesn’t import users passwords - how would one deal with that?

I don’t really feel like telling all my users to change their passwords so how could this be fixed?

Discourse.org hosting doesn’t import passwords and they don’t seem to have trouble getting clients.

There is a migrate password plugin. The vanilla importer doesn’t currently support it. It is not clear whether the plugin will work with vanilla passwords. That said, one importer I wrote worked just fine with the plugin by just putting the password hash in the database as described on the plugin’s README.

Thanks for the reply. What should i change in the importer script to make it import the passwords to the custom field?

Follow the instructions in the migrate password plugin.

There are only 3 lines of ruby code and no example on how to combine it with the import script which is why I wrote here in case someone else has had a go at it.

I only have very basic ruby skills but the one thing I tried failed miserably.

Look at some of the other importers that the plugin supports

Ok… so I’ve setup the Vagrant development envrionment as per directions on github.

I run vagrant ssh
then
cd /vagrant/
then RAILT_ENV=development ruby script/import_scripts/vanilla.rb ./export.txt

I get the following output

    RAILS_ENV=development ruby script/import_scripts/vanilla.rb ./export_2017-01-19_210405.txt 
loading existing groups...
loading existing users...
loading existing categories...
loading existing posts...
loading existing topics...
parsing file...
reading file...
parsing users...
parsing roles...
parsing user_roles...
parsing ranks...
parsing categories...
parsing discussions...
parsing comments...
parsing conversations...
parsing user_conversations...
parsing conversation_messages...

importing users...
script/import_scripts/vanilla.rb:82:in `import_users': undefined method `[]' for nil:NilClass (NoMethodError)
	from script/import_scripts/vanilla.rb:19:in `execute'
	from /vagrant/script/import_scripts/base.rb:45:in `perform'
	from script/import_scripts/vanilla.rb:254:in `<main>'

I’m really stuck as to how to troubleshoot this problem. line 82 of the script is when it is processing users… but the syntax is such that I don’t see what the undefined method is.

I don’t see where @roles gets defined. Any guess on this @zogstrip?

“roles” are used in some “invite” code, but I think this is most likely

https://github.com/discourse/discourse/blob/master/script/import_scripts/vanilla.rb#L79-L110

That line 82 appears to be where the script is hitting a nil value.

Well… line 82 – is searching for “Administrator” – but my data has that group callled “Administrators” – I edited my file changing Administrators to Administrator,
Also there was no role of Moderator defined.
So I commented out line 83 and line 104 that deals with importing moderator[s].

This gets me beyond that, and I’m getting an error now on import_categories…

script/import_scripts/vanilla.rb:117:in `import_categories': undefined method `[]' for nil:NilClass (NoMethodError)
	from script/import_scripts/vanilla.rb:20:in `execute'
	from /vagrant/script/import_scripts/base.rb:45:in `perform'
	from script/import_scripts/vanilla.rb:254:in `<main>'

So – that error is this line:
@root_category_created_at = parse_date(@root_category[:date_inserted])
So … my data doesn’t have create dates for the categories… so… I took hard coded the date like this…
@root_category_created_at = parse_date("2012-11-19 13:39:34")

That took me passed that issue.

So – does this script drop the tables at the beginning? or if I run it over and over again, am I going to build up duplicate data?

After making those changes… I was able to successfully import the data. I had one error, one of the email addresses was malformed, and the user didn’t import.

QUESTION: if I rerun the script, will I get duplicate data?

No. It will import only what wasn’t there before. This lets you run an import and wait for days and then run it again with a new database to add just the new messages.

Thanks… that is awesome. very helpful.

Corrections for the OP:

Import script:

Old: RAILS_ENV=production ruby script/import_scripts/vanilla.rb /shared/export.txt

Should now be: RAILS_ENV=production ruby script/import_scripts/vanilla.rb public/uploads/export.txt

scp command

Also those scp commands call the file exporter.txt when it should be export.txt

So my current import file has 897169 posts… it appears to get slower as the process goes on. When it was first going it ran at 20,000 items/min. Now it’s down to ~1000/min and getting slower.

I’m wondering if I should be stopping and re-starting the import over and over, or just let it run its course?