Migrate a Vanilla forum to Discourse

I know there are a lot of posts about importing contents from a vanilla forum, but theres a lack of comprehensive guide!

Not everyone can afford to hire a developer to migrate their forums and this is my way of giving back to this community :smile:

So here it goes:

You Need:

  1. Terminal (if mac) or alternative for windows
  2. A mind strong enough to understand and follow this guide!
  3. Patience
  4. A cup of coffee

Step 1: Getting the Importer file!

Use this addon to get the porter file: Vanilla Porter Addon

This will give you a .gz file. Extract it.
Now rename the newly extracted file to something short and easy and change the extension to .txt.

For example: ā€œexport_blabla_fdata_20150312_052004.txtā€ becomes ā€œexport.txtā€

(For clarity in this guide, and less typing, rename it to export.txt.)

Step 2: Uploading the Importer file!

Now you need to upload this file to your server: (A lot of people ask me how to do this on Fiverr :o )
On Terminal (MAC) use this command:

scp path/to/exporter.txt root@[IP Address of your droplet]:/var/discourse/shared/standalone

Hereā€™s an example:

scp user/desktop/exporter.txt root@104.236.198.206:/var/discourse/shared/standalone

Hit enter and then input the root password of your droplet!

Voila, your importer file is now on the server!

For Windows users, you can use WinSCP or FileZilla if you already have it. Connect with port 22.

Step 3: Make sure import file has correct permissions

First SSH into your server

ssh root@[ip-address of your server/droplet]

It will then ask you the root password. Enter it!

Now that you are in the server.
Navigate to the place where you uploaded the txt file.
To do so:
Run this command:

cd /var/discourse/shared/standalone/uploads

Now lets assign 666 chmod to the exporter file we just uploaded:

chmod 666 export.txt

Hit enter and you are done! On to the next step

Step 4: Setting the mood for some migration :wink:

Run these commands:

cd /var/discourse/
./launcher enter app
su discourse
cd /var/www/discourse

If you are curious. These commands are just first switching you to the discourse folder, then entering the app, then changing the user to discourse etc. Im sleepy so wont elaborate.

Step 5: Running the importer script!

RAILS_ENV=production ruby script/import_scripts/vanilla.rb /shared/export.txt

Had you been following this guide correctly and obtained a ā€œcorrectā€ export file from the vanilla forum.
Everything should work out and Congrats!

You just migrated your forum from Vanilla to Discourse! Congrats!

Now if you have any questions, Or if you are stuck somewhere; Please feel free to post your issue below! I would try to help!
If you spot a mistake or wish to suggest an improvement to this course do let me know! (How to turn this into wiki post?)

15 Likes

Hey Abdul,

Thanks for the explanation, but I was trying your process and I am getting CSV parsing error. Do you know how I can get it resolved.

Soā€¦ Iā€™m trying to import a forum from YAF. Iā€™ve passed through vanilla and Iā€™m trying to follow these instructions.
Iā€™m running on a bitnami dicourse vm (bitnami-discourse-1.5.3-1-r28-linux-ubuntu-14.04-x86_64).
Iā€™ve come to step 5 :
RAILS_ENV=production ruby script/import_scripts/vanilla.rb /shared/export.txt

I get the following error:
You are running an old version of bundler, please upgrade bundler ASAP, if you are using Discourse docker, rebuild your container.
and
script/import_scripts/vanilla.rb:82:in import_users': undefined methodā€™ for nil:NilClass (NoMethodError)
from script/import_scripts/vanilla.rb:19:in execute' from /opt/bitnami/apps/discourse/htdocs/script/import_scripts/base.rb:45:inperformā€™
from script/import_scripts/vanilla.rb:254:in `ā€™
The bundler version installed in the vm is Bundler version 1.10.4

I donā€™t get the impression that the bundler error is fatal, but the final error is confusing me. any help here would be appreciated.

Update ā€“ if I setup an development environment like described here : Discourse as Your First Rails App
I stil get the same fatal error ā€¦
script/import_scripts/vanilla.rb:82:in import_users': undefined methodā€™ for nil:NilClass (NoMethodError)
from script/import_scripts/vanilla.rb:19:in execute' from /opt/bitnami/apps/discourse/htdocs/script/import_scripts/base.rb:45:inperformā€™
from script/import_scripts/vanilla.rb:254:in `ā€™

Thatā€™s an old version of Discourse,and an unsupported configuration.

You should upgrade to a supported os and docker.

4 Likes

It seems like the script doesnā€™t import users passwords - how would one deal with that?

I donā€™t really feel like telling all my users to change their passwords so how could this be fixed?

Discourse.org hosting doesnā€™t import passwords and they donā€™t seem to have trouble getting clients.

There is a migrate password plugin. The vanilla importer doesnā€™t currently support it. It is not clear whether the plugin will work with vanilla passwords. That said, one importer I wrote worked just fine with the plugin by just putting the password hash in the database as described on the pluginā€™s README.

2 Likes

Thanks for the reply. What should i change in the importer script to make it import the passwords to the custom field?

Follow the instructions in the migrate password plugin.

There are only 3 lines of ruby code and no example on how to combine it with the import script which is why I wrote here in case someone else has had a go at it.

I only have very basic ruby skills but the one thing I tried failed miserably.

1 Like

Look at some of the other importers that the plugin supports

Okā€¦ so Iā€™ve setup the Vagrant development envrionment as per directions on github.

I run vagrant ssh
then
cd /vagrant/
then RAILT_ENV=development ruby script/import_scripts/vanilla.rb ./export.txt

I get the following output

    RAILS_ENV=development ruby script/import_scripts/vanilla.rb ./export_2017-01-19_210405.txt 
loading existing groups...
loading existing users...
loading existing categories...
loading existing posts...
loading existing topics...
parsing file...
reading file...
parsing users...
parsing roles...
parsing user_roles...
parsing ranks...
parsing categories...
parsing discussions...
parsing comments...
parsing conversations...
parsing user_conversations...
parsing conversation_messages...

importing users...
script/import_scripts/vanilla.rb:82:in `import_users': undefined method `[]' for nil:NilClass (NoMethodError)
	from script/import_scripts/vanilla.rb:19:in `execute'
	from /vagrant/script/import_scripts/base.rb:45:in `perform'
	from script/import_scripts/vanilla.rb:254:in `<main>'

Iā€™m really stuck as to how to troubleshoot this problem. line 82 of the script is when it is processing usersā€¦ but the syntax is such that I donā€™t see what the undefined method is.

1 Like

I donā€™t see where @roles gets defined. Any guess on this @zogstrip?

ā€œrolesā€ are used in some ā€œinviteā€ code, but I think this is most likely

https://github.com/discourse/discourse/blob/master/script/import_scripts/vanilla.rb#L79-L110

That line 82 appears to be where the script is hitting a nil value.

Wellā€¦ line 82 ā€“ is searching for ā€œAdministratorā€ ā€“ but my data has that group callled ā€œAdministratorsā€ ā€“ I edited my file changing Administrators to Administrator,
Also there was no role of Moderator defined.
So I commented out line 83 and line 104 that deals with importing moderator[s].

This gets me beyond that, and Iā€™m getting an error now on import_categoriesā€¦

script/import_scripts/vanilla.rb:117:in `import_categories': undefined method `[]' for nil:NilClass (NoMethodError)
	from script/import_scripts/vanilla.rb:20:in `execute'
	from /vagrant/script/import_scripts/base.rb:45:in `perform'
	from script/import_scripts/vanilla.rb:254:in `<main>'

So ā€“ that error is this line:
@root_category_created_at = parse_date(@root_category[:date_inserted])
So ā€¦ my data doesnā€™t have create dates for the categoriesā€¦ soā€¦ I took hard coded the date like thisā€¦
@root_category_created_at = parse_date("2012-11-19 13:39:34")

That took me passed that issue.

So ā€“ does this script drop the tables at the beginning? or if I run it over and over again, am I going to build up duplicate data?

2 Likes

After making those changesā€¦ I was able to successfully import the data. I had one error, one of the email addresses was malformed, and the user didnā€™t import.

QUESTION: if I rerun the script, will I get duplicate data?

No. It will import only what wasnā€™t there before. This lets you run an import and wait for days and then run it again with a new database to add just the new messages.

2 Likes

Thanksā€¦ that is awesome. very helpful.

1 Like

Corrections for the OP:

Import script:

Old: RAILS_ENV=production ruby script/import_scripts/vanilla.rb /shared/export.txt

Should now be: RAILS_ENV=production ruby script/import_scripts/vanilla.rb public/uploads/export.txt

scp command

Also those scp commands call the file exporter.txt when it should be export.txt

:smiley:

2 Likes

So my current import file has 897169 postsā€¦ it appears to get slower as the process goes on. When it was first going it ran at 20,000 items/min. Now itā€™s down to ~1000/min and getting slower.

Iā€™m wondering if I should be stopping and re-starting the import over and over, or just let it run its course?