vBulletin migration guide?


(Tim) #1

I’ve been doing reading on the forums here, many people have mentioned successfully migrating from vBulletin, often times with some changes to the standard process. However, I haven’t actually seen a guide on doing said Migration. I’m not terribly familiar with Ruby so I’m not entirely sure how to go about running the importer script. Any help would be appreciated. If I can point the script to an active copy of the database for vBulletin, that’d be cool, else I can export it and send the file to the Discourse server.


(Tim) #2

To be specific I’m coming from vBulletin 5. I’ve read a lot that the importer script was written for vBulletin 3, however all that appears to be over a year old, every forum post I’ve found about it. So I was wondering if something more recent has developed. The only potentially successful idea I’ve heard is that some people recommended migrating from vBulletin to something else that does have a vBulletin 5 importer, and then from that something else to Discourse. I’ll give that some looking into, but it sounds like a lot more trouble than it should be.


(Gerhard Schlager) #3

There’s currently no importer for vBulletin 5 available. vBulletin 5 is in many parts completely different from its predecessors. I looked into this a few months ago and my conclusion was that it will take a considerate amount of work to develop an importer for it.

So, unless you have a budget for developing a vBulletin 5 importer or at least find enough people who can fund this, you are out of luck. Then your only option is to migrate to a different system first before migrating to Discourse.


(Tim) #4

Thanks for the response, the only other one I feel terribly familiar with is SMF. But they only have officially advertised (that is, they officially recommend certain scripts for, but don’t necessarily provide much official support for seeing as it’s another open source project) importers for vBulletin 1-4. I guess I can look elsewhere, but I’m considering just manually moving over the big stuff and anything anyone specifically requests. We’re a fairly small fan community for a mobile game (because the developer couldn’t be bothered to have any sort of official forum or a wiki).

So, thanks for the info, at this point I’m going to do some discussion with the higher ups in the group to make a decision. If I do get this figured out, I’ll try posting a walkthrough of what I did, so it’s available for anyone else.


(Jeff Atwood) #5

If your budget allows, our Business hosting plan covers a migration, and we’d write an importer for vBulletin 5 as needed: https://discourse.org/buy


(Michael - DiscourseHosting.com) #6

We are currently working on the last bits and pieces of our second VB5 import. When we’re done (probably this Sunday or Monday), we’re going to make a pull request for the importer. We had to do quite some changes for the second import, so we’re going to see if we had any regressions, and do the first import over again to see if we broke something.

EDIT Here’s a prerelease :slight_smile: https://gist.github.com/discoursehosting/579d63db25d3a2a0d9cac85d9465aeff


(Tim) #7

Thanks for that. I’ll look into it in a couple days then.


(Michael - DiscourseHosting.com) #8

I’ve made a pull request for vbulletin5.rb :slight_smile:

It took a bit more time because we found out that we had to support attachments-in-the-database as well.


(Gareth Williams) #9

Hey All,

I have been trying to use the Vbulletin importer and have almost got it going, however, it seems that I miss out on a lot of posts.

I also cannot seem to get over this error:

loading existing groups...
loading existing users...
loading existing categories...
loading existing posts...
loading existing topics...

importing groups...
        6 / 17 ( 35.3%)  Failed to create group id 7 Moderators: ["Name has already been taken"]
       17 / 17 (100.0%)  

Moderators appears to be an automatic group, so I cant delete - any ideas - could this be causing the issue that I cant see any posts?

I also see the following when importing posts:

importing child categories...

importing topics...

importing posts...
      133 / 133 (100.0%)  [1580768 items/min]  
importing attachments...
       40 / 133 ( 30.1%)  
Closing topics...

Postprocessing posts...
       40 / 40 (100.0%)  

updating bumped_at on topics

updating last posted at on users

updating last seen at on users

Updating topic reply counts...
    15987 / 15985 (100.0%)  [7852 items/min]  .Updating first_post_created_at...
Updating user post_count...
Updating user topic_count...

updating featured topic users
       37 / 37 (100.0%)  
updating featured topics in categories
       31 / 31 (100.0%)  
updating user topic reply counts
    15985 / 15985 (100.0%)  
resetting topic counters
       37 / 37 (100.0%)  

This forum has many many posts - in fact you can see it here:

any ideas?

edit - this is the vBulletin 5. and I forgot to say thanks for a great platform!

Diggint further, I wonder if it is something with the content type id - the import is looking for 23, however when i look at these in the sql i see:

+----------+---------------+
| count(*) | contenttypeid |
+----------+---------------+
|     2755 |            15 |
|       11 |            16 |
|      209 |            19 |
|   466518 |            20 |
|       30 |            21 |
|      133 |            22 |
|      800 |            23 |
|     4856 |            24 |
|        1 |            26 |
|        3 |            27 |
|      158 |            28 |

And I see now that I have 133 imported as they are the content type id 22, but most of my content is 20 - am i onto something here? Is it something that a minor tweak to the importer would solve?

One last ninja edit :slight_smile: Here are the content type id’s I have:

+---------------+-----------------------+-----------+----------+-----------+--------+-----------+--------------+
| contenttypeid | class                 | packageid | canplace | cansearch | cantag | canattach | isaggregator |
+---------------+-----------------------+-----------+----------+-----------+--------+-----------+--------------+
|             1 | Post                  |         1 | 0        | 0         | 0      | 1         | 0            |
|             2 | Thread                |         1 | 0        | 0         | 1      | 0         | 0            |
|             3 | Forum                 |         1 | 0        | 0         | 0      | 0         | 0            |
|             4 | Announcement          |         1 | 0        | 0         | 0      | 0         | 0            |
|             5 | SocialGroupMessage    |         1 | 0        | 0         | 0      | 0         | 0            |
|             6 | SocialGroupDiscussion |         1 | 0        | 0         | 0      | 0         | 0            |
|             7 | SocialGroup           |         1 | 0        | 0         | 0      | 1         | 0            |
|             8 | Album                 |         1 | 0        | 0         | 0      | 1         | 0            |
|             9 | Picture               |         1 | 0        | 0         | 0      | 0         | 0            |
|            10 | PictureComment        |         1 | 0        | 0         | 0      | 0         | 0            |
|            11 | VisitorMessage        |         1 | 0        | 0         | 0      | 0         | 0            |
|            12 | User                  |         1 | 0        | 0         | 0      | 0         | 0            |
|            13 | Event                 |         1 | 0        | 0         | 0      | 0         | 0            |
|            14 | Calendar              |         1 | 0        | 0         | 0      | 0         | 0            |
|            15 | PrivateMessage        |         1 | 0        | 0         | 0      | 0         | 0            |
|            16 | Infraction            |         1 | 0        | 0         | 0      | 0         | 0            |
|            17 | Signature             |         1 | 0        | 0         | 0      | 0         | 0            |
|            18 | UserNote              |         1 | 0        | 0         | 0      | 0         | 0            |
|            19 | Channel               |         1 | 1        | 0         | 0      | 0         | 1            |
|            20 | Text                  |         1 | 1        | 1         | 1      | 1         | 0            |
|            21 | Poll                  |         1 | 1        | 1         | 0      | 0         | 0            |
|            22 | Gallery               |         1 | 1        | 1         | 1      | 1         | 1            |
|            23 | Photo                 |         1 | 0        | 1         | 1      | 1         | 1            |
|            24 | Attach                |         1 | 0        | 0         | 0      | 0         | 0            |
|            25 | Video                 |         1 | 1        | 1         | 1      | 1         | 1            |
|            26 | Link                  |         1 | 1        | 1         | 1      | 1         | 0            |
|            27 | Report                |         1 | 0        | 0         | 0      | 0         | 0            |
|            28 | Redirect              |         1 | 0        | 0         | 0      | 0         | 0            |
|            29 | BlogEntry             |         2 | 0        | 0         | 0      | 1         | 0            |
|            30 | BlogComment           |         2 | 0        | 0         | 0      | 1         | 0            |
|            31 | Article               |         3 | 0        | 0         | 1      | 1         | 0            |
+---------------+-----------------------+-----------+----------+-----------+--------+-----------+--------------+

(Michael - DiscourseHosting.com) #10

Indeed the contenttypeid’s are hardcoded in the importer, but apparently they’re not as stable across vBulletin installs as I thought… quick and dirty fix: just add 20 wherever you see 22 or 23 in the code. Let me know if that works for you!

The Moderators category error is harmless, you can safely ignore it.


(Gareth Williams) #11

Hey Michael,

good man! I have worked that out at the same time :slight_smile:

Its importing now I think:

importing top level categories...
       27 / 27 (100.0%)  
importing child categories...

importing topics...
     4852 / 4852 (100.0%)  [478349 items/min]   
importing posts...
   109000 / 466518 ( 23.4%)  [29018 items/min]   

I am struggling with child categories - it didnt import any at all - any thoughts on that? I am SO close :slight_smile:


(Gareth Williams) #12

AHA thats another 23 i needed to turn to a 20 :slight_smile:

Legend - thanks @michaeld

I really appreciate all the effort that went into this, if I am ever able to contribute back , i certainly will


(Gareth Williams) #13

@michaeld

Looks like I spoke too soon, i wonder if 23 is right for the categories? Just running the import now, im seeing this error:

importing child categories...
     1060 / 5523 ( 19.2%)  /home/vagrant/.rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/validations.rb:79:in `raise_record_invalid': Validation failed: Category Name has already been taken (ActiveRecord::RecordInvalid)
	from /home/vagrant/.rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/validations.rb:43:in `save!'
	from /home/vagrant/.rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/attribute_methods/dirty.rb:29:in `save!'
	from /home/vagrant/.rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/transactions.rb:291:in `block in save!'
	from /home/vagrant/.rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/transactions.rb:351:in `block in with_transaction_returning_status'
	from /home/vagrant/.rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `block in transaction'
	from /home/vagrant/.rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/transaction.rb:184:in `within_new_transaction'
	from /home/vagrant/.rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `transaction'
	from /home/vagrant/.rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/transactions.rb:220:in `transaction'
	from /home/vagrant/.rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/transactions.rb:348:in `with_transaction_returning_status'
	from /home/vagrant/.rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/transactions.rb:291:in `save!'
	from /vagrant/script/import_scripts/base.rb:423:in `create_category'
	from /vagrant/script/import_scripts/base.rb:393:in `block in create_categories'
	from /vagrant/script/import_scripts/base.rb:374:in `each'
	from /vagrant/script/import_scripts/base.rb:374:in `create_categories'
	from script/import_scripts/vbulletin5.rb:208:in `import_categories'
	from script/import_scripts/vbulletin5.rb:37:in `execute'
	from /vagrant/script/import_scripts/base.rb:45:in `perform'
	from script/import_scripts/vbulletin5.rb:634:in `<main>'

(Gareth Williams) #14

yup definitely something wrong, I am seeing lots of categories here which are topics, any idea?


(Michael - DiscourseHosting.com) #15

Hard to tell, especially since you modified stuff and the last part of the error message is missing.
Can you post the last few lines of that stack trace please?


(Gareth Williams) #16

Hey @michaeld

Thanks for the assistance, this is all i am getting from the run:

vagrant@discourse:/vagrant$ bundle exec ruby script/import_scripts/vbulletin5.rb
DEPRECATION WARNING: Sprockets method `register_engine` is deprecated.
Please register a mime type using `register_mime_type` then
use `register_compressor` or `register_transformer`.
https://github.com/rails/sprockets/blob/master/guides/extending_sprockets.md#supporting-all-versions-of-sprockets-in-processors
 (called from <top (required)> at /vagrant/lib/es6_module_transpiler/sprockets.rb:3)
DEPRECATION WARNING: You are using the a deprecated processor interface SourceURL.
Please update your processor interface:
https://github.com/rails/sprockets/blob/master/guides/extending_sprockets.md#supporting-all-versions-of-sprockets-in-processors
 (called from block in <top (required)> at /vagrant/config/initializers/010-discourse_iife.rb:8)
DEPRECATION WARNING: Sprockets method `register_engine` is deprecated.
Please register a mime type using `register_mime_type` then
use `register_compressor` or `register_transformer`.
https://github.com/rails/sprockets/blob/master/guides/extending_sprockets.md#supporting-all-versions-of-sprockets-in-processors
 (called from block in <top (required)> at /vagrant/config/initializers/100-sprockets.rb:17)
DEPRECATION WARNING: Sprockets method `register_engine` is deprecated.
Please register a mime type using `register_mime_type` then
use `register_compressor` or `register_transformer`.
https://github.com/rails/sprockets/blob/master/guides/extending_sprockets.md#supporting-all-versions-of-sprockets-in-processors
 (called from block in <top (required)> at /vagrant/config/initializers/100-sprockets.rb:18)
loading existing groups...
loading existing users...
loading existing categories...
loading existing posts...
loading existing topics...

importing groups...
        6 / 17 ( 35.3%)  Failed to create group id 7 Moderators: ["Name has already been taken"]
       17 / 17 (100.0%)  
importing users
     9244 / 15990 ( 57.8%)  Failed to create user id: 10067, username: Conyzooromi, email: xenfsales@gm...ail.com
user errors: ["Email is invalid"]
    11811 / 15990 ( 73.9%)  Failed to create user id: 12710, username: MichaellRunon, email: michael..runon@pcc.mailboxxx.net
user errors: ["Email is invalid"]
    12135 / 15990 ( 75.9%)  Failed to create user id: 13037, username: houseboatsmi, email: houseboatsmi@.nextfash.com
user errors: ["Email is invalid"]
    15103 / 15990 ( 94.5%)  Failed to create user id: 16186, username: Katielou, email: kate...@yahoo.com
user errors: ["Email is invalid"]
    15905 / 15990 ( 99.5%)  Failed to create user id: 17546, username: FashionPolice, email: nicko,acuna@merliongroup.ae
user errors: ["Email is invalid"]
    15952 / 15990 ( 99.8%)  Failed to create user id: 17593, username: Frankevowl, email: ad.a.m.@oddluzamy.nieruchomosci.pl
user errors: ["Email is invalid"]
    15990 / 15990 (100.0%)  
importing top level categories...
       27 / 27 (100.0%)  
importing child categories...
     1060 / 5523 ( 19.2%)  /home/vagrant/.rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/validations.rb:79:in `raise_record_invalid': Validation failed: Category Name has already been taken (ActiveRecord::RecordInvalid)
	from /home/vagrant/.rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/validations.rb:43:in `save!'
	from /home/vagrant/.rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/attribute_methods/dirty.rb:29:in `save!'
	from /home/vagrant/.rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/transactions.rb:291:in `block in save!'
	from /home/vagrant/.rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/transactions.rb:351:in `block in with_transaction_returning_status'
	from /home/vagrant/.rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `block in transaction'
	from /home/vagrant/.rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/transaction.rb:184:in `within_new_transaction'
	from /home/vagrant/.rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `transaction'
	from /home/vagrant/.rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/transactions.rb:220:in `transaction'
	from /home/vagrant/.rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/transactions.rb:348:in `with_transaction_returning_status'
	from /home/vagrant/.rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/transactions.rb:291:in `save!'
	from /vagrant/script/import_scripts/base.rb:423:in `create_category'
	from /vagrant/script/import_scripts/base.rb:393:in `block in create_categories'
	from /vagrant/script/import_scripts/base.rb:374:in `each'
	from /vagrant/script/import_scripts/base.rb:374:in `create_categories'
	from script/import_scripts/vbulletin5.rb:208:in `import_categories'
	from script/import_scripts/vbulletin5.rb:37:in `execute'
	from /vagrant/script/import_scripts/base.rb:45:in `perform'
	from script/import_scripts/vbulletin5.rb:634:in `<main>'

Should I be seeing something else to help debug? Here is the import script (modified) im using:

http://pastebin.com/EFq047ZD


(Gareth Williams) #17

hey @michaeld

I think i have fixed it - the import is running now - I needed 19 in the categories (will find out soon)


(Gareth Williams) #18

Hey @michaeld

The import ran, the numbers are looking better however, I cant see any posts when i load discourse.

The categories are all there 100%, the topics say they imported but nothing is shown - here is what I see:

Here is the import log - the numbers look right - until i see the post processing posts any ideas at all?

loading existing groups...
loading existing users...
loading existing categories...
loading existing posts...
loading existing topics...

importing groups...
       17 / 17 (100.0%)
importing users
<snip email address of problem import>

    15990 / 15990 (100.0%)
importing top level categories...
       27 / 27 (100.0%)
importing child categories...
      157 / 157 (100.0%)
importing topics...
     4852 / 4852 (100.0%)  [445687 items/min]
importing posts...
   466518 / 466518 (100.0%)  [155886 items/min]
importing attachments...
      195 / 466518 (  0.0%)
Closing topics...

Postprocessing posts...
      195 / 195 (100.0%)

updating bumped_at on topics

updating last posted at on users

updating last seen at on users

Updating topic reply counts...
    15987 / 15985 (100.0%)  [6356 items/min]  .Updating first_post_created_at...
Updating user post_count...
Updating user topic_count...

updating featured topic users
      192 / 192 (100.0%)
updating featured topics in categories
      186 / 186 (100.0%)
updating user topic reply counts
    15985 / 15985 (100.0%)  
resetting topic counters
      192 / 192 (100.0%)

(Gareth Williams) #19

Apologies for the blast - I think it might be the topics import, will try adjusting the numbers (i am hacking away with the SQL now to get it right)

will report back.


(Gareth Williams) #20

Hey @michaeld

Looks like we have success - the topics are importing now.

I am on vBullettin cloud so I wonder if this might help a load of people, happy to post my importer script, or can make it more general to not look at the id but the contenttype name if this might assist?

Now, onto the 301 redirects as we have a lot of content I found this :slight_smile:

However, Its a bit over my head - has anyone had any success with 301 for vBullettin 5 (especially cloud) Here is the forum i will be redirecting: