Migrate a vBulletin 4 forum to Discourse

Hi. Just want to share my solution.
As for the problems with quotes. As I said before, i’m was facing with problems - regexp not capture quotes then:

  • username and post id may be enclosed in double quotes
  • nested quotes

I decided to do a search and replace using a different logic. Instead of searching for tags and their contents, I used a regular expression that searches only for tags:

was:
raw.gsub!(%r{\[quote="?([^;]+);(\d+)"?\](.+?)\[\/quote\]}im) do

became:
raw.gsub!(%r{(\[QUOTE(="?([^;]+);(\d+)"?)?\])|(\[\/QUOTE\])}im) do

and then little change the determining the source of a quote:

      if $3 && $4
        if topic_lookup = topic_lookup_from_imported_post_id(post_id)
          post_number = topic_lookup[:post_number]
          topic_id = topic_lookup[:topic_id]
          "\n[quote=\"#{new_username},post:#{post_number},topic:#{topic_id}\"]\n"
        else
          "\n[quote=\"#{new_username}\"]\n"
        end
      elsif $5
        "\n[/quote]\n"        
      end

Also, I change the spoiler’s code. Instead of:

    # [spoiler=Some hidden stuff]SPOILER HERE!![/spoiler]
    raw.gsub!(%r{\[spoiler="?(.+?)"?\](.+?)\[/spoiler\]}im) do
      "\n#{$1}\n[spoiler]#{$2}[/spoiler]\n"
    end

that blured text, I convert it to details tag:

    raw.gsub!(%r{(\[spoiler(="?(.*?)"?)?\])|(\[\/spoiler\])}im) do
      if $3
        "\n[details=#{$3}]\n"
      elsif $1
        "\n[details]\n"
      elsif $4
        "\n[/details]\n"
      end
    end

Because it just so happens that in the vbulletin world - sploiler it is not the blurred content, but rather the collapsed content. So I think it is much more appropriate for the vbulletin import script to convert spoilers to details instead of the blurred spoiler.

I also noticed the mention tag. In my case, in vbulletin, the mentions looked like this:
[mention=XXX]username[/mention]

The regular expression used in the script does not take into account that the tag may contain the user id.

    # [MENTION]<username>[/MENTION]
    raw.gsub!(%r{\[mention\](.+?)\[/mention\]}i) do
      new_username = get_username_for_old_username($1)
      "@#{new_username}"
    end

I fixed this in my own way too:

    # [MENTION]<username>[/MENTION]
    raw.gsub!(%r{\[mention(=\d+)?\](.+?)\[/mention\]}i) do
      new_username = get_username_for_old_username($2)
      "@#{new_username}"
    end
1 Like

This is not true when I started migrating my 24 year old vBulletin forum running vB 3. There were multiple incompatibilities and other issues with the script. However, I put in a lot of effort in creating an importer for vBulletin 3 based on the the script for vB4.

The improved script is included with Discourse, it is called vbulletin3.rb. Usage of the vB3 import script is the same as described in this how-to. Just execute bundle exec ruby script/import_scripts/vbulletin3.rb instead.

The vBulletin3 has some significant changes/improvements:

  1. Forum permissions are copied
  2. Forum moderator groups are created
  3. Joinable user groups are created with proper configuration
  4. Forum nesting in imported up to 3 levels deep (maximum of Discourse)
  5. Permalinks are registered for all threads and posts, preventing link rot
  6. Some basic forum settings are copied over (e.g. title, notification email, company name)
  7. Polls are imported
  8. Major improvements to the bbcode → markdown conversion
  9. URL deep links to threads, post, attachments are converted to discourse references, this requires setting the environment variable FORUM_URL to forum.hostname/path (no protocol).

Instead of trying to convert vBulletin private messages to Discourse private messages users will instead receive a system private message containing an archive of the private messages they had. vBulletin’s PM construction is not really compatible with Discourse. Trying to convert it would also expose some privacy depending how people used PMs in vBulletin.

As it is probably also the case with other importers, it can take quite a bit of time to convert. The conversion script took 5.5 hours on my workstation for 7k users, 16k threads, 415k posts. I have no idea how much time it took for the post processing, I let that run over night. From start to end the forum was down for 30 hours. In the end I’m happy with the result.

2 Likes

Now that’s a throwback :slight_smile:

Your forum looks very nice. I like the alternating colors on the topic rows.

It seems both this thread and the importer are well out of date at this point. I’ve fixed a few problems with help from this thread, but I’m stuck now on the following on the user import step, anyone know how to fix it?

<internal:timev>:286:in at’: can’t convert NilClass into an exact number (TypeError)`

Either the query is wrong or somehow the table is missing a value

It’s rather bizarre to respond to this so many years later, but I’m doing a VB import now with the bulk importer and a bunch of images were missing and the reason is that they moved the attachment filename to a different field.

 SELECT a.filedataid attachment_id, a.userid user_id, a.filename filename
             FROM attachment a
            WHERE a.attachmentid = 383075;

the NUMBER.attach file is now the filedataid field, not the attachment_idfield. So that query needs to be updated in the script.

I have been asked to migrate a vbulletin 4.25 forum to discourse….reading this thread is giving me mixed feelings…it looks like it’s possible, but potentially a massive pain and time drain (both of which I could do with avoiding at the moment)….

is there an updated script for vbulletin 4.25 anywhere? I can only see 3 and 5 on the official page?

Well, there are vbulletin and vbulletin5 scripts in the bulk imports directory that are only a couple months old. Running those bulk imports is pretty tricky, and not well documented.

I’ve done over 100 imports and I’m pretty sure that I’ve never done one that didn’t require tweaking the script for one reason or other.

I wrote several import scripts before I really even learned Ruby (But a professor in the mid 1980s assured me that after his Programming Languages class I could say that I knew any language if I had a weekend and a book; he was mostly right.)

But yeah, it is likely to be at least as much of a pain and time drain as you imagine.

I suspect that the vbulletin script will do a decent job; I probably wouldn’t recommend the bulk import script unless you have over a million posts + users.

1 Like

thanks for that :slight_smile:

guess I need to decide what I’m doing with my weekend :smiley:

1 Like

I would really appreciate some help with this :face_with_spiral_eyes:

I’ve read through the tread, and I’ve followed some steps I’ve seen here, but am getting stuck.

  1. I ssh on to my vps
  2. go into the docker image
  3. install mariadb-server
  4. try to run the mysql command to create a db, but get ‘can’t connect to local server through socket’

the instructions on here are from a few years back - I se a couple of people got the same error, but I don’t see any resolutions to it

has anyone done this process recently and can point me in the right direction - I don’t think there is a step by step guide I’m missing anywhere, is there?

edit: just to add, I’ve tried all kind of shenanigans and ended up installing a docker image of mariadb on the local host (not docker) and then exposing the port - I can now connect to the db from the docker image…but running the script gives: Gemfile: Undefined local variable or method ‘mysql2’ I’ve tried to install the gemfile, but it fails…and before I troubleshoott further, I jsut get the feeling that I’m using outdated information and potentially old packages…I’m jsut very confused and could do with some guidance!

any help appreciated!

…I persisted and finally got as far as the script running! However, after starting to execute it fails with:

"root@vps-xxxxxxxx-app:/var/www/discourse/script/import_scripts# bundle exec ruby vbulletin.rb
/var/www/discourse/config/initializers/013-excon_defaults.rb:4:in `<main>': can't modify frozen Hash: {:chunk_size=>1048576,                                                             :ciphers=>"ECDHE-ECDSA- [................]"

…and I think I’ve about reached my limit of troubleshooting ability

I don’t know if starting it from /var/www/discourse makes a difference, but I always do. Here’s what is recommended in the OP.

I don’t recognize that error.

1 Like

By the point I gave up last night, after hours of “research”, I had tinkered with so many things outside my comfort zone I was worried that I was causing more harm than good. I was also keen to hear if someone had done it recently and there was some magic step I was missing, rather than spiral down recurring rabbit holes :winking_face_with_tongue:

It’s possible that there’s some new issue. I ran an mbox import the other day, and I imagine that it calls that same code that gave you the error.

Oh. I did it in a container and these instructions are for a development environment. Did you set up a development environment? Did you get it working? That’s tricky on a vpn.

I’m going to wipe and try again tomorrow, migraine tonight.

I’ll let you know how it goes :slightly_smiling_face:

1 Like

Did you set up a development environment? Was that your first step?

I tried doing it through the host vps, and later bodging it through the container. By the time I finished I wasn’t sure what I did where. It’s likely a skill issue… I’ll start again from fresh tomorrow

If you’re doing it on a vm the doing it in a container is probably easier. I would look at some other import how-to examples (they’ll include the mysql template).

You’ll enter the container, maybe edit the Gem file and bundle install.

Using a container for mysql or Mariadb probably makes sense. (but then you’ll need to make sure that the containers can see each other)

1 Like

I have some exciting news…

I’ve got the script running (I’m going to make a guide once it’s all done), it’s imported usergroups, but is stuck on importing users, I think to do with the timezone variable?

importing users
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/tzinfo-2.0.6/lib/tzinfo/timestamp.rb:138:in `for': Integer values are not supported (ArgumentError)

            raise ArgumentError, "#{value.class} values are not supported" unless is_time_like?(value)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        from /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/tzinfo-2.0.6/lib/tzinfo/timezone.rb:575:in `utc_to_local'
        from script/import_scripts/vbulletin.rb:1019:in `parse_timestamp'
        from script/import_scripts/vbulletin.rb:166:in `block (2 levels) in import_users'
        from /var/www/discourse/script/import_scripts/base.rb:267:in `block in create_users'
        from /var/www/discourse/script/import_scripts/base.rb:266:in `each'
        from /var/www/discourse/script/import_scripts/base.rb:266:in `create_users'
        from script/import_scripts/vbulletin.rb:148:in `block in import_users'
        from /var/www/discourse/script/import_scripts/base.rb:951:in `block in batches'
        from <internal:kernel>:187:in `loop'
        from /var/www/discourse/script/import_scripts/base.rb:950:in `batches'
        from script/import_scripts/vbulletin.rb:126:in `import_users'
        from script/import_scripts/vbulletin.rb:82:in `execute'
        from /var/www/discourse/script/import_scripts/base.rb:47:in `perform'
        from script/import_scripts/vbulletin.rb:1027:in `<main>'

The default in the script is “America/Los Angeles” - vBulletin forum is set to (GMT) Western Europe Time, London, Lisbon, Casablanca) and the discourse instance I’m importing into has two entries: America/Los Angeles and Europe/Paris

Do you know which I should pick?

It doesn’t matter much which you pick.

I think it might be that the date isn’t stored the way that the script expects.

1 Like