Import scripts vbulletin5.rb

Installation
1

I am using the official import script for vBulletin v5 and I ran into an issue.

My vBulletin forum structure is like:

/forum/
/forum/sub-category1/
/forum/sub-category1/thread1/
/forum/sub-category1/thread2/
/forum/sub-category2/
/forum/sub-category2/thread1/
/forum/sub-category2/thread2/

When I run the import script this results in two categories in Discourse without any threads:

/c/sub-category1
/c/sub-category2

I checked out a different vBulletin forum and noticed that they do not nest their URLs. I now assume that this is the change that I have to make to my Forum before I am able to use the script:

/forum/
/forum/sub-category1/
/forum/sub-category2/
/forum/thread1a/
/forum//thread2a/
/forum/thread1b/
/forum//thread2b/

Afterwards, I would have to manually assign categories to every thread again.

Does someone have experience using this script?

Is there maybe something that jumps to your eyes inside the script, that can be changed so that it can handle a more nested structure? I might be on a wrong path completely.

2

By “thread” do you mean discourse topic? Are the topics being created but not in the correct categories?

3

Hey Jay,

thank you for your reply.

Ah yes - I mean Discourse Topics. There is nothing there except those empty categories. And all the users are imported successfully.

Mike

Untitled
Untitled518Ă—597 12.3 KB

4

Something is keeping topics and posts from getting imported. I’d think that you’d see errors when it ran. Topics and posts get created after users. I believe that topics and posts are in separate functions. The importer should show you progress on those just like it did for the users.

1 Like
5

There are no Error messages during the import - only a few warnings for invalid user email addresses when the create_users function runs. But the user import is successful in the end. Here is the log (I removed the warnings as they contained customer email addresses):

https://gist.github.com/mpolinowski/ae6447b5b95a412856c51aed6556ba84

importing posts...
5 / 5 (100.0%)  [290527 items/min]                                                            
importing attachments...   
20182 / 5 (403640.0%)

Here is where I think it only sees my five categories as posts and imports them. And does not know what to do with all the attachments for the actual topics that weren’t imported?

Disclaimer:

Initially, the import failed because the vBulletin database I was given did not have a table customprofilepic - I added an empty table here.

And there were no custom avatars - so the AVATAR_DIR, referenced in the import script, is just an empty folder I created. I commented out the following lines in the import script that were referencing those assets:

create_users(users, total: user_count, offset: offset) do |user|
        username = @htmlentities.decode(user["username"]).strip
        {
          id: user["userid"],
          name: username,
          username: username,
          email: user["email"].presence || fake_email,
          admin: user['admin'] == 1,
          password: user["password"],
          website: user["homepage"].strip,
          title: @htmlentities.decode(user["usertitle"]).strip,
          primary_group_id: group_id_from_imported_group_id(user["usergroupid"]),
          created_at: parse_timestamp(user["joindate"])
          # post_create_action: proc do |u|
          #   @old_username_to_new_usernames[user["username"]] = u.username
          #   import_profile_picture(user, u)
          #   import_profile_background(user, u)
          # end
        }
      end

But to my understanding, those changes only affect the user creation. And the users are created successfully.

This is the full import script that I am using:

https://gist.github.com/mpolinowski/0ef4df888720ad4a92c4bf5edac2e076

1 Like
6

The script collects all topics from the node table within the vBulletin database. Which is present and holding about 34k postings. The column names are identical to what the script is referencing:

https://gist.github.com/mpolinowski/2d72874a41eb7b8dbbd2afe2da4ad279

It looks as if it should just work :roll_eyes:

7

The import script searches for contenttypeid=23 and contenttypeid=22 to create postings.

By skimming through the node table I can say, that the 23 stands for a posting with an attachment and 22 should be the attachment itself.

The majority of the posts are of contenttypeid=21 - postings without attachments. And I don’t see an import for those. Or is this covered by:

post_count = mysql_query("SELECT COUNT(nodeid) cnt FROM #{DBPREFIX}node WHERE parentid NOT IN (
SELECT nodeid FROM #{DBPREFIX}node WHERE contenttypeid=23 ) AND contenttypeid=22;").first["cnt"]

This goes through everything that is left over after the import is done with 23/22. But there are more content type IDs beside 23/22/21 - so this would get those as well.

And categories are imported relative to a ROOT_NODE that was preset to 2. I might have to set it to 3 to support my “3-level nested” URLs?

UPDATE: I just re-ran the import with ROOT_NODE=3 and and it did not change anything

1 Like
8

It sounds like your vBulletin is different from what the script expects. Before you can get the posts to import, you’ll need to figure out why none of the topics got created.

1 Like
9

Ok. It is the contenttypeid that is causing the issue:

def import_categories
    puts "", "importing top level categories..."

    categories = mysql_query("SELECT nodeid AS forumid, title, description, displayorder, parentid
	      FROM #{DBPREFIX}node
          WHERE parentid=#{ROOT_NODE}
        UNION
          SELECT nodeid, title, description, displayorder, parentid
          FROM #{DBPREFIX}node
          WHERE contenttypeid = 20
            AND parentid IN (SELECT nodeid FROM #{DBPREFIX}node WHERE parentid=#{ROOT_NODE})").to_a

For example the import_categories function uses contenttypeid = 23 which is the same ID the script uses for postings. The ID for categories is 20. By changing this inside the script I am finally able to see my sub-categories / topics :raised_hands:

Untitled
Untitled526Ă—673 17.9 KB

Now all that is missing are the threads / posts

2 Likes
10

I have made a pull request with some updates to the VBulletin5 importer. One of the most important changes is that it got rid of these hard coded ID’s.

There is also better support for uploads and the importer now imports tags and creates permalinks for topics and (sub)categories.

4 Likes