Migrate a XenForo forum to Discourse

Just 5k posts, and about 1k members

I’m still waiting for big board importer, to try and test my 18m post forum.

For Xenforo? We do have other bulk imports but it’s true this would be very slow for 18m posts!

We are working on our own in-house bulk importer for 27 million posts. It went from a bit over a week (not counting attachments) to under a day with everything. We completed our first import test without any errors yesterday. Really exciting stuff.

Thanks Justin, I’ve had a quick look at it - does this seem ok to you? (Feel free to add it to the official importer if you like)

  XENFORO_DB = "xenforo_db_3"
  TABLE_PREFIX = "xf_"
  BATCH_SIZE = 1000
  ATTACHMENT_DIR = '/FULL/PATH/TO/attachments'
  AVATAR_DIR = '/FULL/PATH/TO/avatars'

(Added last line^^ - means you’ll need to copy avatars there)

  def execute
    import_users
    import_categories
    import_posts
    import_avatars
  end

(Added last line^^)

  def import_avatars
    if AVATAR_DIR
      users = User.all
      users.each do |u|
        unless u.custom_fields["import_id"].nil?
          import_id = u.custom_fields["import_id"]
          if import_id.to_i < 1000
            dir_num = "0"
          elsif import_id.to_i > 1000
            dir_num = import_id.first
          end
        
          avatar_filename = "#{import_id}.jpg"
          file_path = "#{AVATAR_DIR}/l/#{dir_num}"
          file_path_and_name = "#{file_path}/#{avatar_filename}"

          if File.exists?(file_path_and_name)
            upload = create_upload(u.id, file_path_and_name, avatar_filename)
            if upload.persisted?
              u.import_mode = false
              u.create_user_avatar
              u.import_mode = true
              u.user_avatar.update(custom_upload_id: upload.id)
              u.update(uploaded_avatar_id: upload.id)
            else
              puts "Error: Upload did not persist for #{u.username} #{avatar_filename}!"
            end
          end
        end
      end
    end
  end

Its very late here so I may have made some mistakes or missed a load of stuff out but this is assuming all avatars are .jpg (which seems like all of mine are). I wasn’t too sure about the u.import_mode switches tho so just commented them out.

Totally untested (it.s nearly 5am here )

I did a test on my dev machine on a forum with 100K posts and it took 90minutes. The one I want to do the import on has a couple million posts so 10 times as long maybe?

Nice! Will you be sharing it with us? Any idea when?

The goal currently is to make sure that it is working perfectly and then once our site is migrated, to eventually push it to the Discourse repo so others may use it.

A step by step guide would be very helpful also, please :).

For the 5k posts, and about 600 attachments, on an older xeon and an SSD disk it took about 10 mins. I would grab a reasonably powered box, do the import offline, and let it chug away.

Fantastic! Looking forward to it.

Ok I’ve done a very quick test and the additions to the script seem to have worked - doing a proper test now with a few thousand members and 100K posts. Will report back if it works…

Hi, anything new? If you need help with testing, I will be glad to help

I’m getting an error that users table doesn’t exist when I run the xenforo script. Any help really aprreciated!

Reading about this for an hour but haven’t figured it out yet.

table_dont_exist1522×351 66.5 KB

There are some variables at the top of the script that you need to fill in so that it can connect to your xenForo MySQL database and query the data. I believe “import_db” might be the placeholder value - double check that those values are filled out correctly.

@Ghan I thought I’m importing to import_db with this: mysql -u root -p import_db < /shared/db.sql

My xenforo.rb is:

xenfororb912×448 88.1 KB

Ah ok. If that is your database name, then check to make sure you have the table name correct. Your table might be xf_user with the prefix, and that would cause an issue. There should be a TABLE_PREFIX variable in the script as well.

@Ghan Thanks you!

Right now, I’m still stack on the import script.
It iterates through users and categories all good. On the import_posts function though it just hangs there until after some hours it times out.

Starting debugging this by just printing the values in the import_posts:

Screenshot from 2021-02-15 16-09-222048×1125 145 KB

All prints after the assignment of the results variable never print.

asdfasdfasdfasdf749×496 24.2 KB

Just in case anyone is looking to import a vb3 forum, this guide might help if you also have a Xenforo licence:

Does this importer import tags from xenforo along with the threads they are attached to?

Always a good idea to look at the source code.

I think the answer is … yes (but not super familiar with Xenforo data structures):

So, it looks like it at least once imported some tags for someone under some conditions. If you’re lucky, it’ll work for you too!