Migrate a XenForo forum to Discourse

It is not. You need permalinks created for each user, post, topic, and category. See admin -> customize -> permalink. Many other importers have support for creating permalinks.

2 Likes

This method work with xenforo version 2.x ?

Yes, I’m on the latest version of XF and thus generally works for it.

Just gonna ask you guys why you move from xenforo to discourse? Im just started a forum a week now and i’ve seem xenforo runned websites and im impressed because of their plugins including pointing system. I just wanted the cons or reasons of moving. Thankyou

It looks like the xenforo script handles “gallery” and “attachments”, whatever that means.

It does not create permalinks to rewrite URLs.

If you’re a programmer, you can look at some other importers (say, vbulletin) for an example of how to create permalinks. I’d probably write the code for $500 for topics, posts, and categories, though someone else might have a better offer if you ask in #marketplace.

1 Like

Just finished up a Xenforo → Discourse migration of a million post board on a Digital Ocean droplet. Here’s what worked for me command by command (very similar to what also just did on a different vbulletin board).

Recommend minimum 4 vCPU/8GB for import.

Thanks to everyone in this thread for helping me get through these migrations, definingly been a fun few days reading and rereading…

1 - Install Digital Ocean Discourse 1-click droplet

2 - Finish discourse install through SSH by following prompts

Open SSH console
root
(yourrootpassword)
(enter)
(yourdomain).com
etc

3 - Login to SFTP

sftp root@XXX.XXX.XX.XX
y (if asked for confirmation)
yes
(yourrootpassword)
put db.sql /var/discourse/shared/standalone/db.sql

4 - Login to Website to setup Admin account

5 - Login to SSH - Begin Process

ssh root@XXX.XXX.XX.XX
cd /var/discourse
./launcher start app
docker exec -it app bash
sudo apt-get update
sudo apt-get upgrade
y

6 - Install MariaDB (replacement for mysql)

apt-get update && apt-get install mariadb-server-10.3 libmariadbd-dev
y

7 - Mysql Database Setup

service mysql start
mysql -u root -p
password
create database import_db;
exit;

8 - Import Dump -> Mysql Database Transfer**

mysql -u root -p import_db < /shared/db.sql
password

9 - GEM File

echo "gem 'mysql2'" >>Gemfile
echo "gem 'mysql2', require: false" >> /var/www/discourse/Gemfile
echo "gem 'php_serialize', require: false" >> /var/www/discourse/Gemfile
cd /var/www/discourse
su discourse -c 'bundle install --no-deployment --without test --without development --path vendor/bundle'
(ignore red text result)

10 - Configure install script

vi /var/www/discourse/script/import_scripts/xenforo.rb

---Make edits to text file as needed for db name/password, prefix, etc---

(esc)
:wq

11 - Bundle Config

bundle config set path 'vendor/bundle'
bundle config set without 'development:test'
bundle config unset deployment
su discourse -c 'bundle install'

12 - Mysql config (may be possible to do this with previous)

mysql --version
sudo mysql -u root -p
password
ALTER USER 'root'@'localhost' IDENTIFIED BY 'password';
FLUSH PRIVILEGES;
exit

13 - Install Script

su discourse -c 'bundle exec ruby script/import_scripts/xenforo.rb'
4 Likes

I am in the process of trying to get this migration complete, however it seems to be throwing an error with the attachments.

RAILS_ENV=production bundle exec ruby script/import_scripts/xenforo.rb
Loading existing groups...
Loading existing users...
Loading existing categories...
Loading existing posts...
Loading existing topics...

creating users
Skipping 407 already imported users

importing categories...
       12 / 12 (100.0%)  [344731 items/min]
creating topics and posts
        2 / 4554 (  0.0%)  [17724 items/min]  Traceback (most recent call last):
        18: from script/import_scripts/xenforo.rb:396:in `<main>'
        17: from /var/www/discourse/script/import_scripts/base.rb:47:in `perform'
        16: from script/import_scripts/xenforo.rb:32:in `execute'
        15: from script/import_scripts/xenforo.rb:174:in `import_posts'
        14: from /var/www/discourse/script/import_scripts/base.rb:866:in `batches'
        13: from /var/www/discourse/script/import_scripts/base.rb:866:in `loop'
        12: from /var/www/discourse/script/import_scripts/base.rb:867:in `block in batches'
        11: from script/import_scripts/xenforo.rb:180:in `block in import_posts'
        10: from /var/www/discourse/script/import_scripts/base.rb:508:in `create_posts'
         9: from /var/www/discourse/script/import_scripts/base.rb:508:in `each'
         8: from /var/www/discourse/script/import_scripts/base.rb:509:in `block in create_posts'
         7: from script/import_scripts/xenforo.rb:186:in `block (2 levels) in import_posts'
         6: from script/import_scripts/xenforo.rb:315:in `process_xenforo_post'
         5: from script/import_scripts/xenforo.rb:324:in `process_xf_attachments'
         4: from /usr/local/lib/ruby/2.7.0/set.rb:328:in `each'
         3: from /usr/local/lib/ruby/2.7.0/set.rb:328:in `each_key'
         2: from script/import_scripts/xenforo.rb:326:in `block in process_xf_attachments'
         1: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/core_ext/string/filters.rb:22:in `squish!'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/core_ext/string/filters.rb:22:in `gsub!': can't modify frozen String: "\\t\\tSELECT a.attachment_id, a.data_id, d.fil\n\\t\\tFROM xf_attachment AS a\\n\\t\\tINNER JOIN xf_attachment_data d ON a.data_id = d.data_id\\n\\t\\tWHERE attachment_id = 13\\n" (FrozenError)


Does anyone have any thoughts on what could be going on here?  What is the "FrozenError"?

It looks like I had to edit the xenforo.rb file and change

# frozen_string_literal: true

to

# frozen_string_literal: false

2 Likes

Thanks for writing this guide!

Does the importer also import avatars and attachments? Do I simply copy them to the /tmp/attachments folder?

From a quick look at the importer, it appears it only imports uploads to posts. Avatar support would need to be added.

2 Likes

Can confirm. It only imports uploads. Avatars are not included. I just completed a Xenforo migration last week.

3 Likes

Board size?
Post/threads/members.
Congrats :slight_smile:

1 Like

Just 5k posts, and about 1k members

2 Likes

I’m still waiting for big board importer, to try and test my 18m post forum.

2 Likes

For Xenforo? We do have other bulk imports but it’s true this would be very slow for 18m posts!

3 Likes

We are working on our own in-house bulk importer for 27 million posts. It went from a bit over a week (not counting attachments) to under a day with everything. We completed our first import test without any errors yesterday. Really exciting stuff.

3 Likes

Thanks Justin, I’ve had a quick look at it - does this seem ok to you? (Feel free to add it to the official importer if you like)

  XENFORO_DB = "xenforo_db_3"
  TABLE_PREFIX = "xf_"
  BATCH_SIZE = 1000
  ATTACHMENT_DIR = '/FULL/PATH/TO/attachments'
  AVATAR_DIR = '/FULL/PATH/TO/avatars'

(Added last line^^ - means you’ll need to copy avatars there)

  def execute
    import_users
    import_categories
    import_posts
    import_avatars
  end

(Added last line^^)

  def import_avatars
    if AVATAR_DIR
      users = User.all
      users.each do |u|
        unless u.custom_fields["import_id"].nil?
          import_id = u.custom_fields["import_id"]
          if import_id.to_i < 1000
            dir_num = "0"
          elsif import_id.to_i > 1000
            dir_num = import_id.first
          end
        
          avatar_filename = "#{import_id}.jpg"
          file_path = "#{AVATAR_DIR}/l/#{dir_num}"
          file_path_and_name = "#{file_path}/#{avatar_filename}"

          if File.exists?(file_path_and_name)
            upload = create_upload(u.id, file_path_and_name, avatar_filename)
            if upload.persisted?
              u.import_mode = false
              u.create_user_avatar
              u.import_mode = true
              u.user_avatar.update(custom_upload_id: upload.id)
              u.update(uploaded_avatar_id: upload.id)
            else
              puts "Error: Upload did not persist for #{u.username} #{avatar_filename}!"
            end
          end
        end
      end
    end
  end

Its very late here so I may have made some mistakes or missed a load of stuff out but this is assuming all avatars are .jpg (which seems like all of mine are). I wasn’t too sure about the u.import_mode switches tho so just commented them out.

Totally untested (it.s nearly 5am here :zzz:)

I did a test on my dev machine on a forum with 100K posts and it took 90minutes. The one I want to do the import on has a couple million posts so 10 times as long maybe?

Nice! Will you be sharing it with us? Any idea when?

3 Likes

The goal currently is to make sure that it is working perfectly and then once our site is migrated, to eventually push it to the Discourse repo so others may use it.

4 Likes

A step by step guide would be very helpful also, please :).

For the 5k posts, and about 600 attachments, on an older xeon and an SSD disk it took about 10 mins. I would grab a reasonably powered box, do the import offline, and let it chug away.

4 Likes