Migrate a XenForo forum to Discourse

It probably works for some people with some version and some configuration, just not that one you’re using. It’s surprising how different each import is.

Is the xenforo official import script broken now?

Users and categories import fine though,

Importing posts with attachment folder getting issues

script/import_scripts/xenforo.rb:180:in `block in import_posts'
/var/www/discourse/script/import_scripts/base.rb:908:in `block in batches'
/var/www/discourse/script/import_scripts/base.rb:907:in `loop'
/var/www/discourse/script/import_scripts/base.rb:907:in `batches'
script/import_scripts/xenforo.rb:174:in `import_posts'
script/import_scripts/xenforo.rb:32:in `execute'
/var/www/discourse/script/import_scripts/base.rb:47:in `perform'
script/import_scripts/xenforo.rb:396:in `<main>'
   113289 / 146471 ( 77.3%)  [3766 items/min]  Exception while creating post 162516. Skipping.
undefined method `synchronize' for #<DiscourseRedis:0x00005571b2d0c5b8>
/var/www/discourse/lib/discourse_redis.rb:44:in `method_missing'
/var/www/discourse/lib/distributed_mutex.rb:84:in `try_to_get_lock'
/var/www/discourse/lib/distributed_mutex.rb:61:in `get_lock'
/var/www/discourse/lib/distributed_mutex.rb:30:in `block in synchronize'
/var/www/discourse/lib/distributed_mutex.rb:29:in `synchronize'
/var/www/discourse/lib/distributed_mutex.rb:29:in `synchronize'
/var/www/discourse/lib/distributed_mutex.rb:14:in `synchronize'
/var/www/discourse/lib/post_creator.rb:407:in `transaction'
/var/www/discourse/lib/post_creator.rb:205:in `create'
/var/www/discourse/script/import_scripts/base.rb:580:in `create_post'
/var/www/discourse/script/import_scripts/base.rb:528:in `block in create_posts'
/var/www/discourse/script/import_scripts/base.rb:515:in `each'
/var/www/discourse/script/import_scripts/base.rb:515:in `create_posts'
script/import_scripts/xenforo.rb:180:in `block in import_posts'
/var/www/discourse/script/import_scripts/base.rb:908:in `block in batches'
/var/www/discourse/script/import_scripts/base.rb:907:in `loop'
/var/www/discourse/script/import_scripts/base.rb:907:in `batches'
script/import_scripts/xenforo.rb:174:in `import_posts'
script/import_scripts/xenforo.rb:32:in `execute'
/var/www/discourse/script/import_scripts/base.rb:47:in `perform'
script/import_scripts/xenforo.rb:396:in `<main>'
   113290 / 146471 ( 77.3%)  [3766 items/min]  Exception while creating post 162517. Skipping.
undefined method `synchronize' for #<DiscourseRedis:0x00005571b2d0c5b8>
/var/www/discourse/lib/discourse_redis.rb:44:in `method_missing'
/var/www/discourse/lib/distributed_mutex.rb:84:in `try_to_get_lock'
/var/www/discourse/lib/distributed_mutex.rb:61:in `get_lock'
/var/www/discourse/lib/distributed_mutex.rb:30:in `block in synchronize'
/var/www/disc

What version of Redis do you have?

I just barely managed to limp my existing Xenforo community into discourse, so I thought I’d share my experience here to help anyone trying to do the same. Keep in mind I don’t know what I’m doing, so some of my inferences may be incorrect. I was able to complete the import to discourse with around a 20% loss in attachments. I can’t really explain why, but the issue seems only to effect the past 2-3 years of posts on my forum, whereas everything before that had around a 95+% successful import rate. My only guess is that is has something to do with the move from XF 1.x to XF 2.x.

Anyone attempting this should know that this script needs a lot of updating to work with XF 2.x. Specifically, lines 375-378 do not have the correct database tables for importing attachments from the Xenforo Media Gallery. I got around this by disconnecting the XFMG from the forum and uninstalling it, since I was only using the XFMG to mirror attachments from the forum anyways.

If you’re curious, line 376 refers to the table xengallery_media, which is obviously a reference to XFMG 1.x, as this table has been changed to xf_mg_media_item in XFMG 2.x. The rest of it I can’t get to match up and I suspect there’s some extra code needed to match up the IDs of attachments to posts.

I ended up commenting out line 314: s = process_xf_attachments(:gallery, s)

So the script wouldn’t error out on the missing SQL table.

So, with that done, I followed @Jack51’s post in this thread:

with the following edits:

step six needs the mariadb version updated. change mariadb-server-10.3 to mariadb-server-10.5

step seven should be service mariadb start instead of service mysql start.

I did change

frozen_string_literal: true

to

frozen_string_literal: false

Since I got the same error someone else mentioned in this thread.

It seems there was a LOT of confusion in this thread about where to place the attachments so the script would see them, and I have to admit I had some difficulty with this as well. For folks completely new to docker (like myself), it would be extremely useful to point out that while dockers are essentially a virtual machine running on your server (so you’re dealing with two different OSes/file systems), there is a shared space for transferring files:

/var/discourse/shared/standalone/tmp

this directory is located on your host machine, and files put there will appear in:

/shared/tmp

when you’ve entered the docker container. So, assuming you’re retaining the same file/folder structure from xenforo, you would copy the folders:

internal_data/attachments

to

/var/discourse/shared/standalone/tmp

on your host machine, and that folder would appear in /shared/tmp after entering your docker container.

Next, when editing the xenforo.rb script, change:

ATTACHMENT_DIR = '/tmp/attachments'

to

ATTACHMENT_DIR = '/shared/tmp/internal_data/attachments'

and you will have your attachments located in the correct location. It’s worth noting that xenforo also has a data/attachments folder, but it’s filled with .jpg files instead of the .data files you’re looking for. I saw someone made this mistake previously in this thread so I figured it was worth pointing out.

Also, I got a permissions error from the script, so you might have to chmod the directory:

sudo chmod a+rw -R /shared/tmp/internal_data/

I’d say the biggest giveaway that the script isn’t picking up on your attachments is that it will be strangely quiet while instead it should probably be putting out the occasional error:

So if the import goes several hours without a single error listed, it’s probably not picking up the attachments.

That’s all I have. Don’t get me wrong, I’m glad this script exists at all, 80% success rate is better than nothing (I had a similar success rate when going from vBulletin to Xenforo in 2009), and I’m excited to get working with the Discourse platform. Hopefully this post helps some folks and maybe someday someone with better skills than myself can update this script to work a little better.

2 Likes

The current version of the import script missed a lot of attachments because they’re not always embedded into the post BBcode. I’ve made a pull request with quite a few improvements and fixes. The script now sees all attachments and also imports likes, avatars and PM’s, and it creates permalinks for categories and topics.

5 Likes

@gerhard I think that PR is on your plate at the moment, could you please merge when you have some time?

2 Likes

Yes, I noticed that yesterday. :blush: I had completely forgotten about it. Will review it today or next week.

1 Like