Is it safe to set up my trust levels, badges, other discourse settings before the import? I’m thinking this importer will just insert/replace posts/threads/profiles? I’d hate to get everything set up in discourse settings and have this wipe out those settings, even though I don’t immediately see anything in the code related to that.
Yes, but make a backup. You will likely need to change things and run it multiple times, starting over from scratch,so keep a backup of your configuration with no imported data.
It inserts. It mostly doesn’t replace anything. If a user exists, it won’t change it. It won’t use categories that you set up. If you run it again, with an updated database from your old site, it’ll add new data, but not change things that were imported on earlier runs. This makes it much faster, but if you need to change the script for some reason you’d need to delete it all and start again.
4 Likes
So, two things. First, is there a way to make this run faster? I’m on a 32 CPU Linode with 64GB of ram and it’s still taking 4-5 hours to import 250k posts.
Second, does it import attachments before or after the posts? Because looking at the script it looks like it does it after. It’s just a bummer to have to wait 4-5 hours just to find out the attachments didn’t import.
Everything is dependent on the data before it,for the most part, the post-processing can take advantage of the multiple cpus, but that doesn’t start until you’re done, for the most part. Get a single cpu that’s faster.
You can modify the script to import only recent uploads, but if you’re not planning on doing major development, if just wait the 5 hours.
2 Likes
Well, I’ve tried this several times now and the attachments aren’t getting imported, and I’m running out of ideas. I’m hoping someone can help me (thanks @pfaffman for your input so far).
I’ve placed the *.data files in their original folder structure from xenforo (so, the ‘attachments’ folder) in /root/attachments in the docker container. I’ve updated the import script to /root/attachments. I can see the files in /root/attachments on the command line with ls.
I’ve run the script, and every time (after about a 4-5 hour wait), all posts with attachments/photos show the old bbcode from xenforo ([ATTACH]123456[/ATTACH]) and I don’t see any pictures or attachments in the discourse install.
What could I be doing wrong here? I feel like I’m 90% there, I’m just missing something.
You’ll have to look at the code. Is the attachment code even running? Are your attachments on the format the script expects?
If the attachment function is run after the post import then you don’t need to run the whole thing again, it’ll just skip over the posts (or comment out the post import to make it a bit faster so).
But there are a bunch of things that could be wrong from you not configuring it correctly to it just not working with your version of zenforo. You’ll have to look at the code and add some debugging output to see what’s going on.
Well, just FYI for anyone coming across this thread, I was not able to get the attachments to work with this script, no matter where I placed them. IMO the script needs work and the instructions especially need a lot of work to be more clear about what goes where and how.
Until then, I would consider this script to be broken.
It probably works for some people with some version and some configuration, just not that one you’re using. It’s surprising how different each import is.
Is the xenforo official import script broken now?
Users and categories import fine though,
Importing posts with attachment folder getting issues
script/import_scripts/xenforo.rb:180:in `block in import_posts'
/var/www/discourse/script/import_scripts/base.rb:908:in `block in batches'
/var/www/discourse/script/import_scripts/base.rb:907:in `loop'
/var/www/discourse/script/import_scripts/base.rb:907:in `batches'
script/import_scripts/xenforo.rb:174:in `import_posts'
script/import_scripts/xenforo.rb:32:in `execute'
/var/www/discourse/script/import_scripts/base.rb:47:in `perform'
script/import_scripts/xenforo.rb:396:in `<main>'
113289 / 146471 ( 77.3%) [3766 items/min] Exception while creating post 162516. Skipping.
undefined method `synchronize' for #<DiscourseRedis:0x00005571b2d0c5b8>
/var/www/discourse/lib/discourse_redis.rb:44:in `method_missing'
/var/www/discourse/lib/distributed_mutex.rb:84:in `try_to_get_lock'
/var/www/discourse/lib/distributed_mutex.rb:61:in `get_lock'
/var/www/discourse/lib/distributed_mutex.rb:30:in `block in synchronize'
/var/www/discourse/lib/distributed_mutex.rb:29:in `synchronize'
/var/www/discourse/lib/distributed_mutex.rb:29:in `synchronize'
/var/www/discourse/lib/distributed_mutex.rb:14:in `synchronize'
/var/www/discourse/lib/post_creator.rb:407:in `transaction'
/var/www/discourse/lib/post_creator.rb:205:in `create'
/var/www/discourse/script/import_scripts/base.rb:580:in `create_post'
/var/www/discourse/script/import_scripts/base.rb:528:in `block in create_posts'
/var/www/discourse/script/import_scripts/base.rb:515:in `each'
/var/www/discourse/script/import_scripts/base.rb:515:in `create_posts'
script/import_scripts/xenforo.rb:180:in `block in import_posts'
/var/www/discourse/script/import_scripts/base.rb:908:in `block in batches'
/var/www/discourse/script/import_scripts/base.rb:907:in `loop'
/var/www/discourse/script/import_scripts/base.rb:907:in `batches'
script/import_scripts/xenforo.rb:174:in `import_posts'
script/import_scripts/xenforo.rb:32:in `execute'
/var/www/discourse/script/import_scripts/base.rb:47:in `perform'
script/import_scripts/xenforo.rb:396:in `<main>'
113290 / 146471 ( 77.3%) [3766 items/min] Exception while creating post 162517. Skipping.
undefined method `synchronize' for #<DiscourseRedis:0x00005571b2d0c5b8>
/var/www/discourse/lib/discourse_redis.rb:44:in `method_missing'
/var/www/discourse/lib/distributed_mutex.rb:84:in `try_to_get_lock'
/var/www/discourse/lib/distributed_mutex.rb:61:in `get_lock'
/var/www/discourse/lib/distributed_mutex.rb:30:in `block in synchronize'
/var/www/disc
What version of Redis do you have?
I just barely managed to limp my existing Xenforo community into discourse, so I thought I’d share my experience here to help anyone trying to do the same. Keep in mind I don’t know what I’m doing, so some of my inferences may be incorrect. I was able to complete the import to discourse with around a 20% loss in attachments. I can’t really explain why, but the issue seems only to effect the past 2-3 years of posts on my forum, whereas everything before that had around a 95+% successful import rate. My only guess is that is has something to do with the move from XF 1.x to XF 2.x.
Anyone attempting this should know that this script needs a lot of updating to work with XF 2.x. Specifically, lines 375-378 do not have the correct database tables for importing attachments from the Xenforo Media Gallery. I got around this by disconnecting the XFMG from the forum and uninstalling it, since I was only using the XFMG to mirror attachments from the forum anyways.
If you’re curious, line 376 refers to the table xengallery_media, which is obviously a reference to XFMG 1.x, as this table has been changed to xf_mg_media_item in XFMG 2.x. The rest of it I can’t get to match up and I suspect there’s some extra code needed to match up the IDs of attachments to posts.
I ended up commenting out line 314: s = process_xf_attachments(:gallery, s)
So the script wouldn’t error out on the missing SQL table.
So, with that done, I followed @Jack51’s post in this thread:
with the following edits:
step six needs the mariadb version updated. change mariadb-server-10.3 to mariadb-server-10.5
step seven should be service mariadb start instead of service mysql start.
I did change
frozen_string_literal: true
to
frozen_string_literal: false
Since I got the same error someone else mentioned in this thread.
It seems there was a LOT of confusion in this thread about where to place the attachments so the script would see them, and I have to admit I had some difficulty with this as well. For folks completely new to docker (like myself), it would be extremely useful to point out that while dockers are essentially a virtual machine running on your server (so you’re dealing with two different OSes/file systems), there is a shared space for transferring files:
/var/discourse/shared/standalone/tmp
this directory is located on your host machine, and files put there will appear in:
/shared/tmp
when you’ve entered the docker container. So, assuming you’re retaining the same file/folder structure from xenforo, you would copy the folders:
internal_data/attachments
to
/var/discourse/shared/standalone/tmp
on your host machine, and that folder would appear in /shared/tmp after entering your docker container.
Next, when editing the xenforo.rb script, change:
ATTACHMENT_DIR = '/tmp/attachments'
to
ATTACHMENT_DIR = '/shared/tmp/internal_data/attachments'
and you will have your attachments located in the correct location. It’s worth noting that xenforo also has a data/attachments folder, but it’s filled with .jpg files instead of the .data files you’re looking for. I saw someone made this mistake previously in this thread so I figured it was worth pointing out.
Also, I got a permissions error from the script, so you might have to chmod the directory:
sudo chmod a+rw -R /shared/tmp/internal_data/
I’d say the biggest giveaway that the script isn’t picking up on your attachments is that it will be strangely quiet while instead it should probably be putting out the occasional error:
So if the import goes several hours without a single error listed, it’s probably not picking up the attachments.
That’s all I have. Don’t get me wrong, I’m glad this script exists at all, 80% success rate is better than nothing (I had a similar success rate when going from vBulletin to Xenforo in 2009), and I’m excited to get working with the Discourse platform. Hopefully this post helps some folks and maybe someday someone with better skills than myself can update this script to work a little better.
3 Likes
The current version of the import script missed a lot of attachments because they’re not always embedded into the post BBcode. I’ve made a pull request with quite a few improvements and fixes. The script now sees all attachments and also imports likes, avatars and PM’s, and it creates permalinks for categories and topics.
5 Likes
@gerhard I think that PR is on your plate at the moment, could you please merge when you have some time?
2 Likes
Yes, I noticed that yesterday. 
I had completely forgotten about it. Will review it today or next week.
1 Like
Is this merged into the code now? I had to go back to xenforo for other reasons, but I’d like to come over to discourse when this script is working.
Sorry, I should have just checked the github, it looks like it was merged on the 28th of Nov.
1 Like
6 posts were split to a new topic: AVATAR_DIR constant not specified in XenForo import script
I just wanted to follow up and let anyone trying to move away from Xenforo (that find this thread through google or whatever) that there’s an importer being tested for the Flarum forum software here:
1 Like
Interesting , but it would be even more interesting if we had a solid Xenforo to Discourse importer.
The gent seems very confident in what he is doing and I applaud that!
1 Like
I did a xenforo import recently. I did make some changes to add some features but I can’t remember now which were not specific to that forum.
