Migrate a XenForo forum to Discourse

Timelord · January 25, 2021, 1:08pm

For the 5k posts, and about 600 attachments, on an older xeon and an SSD disk it took about 10 mins. I would grab a reasonably powered box, do the import offline, and let it chug away.

Timelord · January 25, 2021, 1:08pm

Fantastic! Looking forward to it.

AstonJ · January 26, 2021, 3:13am

Ok I’ve done a very quick test and the additions to the script seem to have worked - doing a proper test now with a few thousand members and 100K posts. Will report back if it works…

Алексей_Батурин1 · February 3, 2021, 6:27pm

Hi, anything new? If you need help with testing, I will be glad to help

akapetzoglou1 · February 4, 2021, 1:17pm

I’m getting an error that users table doesn’t exist when I run the xenforo script. Any help really aprreciated!

Reading about this for an hour but haven’t figured it out yet.

Ghan · February 4, 2021, 1:19pm

There are some variables at the top of the script that you need to fill in so that it can connect to your xenForo MySQL database and query the data. I believe “import_db” might be the placeholder value - double check that those values are filled out correctly.

akapetzoglou1 · February 4, 2021, 3:29pm

@Ghan I thought I’m importing to import_db with this: mysql -u root -p import_db < /shared/db.sql

My xenforo.rb is:

Ghan · February 4, 2021, 3:31pm

Ah ok. If that is your database name, then check to make sure you have the table name correct. Your table might be xf_user with the prefix, and that would cause an issue. There should be a TABLE_PREFIX variable in the script as well.

akapetzoglou1 · February 15, 2021, 3:13pm

@Ghan Thanks you!

Right now, I’m still stack on the import script.
It iterates through users and categories all good. On the import_posts function though it just hangs there until after some hours it times out.

Starting debugging this by just printing the values in the import_posts:

All prints after the assignment of the results variable never print.

AstonJ · February 15, 2021, 4:12pm

Just in case anyone is looking to import a vb3 forum, this guide might help if you also have a Xenforo licence:

moshbit · February 18, 2022, 10:35pm

Does this importer import tags from xenforo along with the threads they are attached to?

merefield · February 20, 2022, 9:37pm

Always a good idea to look at the source code.

I think the answer is … yes (but not super familiar with Xenforo data structures):

github.com

discourse/discourse/blob/189df5ef438fe23058ed6c2c9ae67e10ad0b15c6/script/import_scripts/xenforo.rb#L211

      
        
                  else
                    puts "Parent post #{m['first_post_id']} doesn't exist. Skipping #{m["id"]}: #{m["title"][0..40]}"
                    skip = true
                  end
                end
            
            
    skip ? nil : mapped
              end
            end
            
            
# Apply tags
            batches(BATCH_SIZE) do |offset|
              results = mysql_query("#{posts_sql} OFFSET #{offset};").to_a
              break if results.size < 1
            
            
  results.each do |m|
                next unless m['id'] == m['first_post_id'] && m['category_id'].to_i > 0
                next unless tag = @category_mappings[m['category_id']].try(:[], :tag)
                next unless topic_mapping = topic_lookup_from_imported_post_id(m['id'])
            
            
    topic = Topic.find_by_id(topic_mapping[:topic_id])

pfaffman · February 20, 2022, 10:00pm

So, it looks like it at least once imported some tags for someone under some conditions. If you’re lucky, it’ll work for you too!

moshbit · February 22, 2022, 9:57pm

Is it safe to set up my trust levels, badges, other discourse settings before the import? I’m thinking this importer will just insert/replace posts/threads/profiles? I’d hate to get everything set up in discourse settings and have this wipe out those settings, even though I don’t immediately see anything in the code related to that.

pfaffman · February 23, 2022, 10:50am

Yes, but make a backup. You will likely need to change things and run it multiple times, starting over from scratch,so keep a backup of your configuration with no imported data.

It inserts. It mostly doesn’t replace anything. If a user exists, it won’t change it. It won’t use categories that you set up. If you run it again, with an updated database from your old site, it’ll add new data, but not change things that were imported on earlier runs. This makes it much faster, but if you need to change the script for some reason you’d need to delete it all and start again.

moshbit · February 26, 2022, 10:12pm

So, two things. First, is there a way to make this run faster? I’m on a 32 CPU Linode with 64GB of ram and it’s still taking 4-5 hours to import 250k posts.

Second, does it import attachments before or after the posts? Because looking at the script it looks like it does it after. It’s just a bummer to have to wait 4-5 hours just to find out the attachments didn’t import.

pfaffman · February 26, 2022, 11:34pm

Everything is dependent on the data before it,for the most part, the post-processing can take advantage of the multiple cpus, but that doesn’t start until you’re done, for the most part. Get a single cpu that’s faster.

You can modify the script to import only recent uploads, but if you’re not planning on doing major development, if just wait the 5 hours.

moshbit · February 27, 2022, 4:13am

Well, I’ve tried this several times now and the attachments aren’t getting imported, and I’m running out of ideas. I’m hoping someone can help me (thanks @pfaffman for your input so far).

I’ve placed the *.data files in their original folder structure from xenforo (so, the ‘attachments’ folder) in /root/attachments in the docker container. I’ve updated the import script to /root/attachments. I can see the files in /root/attachments on the command line with ls.

I’ve run the script, and every time (after about a 4-5 hour wait), all posts with attachments/photos show the old bbcode from xenforo ([ATTACH]123456[/ATTACH]) and I don’t see any pictures or attachments in the discourse install.

What could I be doing wrong here? I feel like I’m 90% there, I’m just missing something.

pfaffman · February 27, 2022, 10:23am

You’ll have to look at the code. Is the attachment code even running? Are your attachments on the format the script expects?

If the attachment function is run after the post import then you don’t need to run the whole thing again, it’ll just skip over the posts (or comment out the post import to make it a bit faster so).

But there are a bunch of things that could be wrong from you not configuring it correctly to it just not working with your version of zenforo. You’ll have to look at the code and add some debugging output to see what’s going on.

moshbit · February 28, 2022, 4:30pm

Well, just FYI for anyone coming across this thread, I was not able to get the attachments to work with this script, no matter where I placed them. IMO the script needs work and the instructions especially need a lot of work to be more clear about what goes where and how.

Until then, I would consider this script to be broken.