I’m migrating a many-years-old large forum from Discuz to discourse. there is about 100k users, 3M posts and 150GB attachments. I’m using the discuz_x import script to do this now, but it’s very slow, and may take many many days to finish it. is there any advice for me writing a faster import script?

We have some ideas for the future, but right now the unfortunate truth is you have to just wait as long as it takes. We have a very fast server set up to perform the imports we do at Discourse.

This post I made in the discussion from phpBB 3 Importer (old) might be of interest to you: [תמונה]phpBB 3 Importer (old) My largest final import output 424.7K posts over 39.6K topics and 10.2K users. Creating the Digital Ocean instance at 2GB with a 2GB swap and then bumping it to a 16GB instance for the import only required 3 restarts. I monitored the progress (in a spreadsheet) and when the post imports per second decreased and started to tail off - I just Ctrl+C the import and then started it again. The import process slows because of the memory hole. All the while I had at least 25 sidekiq’s running - basically enough to keep the CPU at ~80% clearing the backlog of stuff at the same time as the import. Yes, this was an older version of in the import script but might still be relevant. It turns out the import was more like just over 750K posts and 150K topics including private messages. The sidekiq’s execute background tasks created by each imported post and it’s always worth a little look in /sidekiq to see if you have a backlog. That’s the reason I mention “at least 25 sidekiq’s running” to keep that those tasks down. My import ran for well over 36 hours on a 64GB Memory, 20 Core Processor machine which I ended up using.

What’s more important to the speed of import - ram or CPU?

Cpu and ssd speed are very important. You need sufficient ram, but it’s likely not important to have more than 8gb.

And the more cores I throw at it the better? Or is there a limit on that that will return benefit, assuming a fast SSD? Does the import function leverage hyper-threading? Thank you.

[תמונה] ronan0: And the more cores I throw at it the better? Not that much. Maybe one for the database and one for the importer (and a third for redis?) A recent non-scientific test I did the same import on a couple machines. One was a Intel(R) Core(TM) i7-7500U CPU @ 2.70GHz with 16GB ram. The other Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz with 32 GB. I’m pretty sure both have the same Samsung 850EVO (oh! but the first machine has 1GB). The i7 won. I think that it’s mostly because I had the SSD on a slow SATA port, but I haven’t tested again.

No, you’re 100% right, fast CPU (single thread speed, not “m0ar cores”) and fast disk is the main thing.

Thanks for the great insights. Just trying to make a judgement here before diving in. Downtime is a consideration, and we’re talking ~10 million posts here. So, limited to a single thread (is SideKiq multithreaded, may there be someway to reconfigure the script to hive off separate threads), we’re likely talking days here, even with latest, greatest cpu and reading from one ssd and writing to another?

The script can be rerun to import data that is new since the previous run. So you don’t need to take the forum down until you do the final import. Scripts that I’ve touched allow you to set an IMPORT_AFTER environment variable that will omit data from the import (otherwise it reads but skips the data). A couple importers have a bulk importer which speeds things up, but it’s more complex to run. You’re likely looking at a week or two of runtime alone for the first run.

Migrating a large forum

Product Support

merefield (Robert) 7 בדצמבר,‏ 2018,‏ 9:37am 10

Why don’t you perform a dress rehearsal or two? Optimise that and then you also get more predictability.

נושא		תגובות	צפיות
Import from vbulletin to discourse forums Development	5	2370	28 באוגוסט,‏ 2018
Ruby multi-CPU threading Migration rails-console , drupal	14	1955	20 בדצמבר,‏ 2022
PHPBB to Discourse Migration Speed Optimization Migration phpbb	18	849	17 באוגוסט,‏ 2026
Vanilla to Discourse Large Data Import (decreasing speed) Product Support	11	1095	12 בנובמבר,‏ 2020
Lot of sidekiq jobs during data import Product Support	1	1694	25 בנובמבר,‏ 2016

Migrating a large forum

נושאים קשורים