Migrating a large forum

(zh99998) #1

I’m migrating a many-years-old large forum from Discuz to discourse.
there is about 100k users, 3M posts and 150GB attachments.

I’m using the discuz_x import script to do this now, but it’s very slow, and may take many many days to finish it.

is there any advice for me writing a faster import script?

(Robin Ward) #2

We have some ideas for the future, but right now the unfortunate truth is you have to just wait as long as it takes. We have a very fast server set up to perform the imports we do at Discourse.

(Dean Taylor) #3

This post I made in the discussion from phpBB 3 Importer (old) might be of interest to you:

Yes, this was an older version of in the import script but might still be relevant.

It turns out the import was more like just over 750K posts and 150K topics including private messages.

The sidekiq’s execute background tasks created by each imported post and it’s always worth a little look in /sidekiq to see if you have a backlog. That’s the reason I mention “at least 25 sidekiq’s running” to keep that those tasks down.

My import ran for well over 36 hours on a 64GB Memory, 20 Core Processor machine which I ended up using.