# Migrating a large forum
**URL:** https://meta.discourse.org/t/migrating-a-large-forum/37838
**Category:** Support
**Created:** [January 12, 2016, 3:59pm UTC](https://meta.discourse.org/t/migrating-a-large-forum/37838 "2016-01-12T15:59:25Z")
**Posts on this page:** 12
**Page:** 1
### Author:  [@zh99998](https://meta.discourse.org/u/zh99998)
#### Post date: [January 12, 2016, 3:59pm UTC](https://meta.discourse.org/t/migrating-a-large-forum/37838/1 "2016-01-12T15:59:25Z")
I’m migrating a many-years-old large forum from Discuz to discourse.
there is about 100k users, 3M posts and 150GB attachments.
I’m using the discuz\_x import script to do this now, but it’s very slow, and may take many many days to finish it.
is there any advice for me writing a faster import script?
---
### Author:  [@eviltrout](https://meta.discourse.org/u/eviltrout)
#### Post date: [January 12, 2016, 5:15pm UTC](https://meta.discourse.org/t/migrating-a-large-forum/37838/2 "2016-01-12T17:15:10Z")
We have some ideas for the future, but right now the unfortunate truth is you have to just wait as long as it takes. We have a very fast server set up to perform the imports we do at Discourse.
---
### Author:  [@DeanMarkTaylor](https://meta.discourse.org/u/DeanMarkTaylor)
#### Post date: [January 12, 2016, 5:31pm UTC](https://meta.discourse.org/t/migrating-a-large-forum/37838/3 "2016-01-12T17:31:22Z")
This post I made in the discussion from [phpBB 3 Importer (old)](https://meta.discourse.org/t/phpbb-3-importer-old/17397/75) might be of interest to you:
> [@phpBB 3 Importer (old)](https://meta.discourse.org/t/phpbb-3-importer-old/17397/75):
>
> My largest final import output 424.7K posts over 39.6K topics and 10.2K users.
>
> Creating the Digital Ocean instance at 2GB with a 2GB swap and then bumping it to a 16GB instance for the import only required 3 restarts.
>
> I monitored the progress (in a spreadsheet) and when the post `imports per second decreased` and started to tail off - I just Ctrl+C the import and then started it again.
> The import process slows because of the memory hole.
>
> All the while I had at least 25 sidekiq’s running - basically enough to keep the CPU at ~80% clearing the backlog of stuff at the same time as the import.
Yes, this was an older version of in the import script but might still be relevant.
It turns out the import was more like just over 750K posts and 150K topics including private messages.
The sidekiq’s execute background tasks created by each imported post and it’s always worth a little look in `/sidekiq` to see if you have a backlog. That’s the reason I mention “at least 25 sidekiq’s running” to keep that those tasks down.
My import ran for well over 36 hours on a 64GB Memory, 20 Core Processor machine which I ended up using.
---
### Author:  [@ronan0](https://meta.discourse.org/u/ronan0)
#### Post date: [December 5, 2018, 1:08pm UTC](https://meta.discourse.org/t/migrating-a-large-forum/37838/4 "2018-12-05T13:08:05Z")
What’s more important to the speed of import - ram or CPU?
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [December 5, 2018, 1:11pm UTC](https://meta.discourse.org/t/migrating-a-large-forum/37838/5 "2018-12-05T13:11:32Z")
Cpu and ssd speed are very important. You need sufficient ram, but it’s likely not important to have more than 8gb.
---
### Author:  [@ronan0](https://meta.discourse.org/u/ronan0)
#### Post date: [December 5, 2018, 3:55pm UTC](https://meta.discourse.org/t/migrating-a-large-forum/37838/6 "2018-12-05T15:55:17Z")
And the more cores I throw at it the better? Or is there a limit on that that will return benefit, assuming a fast SSD? Does the import function leverage hyper-threading? Thank you.
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [December 7, 2018, 12:09am UTC](https://meta.discourse.org/t/migrating-a-large-forum/37838/7 "2018-12-07T00:09:54Z")
> [@ronan0](#):
>
> And the more cores I throw at it the better?
Not that much. Maybe one for the database and one for the importer (and a third for redis?)
A recent non-scientific test I did the same import on a couple machines. One was a `Intel(R) Core(TM) i7-7500U CPU @ 2.70GHz` with 16GB ram. The other `Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz` with 32 GB. I’m pretty sure both have the same Samsung 850EVO (oh! but the first machine has 1GB). The i7 won. I think that it’s mostly because I had the SSD on a slow SATA port, but I haven’t tested again.
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [December 7, 2018, 12:15am UTC](https://meta.discourse.org/t/migrating-a-large-forum/37838/8 "2018-12-07T00:15:09Z")
No, you’re 100% right, fast CPU (single thread speed, not “m0ar cores”) and fast disk is the main thing.
---
### Author:  [@ronan0](https://meta.discourse.org/u/ronan0)
#### Post date: [December 7, 2018, 9:27am UTC](https://meta.discourse.org/t/migrating-a-large-forum/37838/9 "2018-12-07T09:27:29Z")
Thanks for the great insights. Just trying to make a judgement here before diving in. Downtime is a consideration, and we’re talking ~10 million posts here. So, limited to a single thread (is SideKiq multithreaded, may there be someway to reconfigure the script to hive off separate threads), **we’re likely talking days here** , even with latest, greatest cpu and reading from one ssd and writing to another?
---
### Author:  [@merefield](https://meta.discourse.org/u/merefield)
#### Post date: [December 7, 2018, 9:37am UTC](https://meta.discourse.org/t/migrating-a-large-forum/37838/10 "2018-12-07T09:37:34Z")
Why don’t you perform a dress rehearsal or two? Optimise that and then you also get more predictability.
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [December 7, 2018, 1:34pm UTC](https://meta.discourse.org/t/migrating-a-large-forum/37838/11 "2018-12-07T13:34:58Z")
The script can be rerun to import data that is new since the previous run. So you don’t need to take the forum down until you do the final import. Scripts that I’ve touched allow you to set an IMPORT\_AFTER environment variable that will omit data from the import (otherwise it reads but skips the data).
A couple importers have a bulk importer which speeds things up, but it’s more complex to run.
You’re likely looking at a week or two of runtime alone for the first run.
---
### Author:  [@system](https://meta.discourse.org/u/system)
#### Post date: [October 14, 2021, 9:34pm UTC](https://meta.discourse.org/t/migrating-a-large-forum/37838/12 "2021-10-14T21:34:18Z")