Import scripts - how to improve performance?

(Thomas Wilson) #1

Any tips would be greatly appreciated, as my current calculations suggest our forum import will take 6 days to complete.

Some info on the underlying tech would be great. I’ve upgraded my DO droplet to 4 CPU cores / 8GB RAM which has improved things a little, but things are still quite sluggish. Users (200,000) of them are done, but posts (around 2 million) are slow (around 3-5 / second).

Should I increase the number of sidekiq workers available in the app.yml file (this is currently set using the single CPU installation instructions), or is that not relevant here?

(Régis Hanol) #2

One thing I found that improved the speed of the import was to make chunks of about 200k posts. It took about 14 hours per 200k posts to import whereas when I tried to import 1M posts in one go, it took 1 week and didn’t even finish.

(Kane York) #3

Feels like a good old-fashioned case of algorithmic complexity to me…