Import Script High Memory Usage

[Ghan]

@pfaffman

I suppose you could have parallel processes for users and topics, but not for posts unless you re-write the script to pull in all the posts for a topic, which would allow parallel imports; that is certainly do-able, but not the way that any of the scripts that I’ve used works.

This is what I’m doing. I’ve split up the topics so that each topic is only handled by one process. It’s not a perfect split, but several times faster than a single linear process.

@pfaffman

But you’ll still have the problem with them keeping in ram an old-to-new ID map.

This is the most confusing part. I assume it is doing this as it goes along, and that’s why I saw the memory usage for each process increase over 3 days. They were getting up around 2 - 2.5 GB of memory each before database connection was lost.

Would each process only be maintaining the map for the posts it has imported? If that’s the case then it might make sense why the memory usage has exploded after restarting the import.