Import Script High Memory Usage

See discourse/script/bulk_import at main · discourse/discourse · GitHub.

For topics and posts it’s not really feasible to run parallel imports anyway, as you can’t import a post in a topic if the topic and all previous posts are not already imported. I suppose you could have parallel processes for users and topics, but not for posts unless you re-write the script to pull in all the posts for a topic, which would allow parallel imports; that is certainly do-able, but not the way that any of the scripts that I’ve used works. But you’ll still have the problem with them keeping in ram an old-to-new ID map.

Importing 25M posts isn’t convenient. :slight_smile: