I’m trying to do an import test from our existing board. We have about 25 million posts to import (normal posts + private conversations), so to speed this up, I have created multiple copies of the import script to run simultaneously and split up the topic load. This worked fine for a couple of days, and over time I noticed the memory usage for each process increasing slowly up to around 2 GB each. Then, the server finally ran out of memory and killed the MySQL source database around the 16 million post mark.
I’ve increased the system memory from 24 GB to 32 GB, but now when I attempt to restart even 1 import process and pick up where it left off, that process is consuming about 10 GB of memory out of the gate before it even starts importing posts. Where before I was able to run 8 simultaneous import processes, I can now only fit 2 into a larger memory pool. Why is there this huge discrepancy between memory usage from a clean install and memory usage when restarting an import after a failure? Is there any way I can reduce this memory footprint so I can speed up the import process again? A server with 128 GB - 256 GB of memory will be prohibitively expensive (and not needed after the import) and running with only 2 import processes will mean the import will take weeks to complete.