Smf2 Import Script Running Slow
The user running the smf2 import script is experiencing slow progress, with only 8.2% completion after 18 hours. domainghost is seeking advice on speeding up the operation.
domainghost reports that the script is importing users at a claimed rate of 1900 items per minute, but actual progress is around 60-70 items per minute. The user has approximately 5.8 lakh (580,000) users to import.
pfaffman suggests that subsequent runs will be faster since they skip already imported data. He also inquires about the system’s CPU, RAM, and Postgres optimization.
pfaffman explains that the reported import rate may be inflated due to counting existing users. He advises checking the machine’s speed, RAM, and Postgres tuning, as well as the source database, for potential bottlenecks.
pfaffman recommends sticking with the script, as the final import will take less time, and is not aware of any secret bulk import scripts for smf2.
Wanna ask about smf2 importer. It works fine, but Its running very slow.
And it goes slower and slower. I tried migration with drupal before it was little fast, it worked 3-4 days. Over the whole night smf2.rb was 3% progress. In 18 hours its only progressed 8.2%.
My questions is What’s the best way to speed up the operation?
How many posts per minute is it reporting? How many posts do you have?
Subsequent runs will be faster since they skip the already imported data, so you can have not much downtime when you make the final switch.
Do you have a fast cpu and plenty of ram? Is postgres optimized?
Hi @pfaffman, post migrating not yet started now user import is going on and its showing around 1900 item per min but its not going like the rate per min around 60-70 item are reporting.
any suggestion how can i speed up this operation ?
How many users are there?
1900/minute is as high as I’ve seen.
What do you mean it’s 60-70 per minute?
There are some bulk import scripts that are faster but I don’t think there’s one for smf2.
Total user around 5.8 Lakh. and yes 1900/minute is very high but actually its not happening like this rate as i am checking from backend as well as UI its migrating around 60-70 per min. I have checked that folder as well no script is available for smf2, Is there any bulk import script is available anywhere else for smf2 ?
Unless the users are not getting imported for some reason (they already exist?) then the number it reports is the number importered.
Ah, but if you are running it after the initial import then that number is inflated because it’s counting users that existed when the script started. So that’s about a week to import those.
How many posts?
How fast is the machine? How much ram? What have you done to tune postgres? Is your source database a bottleneck?
I’m not aware of a secret bulk import script for smf2. I have had an import that took weeks to run. The final import won’t take that long. I’d just stick it out.