Hi there, as I’ve mention in previous posts, I’m doing test runs of my Drupal → Discourse migration to have all the solutions in place before I ultimately take down the old site to migrate the production data with its ~2M posts. What I’ve learned is that on a fairly fast VPS with 3 vCPU cores, the import process takes forever, somewhere around 48 hours. And then I’ll probably have to do some more cleanup with rake tasks and/or rails c, and for anything that requires a rake posts:rebake it will take another 20 hours approx. I don’t really understand the fundamentals of the Ruby toolchain. But if I throw more CPU cores at the job will it significantly reduce the amount of time that any one of these processes require to complete? For example will a bundle command or a rake command be able to divide its work between the available CPUs, or are the additional cores mainly useful to run multiple concurrent processes when multiple users are hitting the website?

I think that during the user creation state there are more actions that are handled async by Sidekiq. A large part of the import will unfortunately not benefit from parallelization, you should optimize for single core CPU speed instead. Theoretically you could run different chunks of the topics import in parallel but it would require quite some refactoring of the importer and making sure everything is processed in order. Not worth it for a one off task with a few iterations.

@Canapin Actually thanks a lot for mentioning that, I would really like to know how you did it. I’ve been wanting to do the same thing, but I threw out the idea because I assumed I would run into database inconsistencies with a partial import. So I ended up creating a skeleton Drupal test forum to test on. But I’d prefer to test a copy of the production DB. I’m mainly concerned about the eventual final production migration; I’ll have to take the old forum offline or at least make it read-only, and it’s looking like at least a 48h downtime at best, unless throwing double the CPU cores at it would cut the time in half?

The tasks that take a long time to restore are indeed multi-threaded. One caveat is that 2x the CPUs almost never is 2x the performance. Other point is that usually the tasks for rake posts:rebake and the heavy lifting of the forum itself to recover and optimize the content can happen with the forum live. Which might reduce the time you need to have the forum either off-line or read-only and be able to offer a somewhat degraded experience. My recommendation would be: first, test. Do the migration, see how much times it takes and how the forum looks without all the rebakes in place. If it’s good enough time the migration to end around the lower traffic time of your forum that way you earn some 4-10h of migration without a lot of people complaining.

[image] Mariano Rodriguez: Other point is that usually the tasks for rake posts:rebake and the heavy lifting of the forum itself to recover and optimize the content can happen with the forum live. Excellent, thanks for confirming this, I was wondering about this option as well.

[image] rahim123: I would really like to know how you did it Unfortunately, I haven’t written that down and I forgot… But if you know coding that shouldn’t be very difficult. I might have tweaked the BATCH_SIZE and offset values among other things to alter the loop and make it skip batches of posts or something like that… I can’t re-try now because I don’t have any forum to import right now, but I’ll make a quick tutorial next time because I think it’s quite useful.

I want to mention two things. Yes, CPUs matter, so just get a bigger VPS and run multiple sidekiq instances for rebaking and image processing, it will go faster When your import is totally completed it’s always a good idea to do a backup / restore, it will give you better database performance. These two together: get a big VPS for the import and when you’re done move it to a smaller production VPS (using backup and restore). Generally an import will not require you to rebake posts afterwards.

[image] Richard - Communiteq: run multiple sidekiq instances for rebaking and image processing, it will go faster Thanks a lot Richard for the reply. So which one(s) of these? UNICORN_WORKERS UNICORN_SIDEKIQS DISCOURSE_SIDEKIQ_WORKERS [image] Richard - Communiteq: When your import is totally completed it’s always a good idea to do a backup / restore, it will give you better database performance. Interesting, haven’t seen this recommendation before. Does that reduce fragmentation or something? [image] Richard - Communiteq: Generally an import will not require you to rebake posts afterwards. Yeah, I was initially going to try to fix some [QUOTE] problems and Textile → Markdown conversion with regexp_replace() in the Postgres console and then rebake all posts, because the rake posts:remap commands were just too slow. But then I discovered that the regexp flavor that Postgres uses is not PCRE compatible, and there are just too many unexpected anomalies to rely on it. So I’m going to try to run the posts through Pandoc during the import process, which should allow me to get the imported site up and running in a presentable state and then fix smaller stuff like emoji keywords with rake posts:remap .

UNICORN_SIDEKIQS → number of processes (default 1) DISCOURSE_SIDEKIQ_WORKERS → number of threads inside a process (default 5) [image] rahim123: Does that reduce fragmentation or something? It reduces fragmentation and it fixes the fact that the Postgres stats can be skewed because of the importing.

[image] Richard - Communiteq: When your import is totally completed it’s always a good idea to do a backup / restore, it will give you better database performance. :+1: I haven’t seen this advice before either. If this is “always a good idea”, maybe it should be added to Pre-launch checklist after migrating from another platform?

I think it was Sam or Jeff who gave me this advice many years ago. I can’t find it anymore. Maybe we should check if it’s still a good idea and/or worth the effort :wink:

Ruby multi-CPU threading

Product Support Migration

Canapin (Coin-coin le Canapin) December 9, 2022, 11:19pm 2

I’m off-topic, but when I was working on a forum migration with the same number of posts, I modified the import script to import only 1/100 or 1/1000 topics and posts.

It’s a faster way to see if your import is reliable and if tweaks or debugging are needed.

Topic		Replies	Views
Migrating a large forum Product Support	10	1873	December 7, 2018
PHPBB to Discourse Migration Speed Optimization Migration phpbb	18	847	August 17, 2026
Are there any tasks to do after a forum import to improve speed? Migration	2	781	December 12, 2020
Lot of sidekiq jobs during data import Product Support	1	1694	November 25, 2016
Issue: Extremely Slow Sidekiq Processing After Large Imports on Multisite Instance Product Support	2	248	April 23, 2025

Ruby multi-CPU threading

Related topics