Hi there, as I’ve mention in previous posts, I’m doing test runs of my Drupal → Discourse migration to have all the solutions in place before I ultimately take down the old site to migrate the production data with its ~2M posts. What I’ve learned is that on a fairly fast VPS with 3 vCPU cores, the i…

I think that during the user creation state there are more actions that are handled async by Sidekiq. A large part of the import will unfortunately not benefit from parallelization, you should optimize for single core CPU speed instead. Theoretically you could run different chunks of the topics im…

[이미지] rahim123: And then I’ll probably have to do some more cleanup with rake tasks I’m off-topic, but when I was working on a forum migration with the same number of posts, I modified the import script to import only 1/100 or 1/1000 topics and posts. It’s a faster way to see if your import …

@Canapin Actually thanks a lot for mentioning that, I would really like to know how you did it. I’ve been wanting to do the same thing, but I threw out the idea because I assumed I would run into database inconsistencies with a partial import. So I ended up creating a skeleton Drupal test forum to …

The tasks that take a long time to restore are indeed multi-threaded. One caveat is that 2x the CPUs almost never is 2x the performance. Other point is that usually the tasks for rake posts:rebake and the heavy lifting of the forum itself to recover and optimize the content can happen with the foru…

[이미지] Mariano Rodriguez: Other point is that usually the tasks for rake posts:rebake and the heavy lifting of the forum itself to recover and optimize the content can happen with the forum live. Excellent, thanks for confirming this, I was wondering about this option as well.

[이미지] rahim123: I would really like to know how you did it Unfortunately, I haven’t written that down and I forgot… But if you know coding that shouldn’t be very difficult. I might have tweaked the BATCH_SIZE and offset values among other things to alter the loop and make it skip batches of …

I want to mention two things. Yes, CPUs matter, so just get a bigger VPS and run multiple sidekiq instances for rebaking and image processing, it will go faster When your import is totally completed it’s always a good idea to do a backup / restore, it will give you better database performance. Th…

[이미지] Richard - Communiteq: run multiple sidekiq instances for rebaking and image processing, it will go faster Thanks a lot Richard for the reply. So which one(s) of these? UNICORN_WORKERS UNICORN_SIDEKIQS DISCOURSE_SIDEKIQ_WORKERS [이미지] Richard - Communiteq: When your import is total…

UNICORN_SIDEKIQS → number of processes (default 1) DISCOURSE_SIDEKIQ_WORKERS → number of threads inside a process (default 5) [이미지] rahim123: Does that reduce fragmentation or something? It reduces fragmentation and it fixes the fact that the Postgres stats can be skewed because of the impo…

[이미지] Richard - Communiteq: When your import is totally completed it’s always a good idea to do a backup / restore, it will give you better database performance. :+1: I haven’t seen this advice before either. If this is “always a good idea”, maybe it should be added to Pre-launch checklist a…

I think it was Sam or Jeff who gave me this advice many years ago. I can’t find it anymore. Maybe we should check if it’s still a good idea and/or worth the effort :wink:

Ruby 멀티 CPU 스레딩

제품 지원 이주

Canapin (Coin-coin le Canapin) 12월 10, 2022, 1:43오후 10

저도 이전에 이 조언을 본 적이 없습니다. 만약 이것이 "항상 좋은 아이디어"라면, Pre-launch checklist after migrating from another platform? 에 추가해야 하지 않을까요?

글		댓글	조회수
Migrating a large forum Product Support	10	1833	12월 7, 2018
PHPBB to Discourse Migration Speed Optimization Migration phpbb	18	772	8월 17, 2026
Are there any tasks to do after a forum import to improve speed? Migration	2	772	12월 12, 2020
Lot of sidekiq jobs during data import Product Support	1	1681	11월 25, 2016
Issue: Extremely Slow Sidekiq Processing After Large Imports on Multisite Instance Product Support	2	217	4월 23, 2025

Ruby 멀티 CPU 스레딩

Related topics