Hi there, as I’ve mention in previous posts, I’m doing test runs of my Drupal → Discourse migration to have all the solutions in place before I ultimately take down the old site to migrate the production data with its ~2M posts. What I’ve learned is that on a fairly fast VPS with 3 vCPU cores, the i…

I think that during the user creation state there are more actions that are handled async by Sidekiq. A large part of the import will unfortunately not benefit from parallelization, you should optimize for single core CPU speed instead. Theoretically you could run different chunks of the topics im…

[изображение] rahim123: And then I’ll probably have to do some more cleanup with rake tasks I’m off-topic, but when I was working on a forum migration with the same number of posts, I modified the import script to import only 1/100 or 1/1000 topics and posts. It’s a faster way to see if your…

@Canapin Actually thanks a lot for mentioning that, I would really like to know how you did it. I’ve been wanting to do the same thing, but I threw out the idea because I assumed I would run into database inconsistencies with a partial import. So I ended up creating a skeleton Drupal test forum to …

The tasks that take a long time to restore are indeed multi-threaded. One caveat is that 2x the CPUs almost never is 2x the performance. Other point is that usually the tasks for rake posts:rebake and the heavy lifting of the forum itself to recover and optimize the content can happen with the foru…

[изображение] Mariano Rodriguez: Other point is that usually the tasks for rake posts:rebake and the heavy lifting of the forum itself to recover and optimize the content can happen with the forum live. Excellent, thanks for confirming this, I was wondering about this option as well.

[изображение] rahim123: I would really like to know how you did it Unfortunately, I haven’t written that down and I forgot… But if you know coding that shouldn’t be very difficult. I might have tweaked the BATCH_SIZE and offset values among other things to alter the loop and make it skip bat…

I want to mention two things. Yes, CPUs matter, so just get a bigger VPS and run multiple sidekiq instances for rebaking and image processing, it will go faster When your import is totally completed it’s always a good idea to do a backup / restore, it will give you better database performance. Th…

[изображение] Richard - Communiteq: run multiple sidekiq instances for rebaking and image processing, it will go faster Thanks a lot Richard for the reply. So which one(s) of these? UNICORN_WORKERS UNICORN_SIDEKIQS DISCOURSE_SIDEKIQ_WORKERS [изображение] Richard - Communiteq: When your…

UNICORN_SIDEKIQS → number of processes (default 1) DISCOURSE_SIDEKIQ_WORKERS → number of threads inside a process (default 5) [изображение] rahim123: Does that reduce fragmentation or something? It reduces fragmentation and it fixes the fact that the Postgres stats can be skewed because of …

[изображение] Richard - Communiteq: When your import is totally completed it’s always a good idea to do a backup / restore, it will give you better database performance. :+1: I haven’t seen this advice before either. If this is “always a good idea”, maybe it should be added to Pre-launch che…

I think it was Sam or Jeff who gave me this advice many years ago. I can’t find it anymore. Maybe we should check if it’s still a good idea and/or worth the effort :wink:

Многопоточность Ruby на нескольких процессорах

Поддержка Миграция

Canapin (Coin-coin le Canapin) 10.Декабрь.2022 13:43:26 10

Я тоже раньше не встречал этого совета. Если это «всегда полезно», возможно, стоит добавить эту рекомендацию в Pre-launch checklist after migrating from another platform?

Тема		Ответов	Просм.
Migrating a large forum Support	11	1666	14.10.2021
Are there any tasks to do after a forum import to improve speed? Migration	2	728	12.12.2020
Lot of sidekiq jobs during data import Support	3	1645	02.04.2019
Issue: Extremely Slow Sidekiq Processing After Large Imports on Multisite Instance Support	2	138	23.04.2025
-1 replies even though I have imported Posts properly Migration	17	861	20.11.2020

Многопоточность Ruby на нескольких процессорах

Связанные темы