I’ve been doing import testing from another software with a large board (~10 million posts) and I’ve noticed that while the import process is running, I see another Ruby process that appears to be sidekiq stacking up tons of queued processes (sidekiq showing 100,000+ enqueued processes). Is there a better way to handle this? Is it possible (or recommended) to disable sidekiq while running a large import? If so, how would I do that?
AI-generated summary
During a large-scale import (~10 million posts), Ghan observed Sidekiq queuing over 100,000 jobs, causing performance concerns. pfaffman suggested disabling the discourse narrative bot enabled setting to reduce unnecessary jobs, noting most queued tasks are essential, and pointed to discussions on scaling Sidekiq workers. Ghan agreed to try adding more Sidekiq workers and temporarily disabling the bot setting, but emphasized that the bulk of the load stems from processing millions of posts. Ghan also referenced a thread on adjusting Sidekiq worker counts. codinghorror expressed surprise that migrations don’t automatically disable Discobot during imports, calling it “kind of shocking.” The discussion centers on optimizing Sidekiq during large imports by scaling workers and selectively disabling non-critical features like Discobot to reduce queue pressure.
You could turn off site setting discourse narrative bot enabled to stop creating that message for every new user, but most of those jobs need to happen. There are some topics here about increasing the number of sidekiq jobs to process the queue more quickly.
Ok, sounds good. I think adding another sidekiq worker could help. I will probably also disable that setting as you suggest, at least for the import, though that is just from the users. My main problem will be processing that needs to happen on the millions of posts.
I found this thread on how to adjust the number of sidekiq workers: Imported users not receiving verification emails due to massive Sidekiq backlog - #10 by gerhard
Thanks!
Migrations don’t turn discobot off during the migration? That’s kind of shocking.