בעיה: עיבוד Sidekiq איטי מאוד לאחר ייבואים גדולים במערכת מרובה אתרים

תמיכה
1

We’re running several Discourse sites with multisite under a single app. Recently, we did a batch of large user imports (hundreds of thousands of users across 6 sites). After the imports, Sidekiq is processing background jobs very slowly. The Sidekiq dashboard shows a huge backlog, and jobs are clearing at a much slower rate than expected.

Environment details:

  • The VM was upgraded to 16 CPUs / 16GB RAM.
  • However, in the Sidekiq interface, we only see 5 threads and it seems like only a small portion of the resources are being used.
  • The main import queue (“nursingjobs” as multisite parent) is handling jobs for all the child sites, but job throughput is very low.
  • Server metrics: CPU sometimes at 80–90%, memory at around 6.7/7.2GB.

We’re looking to:

  • Speed up Sidekiq/background job processing to clear large backlogs post-import.
  • Ensure Discourse is making use of all the available resources (CPU/RAM).
  • Understand if there are thread/process limits that need adjustment.

Questions:

  1. What’s the best way to configure Sidekiq/Discourse for high-throughput post-import?
  2. What are the recommended settings for UNICORN_SIDEKIQS and DISCOURSE_SIDEKIQ_WORKERS on large multi-core systems?
  3. Are there Postgres or other app.yml settings we should tweak to avoid DB pool errors when raising Sidekiq concurrency?
  4. Any best practices for clearing huge Sidekiq backlogs quickly and safely after imports?

The image displays a Sidekick.io dashboard showing process counts, failed instances, latency, and three queued tasks, each with options for deletion. (Captioned by AI)
The image displays a Sidekick.io dashboard showing process counts, failed instances, latency, and three queued tasks, each with options for deletion. (Captioned by AI)1920×590 139 KB

Sidekiq stats/screenshots available if helpful!

3

התשובה לכל השאלות האלה היא, פחות או יותר, להעלות את DISCOURSE_SIDEKIQ_WORKERS.

אני אעלה את זה לאולי 32 כי אתה יודע שיש לך הרבה מעבדי CPU פנויים. אם עדיין יש לך הרבה CPU פנוי אחרי שזה רץ במשך זמן מה, תוכל להעלות את זה עוד.

אתה יכול כנראה להוריד את זה חזרה ל, נגיד, 8 או 12 לפעילות רגילה.

ודא שיש לך מספיק max_connections ל-Postgres. כנראה שהגדרת את זה כבר יותר כי אתה מריץ סביבה מרובת אתרים, אבל תערך על זה.

2 לייקים
4

Thanks @supermathie it’s working now.
I updated config to below

  UNICORN_WORKERS: 8
  UNICORN_SIDEKIQS: 7
  DISCOURSE_SIDEKIQ_WORKERS: 10
  DISCOURSE_DB_POOL: 20

And increased CPU to

8vCPU
16GB Memory
לייק 1