In a discussion titled “Understanding DB Pooling for Sidekiq Workers,” Mudit_Maheshwari seeks clarification on why database connection spikes exceed the expected limit of 30 (calculated as 15 pool size × 2 Sidekiq workers, each with 10 threads), particularly during the PeriodicalUpdates job. Despite setting DISCOURSE_DB_POOL=15, DISCOURSE_SIDEKIQ_WORKERS=10, and UNICORN_SIDEKIQS=2, the observed spikes suggest the app may not be respecting the pool limit. david asks whether the graph reflects concurrent connections or connection creation/destruction, and whether the proxy or configuration (e.g., UNICORN_SIDEKIQS) might be contributing. Mudit_Maheshwari confirms the graph shows current client connections and clarifies the goal is to properly size the RDS proxy’s connection pool by understanding application-side pooling behavior. Mudit_Maheshwari also asks if additional config files beyond DISCOURSE_DB_POOL are needed. sam questions whether the RDS proxy itself might be causing the issue, but Mudit_Maheshwari rules that out. The core mystery remains: why are connections spiking beyond the configured pool limit during scheduled jobs?
Can some please provide clarity on Sidekiq worker database pooling?
Currently, we’ve set DISCOURSE_DB_POOL to 15, with 2 Sidekiq workers each having a concurrency of 10. After reading the Sidekiq documentation, it appears that Sidekiq threads share the DB pool, hence I anticipate a maximum of 15 * 2 = 30 DB connections from Sidekiq workers. However, despite this understanding, we’re observing spikes in DB connections exceeding the expected maximum of 30.
I’m afraid I don’t know the details of how all this works off-hand.
It may be worth checking whether you’ve customized the UNICORN_SIDEKIQS env. That indicates how many sidekiq processes are launched (each of which will have many threads).
What exactly is your graph showing? Is it ‘concurrent connections’? Or is it ‘number of connections created’? If it’s the latter, perhaps some connections are being created & destroyed very quickly.
And one last question: what problem are you trying to solve here? Is there some issue being caused by the number of connections?
This means we have 2 Sidekiq processes, each with 10 threads.
The graph shows the current number of client connections.
We are trying to configure the size of the connection pool (RDS proxy sitting in front of PostgreSQL) correctly. To do so, we need a better understanding of the connection pooling on the application side. Why is the application not respecting the DISCOURSE_DB_POOL config as expected?
If DISCOURSE_DB_POOL is working as expected, why do we see such big spikes in DB connections? These spikes seem to be aligned with the PeriodicalUpdates scheduled job runs.
Please let me know if you have more questions. Appreciate the help to uncover the mystery.
