We have an issue with Sidekiq running on multi pod. Scenario 1: On development, we have a single replicas which means single pod running. We don’t have any issue here. FYI, Checked: forum_url/sidekiq/busy( 1 Processes running) Scenario 2: On preprod and prod we have more then one replicas (more than one pod running). We’re facing Sidekiq concurrency issue (The job is running but skipping the code execution). FYI, Checked: forum_url/sidekiq/busy (More than 1 processes running). For each pod one Sidekiq processes running Correct me, Is that okay to run more than one sidekiq processes? If It’s not okay how to fix the job concurrency issue? FYI, We’re using the Discourse Docker What are the appropriate steps should I need to follow. Thanks in advance

What do you mean by exactly by “concurrency” issue? Can you please elaborate? What is happening out of order exactly? A sidekiq job being added to a queue does not guarantee overall order of execution as there are different queues with different priority so some jobs can get ahead of others. Also “skipping code execution”. Can you elaborate? What code? When a job runs the code will not be skipped. Have you checked your /logs that nothing is breaking? And be aware you might be running an unsupported install which might be at the root of your issues …

Hi @merefield Thank you for the replay What do you mean by exactly by “concurrency” issue? Can you please elaborate? What is happening out of order exactly? As I mentioned above, on preprod and production environment more than one sidekiq process running. [Screenshot 2023-08-11 at 10.16.58 AM] I’m not sure! The job might confuse when it tiger which process it should take because of the multi worker process. Also “skipping code execution”. Can you elaborate? What code? When a job runs the code will not be skipped. Have you checked your /logs that nothing is breaking? We’re running some custom sidekiq jobs which are export some custom sats on CSV file. The job is running periodically. Sometimes the file export does not happening. The job runs without any error message. For ex: [Screenshot 2023-08-11 at 10.37.32 AM] For the testing purpose, If I click this trigger button the job is running but the file export does not happening sometimes. FYI, on Dev env same code running but it’s only one sidekiq worker process.Here working fine. I’m wondering the issue is because of more than one sidekiq process running on the prod/preprod. Have you checked your /logs that nothing is breaking? Yes, We didn’t get any error related to the job And be aware you might be running an unsupported install We’re running latest discourse version. Thank you

[image] Prasanth Balan: I’m wondering the issue is because of more than one sidekiq process running on the prod/preprod. You have to expect concurrency. It’s a multi threaded batch system. [image] Prasanth Balan: We’re running latest discourse version. So are some people on Kubernetes but that’s not supported here. I think you are going to have to add some logging to your job to find out what is going on. This is your own custom code, right?

[image] Robert: I think you are going to have to add some logging to your job to find out what is going on. We have already added logging on my custom code. No issue on our code. [image] Robert: This is your own custom code, right? Yes [image] Robert: So are some people on Kubernetes but that’s not supported here. Yes, We’re using Kubernetes cluster. Just one question! The sidekiq should run only one process? or running more than one sidekiq process is fine?

Sidekiq job concurrency issue

Product Support Self-hosting

merefield (Robert) August 11, 2023, 8:58am 6

Like any decent batch system it is designed to run more than one job at a time, otherwise it wouldn’t scale.

Make sure you take that into account when designing your jobs.

Topic		Replies	Views
Issue: Extremely Slow Sidekiq Processing After Large Imports on Multisite Instance Product Support	2	248	April 23, 2025
Spin up additional sidekiq workers Product Support	2	1282	February 28, 2018
Sidekiq has a lot of errors and queued jobs Product Support	18	1600	March 1, 2024
Sidekiq stops after some time Self-hosting	7	1363	June 14, 2023
"Ensure sidekiq is running." when it is definitely running Self-hosting	18	8269	October 2, 2015

Sidekiq job concurrency issue

Related topics