Sidekiq job concurrency issue

Installation
1

We have an issue with Sidekiq running on multi pod.

Scenario 1:
On development, we have a single replicas which means single pod running. We don’t have any issue here.
FYI,
Checked: forum_url/sidekiq/busy( 1 Processes running)

Scenario 2:
On preprod and prod we have more then one replicas (more than one pod running). We’re facing Sidekiq concurrency issue (The job is running but skipping the code execution).

FYI,
Checked: forum_url/sidekiq/busy (More than 1 processes running). For each pod one Sidekiq processes running

Correct me,
Is that okay to run more than one sidekiq processes?
If It’s not okay how to fix the job concurrency issue?

FYI, We’re using the Discourse Docker

What are the appropriate steps should I need to follow. Thanks in advance

2

What do you mean by exactly by “concurrency” issue? Can you please elaborate? What is happening out of order exactly?

A sidekiq job being added to a queue does not guarantee overall order of execution as there are different queues with different priority so some jobs can get ahead of others.

Also “skipping code execution”. Can you elaborate? What code? When a job runs the code will not be skipped. Have you checked your /logs that nothing is breaking?

And be aware you might be running an unsupported install which might be at the root of your issues …

4 Likes
3

Hi @merefield Thank you for the replay

What do you mean by exactly by “concurrency” issue? Can you please elaborate? What is happening out of order exactly?

As I mentioned above, on preprod and production environment more than one sidekiq process running.

Screenshot 2023-08-11 at 10.16.58 AM
Screenshot 2023-08-11 at 10.16.58 AM1920×609 38.3 KB

I’m not sure! The job might confuse when it tiger which process it should take because of the multi worker process.

Also “skipping code execution”. Can you elaborate? What code? When a job runs the code will not be skipped. Have you checked your /logs that nothing is breaking?

We’re running some custom sidekiq jobs which are export some custom sats on CSV file. The job is running periodically. Sometimes the file export does not happening. The job runs without any error message.

For ex:

Screenshot 2023-08-11 at 10.37.32 AM
Screenshot 2023-08-11 at 10.37.32 AM2530×170 53.3 KB

For the testing purpose, If I click this trigger button the job is running but the file export does not happening sometimes.

FYI, on Dev env same code running but it’s only one sidekiq worker process.Here working fine. I’m wondering the issue is because of more than one sidekiq process running on the prod/preprod.

Have you checked your /logs that nothing is breaking?

Yes, We didn’t get any error related to the job

And be aware you might be running an unsupported install

We’re running latest discourse version.

Thank you

4

You have to expect concurrency. It’s a multi threaded batch system.

So are some people on Kubernetes but that’s not supported here.

I think you are going to have to add some logging to your job to find out what is going on.

This is your own custom code, right?

5

We have already added logging on my custom code. No issue on our code.

Yes

Yes, We’re using Kubernetes cluster.

Just one question!

The sidekiq should run only one process? or running more than one sidekiq process is fine?

6

Like any decent batch system it is designed to run more than one job at a time, otherwise it wouldn’t scale.

Make sure you take that into account when designing your jobs.