Sidekiq is consuming too much memory even with UNICORN_SIDEKIQ_MAX_RSS adjusted

Support Self-hosting

AI-generated summary

Users are experiencing issues with Sidekiq consuming too much memory and restarting, despite adjusting the UNICORN_SIDEKIQ_MAX_RSS setting. pfaffman reports that even with UNICORN_SIDEKIQ_MAX_RSS set to 1000, Sidekiq restarts when using around 535MB. sam suggests checking if increasing UNICORN_SIDEKIQS results in multiple Sidekiq processes. sandra.mccollum also encounters the same issue on a less busy site and asks for further troubleshooting steps. Additionally, sandra.mccollum and haydenjames discuss whether this is a Sidekiq issue and if it will be fixed, with haydenjames mentioning a similar problem in another thread.

1

Continuing the discussion from Sidekiq is consuming too much memory, restarting:

I’ve adjusted the setting and see this in the container:

root@ip-10-4-2-52:/# set|grep SIDE
UNICORN_SIDEKIQS=1
UNICORN_SIDEKIQ_MAX_RSS=1000

But I see this in the logs:

Sidekiq is consuming too much memory (using: 535.10M) for 'site', restarting

It would seem like 535.10M < 1000, right? I feel like I must have a typo or something else really silly.

This is a very busy site with 600-800K pageviews/day with 7 (largely idle) web containers running in ECS.

There are 4 CPUs in the task, should I increase UNICORN_SIDEKIQS?

If I run [ENV["UNICORN_SIDEKIQ_MAX_RSS"].to_i, 500].max.megabytes in rails, I get 1048576000, so it seems like Rails should have access to the ENV, but maybe running rails myself is somehow different from the one that gets cranked up on boot?

2

it looks like you are doing the right thing… if you raise UNICORN_SIDEKIQS to 2 do you see 2 sidekiqs?

3

I’ll try increasing the number of sidekiq next, but I’m confused that it says 538 mb is to much when it’s set for 1000.

4

Agree, just want to see that at least one of the env vars is “sticking”

5

Was there any resolution to this, as I see in my logs:

Sidekiq is consuming too much memory (using: 501.37M) , restarting

UNICORN_SIDEKIQ_MAX_RSS: 1000

and the site is not a very busy one

What else should I try?

6

Is this a Sidekiq issue, and whether it is nor not, will this issue be fixed?

Job exception: MiniScheduler::DistributedMutex::Timeout
We 6:21 pm

mini_scheduler-0.17.0/lib/mini_scheduler/distributed_mutex.rb:43:in `synchronize' 
mini_scheduler-0.17.0/lib/mini_scheduler/distributed_mutex.rb:15:in `synchronize' 
mini_scheduler-0.17.0/lib/mini_scheduler/manager.rb:365:in `lock' 
mini_scheduler-0.17.0/lib/mini_scheduler/manager.rb:316:in `tick' 
mini_scheduler-0.17.0/lib/mini_scheduler.rb:76:in `block (2 levels) in start'

Posted by NateDhaliwal:

The image shows a code snippet from the GitHub repository "discourse" containing code related to the Sidekiq job scheduler, with a question about whether it might be a Sidekiq issue. (Captioned by AI)
The image shows a code snippet from the GitHub repository "discourse" containing code related to the Sidekiq job scheduler, with a question about whether it might be a Sidekiq issue. (Captioned by AI)784×822 64.3 KB

7

@sandra.mccollum I also have a similar issue.

Were you able to solve?