So have an issue with sidekiq.
It will run amazingly fast through jobs when monitoring via the sidekiq web ui. But occasionally it appears like it gets overwhelmed and starts running extremely slow. Running at 1-5% or so of it’s normal speed and does not recover unless I flush redis, despite server resource usage being fine/low.
It appears like once the queue hits a certain size, it seizes up and slows down drastically. Causing the queue to grow even more. I’m just guessing here though, maybe the queue is just large due to it slowing down for some other reason.
This gif describes what it looks like to me.
There are plenty of server resources available, CPU usage is very low right now - under 10%. Plenty of ram and ssd available also. Regarding the server, it has 16 CPU cores with 32 threads. I’ve tried running between 8-14 unicorn_sidekiqs. I also tried 20, but that created a lot of 5xx errors.
I was able to speed up slow jobs displayed on the ‘busy’ tab of the sidekiq web ui using
Could sidekiq queue be reason for 500 errors? (
adding ‘vm.overcommit_memory = 1’ to /etc/sysctl.conf file and rebooting) and also decreasing unicon_sidekiqs down to 8 (from 12).
It’s still running slow though. I did see this in the redis log yesterday (the only other warning was regarding not having overcommit_memory set to 1, which I modified above):
# WARNING: /proc/sys/net/core/somaxconn is set to the lower value of 128
^ Has anyone fixed this warning above?
Anyhow, if anyone has any ideas as to what could be the cause and/or fix - please let me know. I’d appreciate it.
Would be really great to resolve this issue so it doesn’t happen again, rather than flushing.
Here is a screenshot of what I’m seeing on the sidekiq dashboard:
And some screenshots of the jobs under the busy tab:
Also, does anyone know if it is safe to use this option? Deleting the low priority queue from the sidekiq web ui?