We turned off the plugin for about 24hr, then turned it back on for 24hr.
Below I’m pasting notes from our other admin.
The plugin is causing jobs being scheduled to slow.
The more jobs are in the queue, the more the CPU struggles to keep up.
so what happens is the server sees a lot of traffic, but things seem normal for a while until the jobs start backing up.
Eventually the CPU struggles to keep up and usage spikes up and down as it completes backlogged jobs.
That’s why we saw earlier there was 3000+ backlogged jobs.
under normal circumstances there’s never more than 5 jobs. but almost always there are 0 because no jobs should be backlogged, they should be processed immediately.
Image attached shows the current jobs getting backlogged. Right now it’s hovering 30-35 backlogged jobs.
All the jobs are new from the last post in the sidecar.
I can’t tell why yet but this only happens when the plugin is on.
Last hour of the resources on CPU
last 24 hours, pointing out approx. where the plugin was enabled (See the up trend after the spike)
It seems like the pattern is longer than 24 hours to notice but the plugin causes general resource usage in the 80-90+% usage most of the time.
Once we turn it off, we’ll start noticing that over the next 24 hours the server will average 60-75% and the jobs won’t get backlogged.
Also when we want to, I changed the app.yml to have 16 unicorn workers. when we want to rebuild I think we should disable the plugin, and have 16 unicorn workers. We observe the server’s performance over that period of time, and adjust the workers value to what we think is best.
Overthe last 7 days
red = enabled
blue = disabled
After turning on the plugin again, the CPU is spiking. I’m less confident in the jobs part being a major thing. I noticed the job counts go up with more unicorn workers. I think the number is unrelated.
I do still see the issue about the plugin holding up resources though.
Still not 100% on it but im very confident the plugin is an issue.
plugin + 16 workers: pins the server
16 workers no plugin - working well
plugin + 8 workers - sluggish but it works
graph after turning off the plugin