Memory creep in last couple of updates

Installation Hosting
22

Thanks

This is a default installation, there are no customizations other than adding 3 official plugins to it. So part of the question is, did the change a recent update which is why it’s beginning to creep up now?

What about sidekiq, is it normal for it to almost double memory consumption over time?

I’ve seen this question asked in a couple of places but didn’t find the answer? Is this the UNICORN_WORKERS in app.yml
I see it’s currently set to 8 and I’m guessing from what I read on the forum this number is set when it rebuilds based on the number on CPU, which in this case is 4, so I’m struggling to understand why the memory consumption started increasing lately if there’s been no changes to the hardware/setup?

23

Okay an update, after the last fork unicorn patch, the utilization seems to have stabilized around 73% (higher than before but atleast not going up past 85%)

I’m reducing the number of unicorns from 8 to 5 as suggested and will let you know how it goes.

I’m still not clear as to why SideKiq is increasing its memory consumption and if something can/should be done about it or if it should be ignored.

Thanks for your help

2 Likes
24

How high is sidekiq? We have protection in place in case it goes above 500

2 Likes
25

At last check ps aux reported this (when utilization was 71%)

49418  0.6  8.0 1041604 324192 ?      SNl  May15   7:19 sidekiq 5.2.7 discourse [0 of 5 busy]

However in the error logs I did see sidekiq being restarted every night around midnight with this error:

Sidekiq is consuming too much memory (using: 502.99M)

I’ll continue to keep an eye on it

1 Like
26

@sam don’t we have sidekiq logging now for better tracking this courtesy of @david?

2 Likes
27

We have better logging in that we know what jobs ran, but finding memory leaks is still a very involved process. I will do an internal review to see if we are seeing this on any of our sites.

6 Likes
28

Just an update here, after reducing the number of unicorn workers from 8 to 5 as suggested the memory utilization has now stabilized at 57% and this is the output of ps aux --sort -%mem

PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
8555  0.6  8.4 1114316 341412 ?      SNl  May17  24:03 sidekiq 5.2.7 
8595  0.4  8.3 958024 334536 ?       Sl   May17  16:10 unicorn worker
8614  0.4  8.1 950856 329196 ?       Sl   May17  16:01 unicorn worker
8604  0.4  8.1 955980 328432 ?       Sl   May17  15:56 unicorn worker
8586  0.4  8.0 958544 323008 ?       Sl   May17  15:58 unicorn worker
8577  0.4  7.9 1072200 321624 ?      Sl   May17  16:05 unicorn worker
8446  0.1  4.9 481532 197740 ?       Sl   May17   6:03 unicorn master

@sam a follow up clarification on the impact of reducing the number of unicorn workers. I was reading on the forum that each worker supports upto 5 jobs, so with 5 workers that’s 25 jobs. I read somewhere else that each worker is good for about 400 connections. I’m not really clear on what this means for system scalability, I suspect it’ll be fine but it would nice if you could outline what the unicorn workers are used for and how many concurrent users (rough ballpark) could the system support with 5 workers. Thanks in advance.

2 Likes
29

Sidekiq is the thing that does the jobs.

Unicorn is for web requests. Running 5 means you can handle 5 “slow” web requests concurrently. Certain web requests like “message bus” and “avatar caching” and “uploads” run in background threads, so the real number of concurrent requests tends to be a lot higher.

“concurrent users” highly depends on what the users are doing and logged in vs anon which is heavily cached. I can’t really provide a particular guideline, but I can provide instrumentation that will tell you if there is a problem.

7 Likes