Memory creep negli ultimi aggiornamenti

Supporto Installazione
1

Negli ultimi settimane ho notato che l’utilizzo della memoria di sistema aumenta gradualmente ogni giorno fino a saturarla.

Storicamente, l’utilizzo della memoria era intorno al 50% - 55% (su un sistema da 3 GB). Ora, dopo un aggiornamento, inizia al 50% ma nei giorni successivi sale lentamente fino all’85% e poi inizia a utilizzare lo swap.

Esiste un modo per individuare cosa in Discourse sta causando questo aumento e consumando memoria? Il task manager mostra solo Ruby che aumenta gradualmente la quantità di memoria che sta utilizzando. Ogni processo Ruby sembra occupare 350 MB e sta crescendo (inizia con meno di 200 MB dopo un aggiornamento).

Aggiornato a v2.3.0.beta9 +392 due giorni fa, l’utilizzo è già passato dal 50% al 75% e non sembra stabilizzarsi.

2

Try updating again. We noticed the same issue, and applied a fix a few hours ago. (commit 1, commit 2)

3

Okay updated and it’s restarted with 47%, will keep an eye on it. Thanks for the quick response.

4

It’s already creeped back up to 61% 64%, the Ruby processes are all now in the range of 310M-340M, will watch it for a day and report back.

Not sure if it’s related but I’m seeing this every night for the past week or so around 1am in the logs:

Sidekiq is consuming too much memory (using: 502.99M)

5

You could try enabling the sidekiq logs, and then look for which job is causing the problem. Some information on those logs can be found in this commit message

https://github.com/discourse/discourse/commit/8963f1af30cd72627e35ce04b317b775435c6f22

6

The memory utilization is back up to 73% and doesnt’ seem to slowing down. It’s now beginning to take up swap space.

I’m not sure how to do this, would need some guidance. I had a look at the commit and it talks about setting 2 environment variables. How do I do this? I’m not familiar with ruby/docker and don’t want mess anything up as this is live.

Is there anything else I can look at to see why the memory utilization is creeping up?

I’m also seeing a new error in the logs after the update (2 since yesterday):

Job exception: post_revision_id

7

Did you do a rebuild? Are you on the default branch of tests-passed?

8

Yes and yes I assume, using the default setup (is there a way to select a different branch?)

9

There is, but that’s the right release to be getting any fixes.

10

@sam is this commit related to this issue? If so is it stable enough to update?

https://github.com/discourse/discourse/commit/76173dea87115589414a200ad7fc1f08a4deb280

11

The issue itself was fixed days ago, it is stable enough to upgrade.

12

Okay updated, I’ll keep an on eye on it, hopefully this will fix it.

I didn’t get what you meant by the issue was fixed days ago. The memory consumption as of this evening is still creeping up.

13

Does this fix require rebuild or can I just upgrade via UI?

14

Via the UI should be fine

15

Okay so I did a update and rebuild last night. The memory usage is back up to 71% and still growing. The only way to reduce it is to restart discourse at which point it drops back down to under 50% and then starts working it’s way up again. The CPU utilization is about 1% on average.

16

What process is growing ? Sidekiq? Unicorn worker? Redis? PG?

17

That’s a good question, which is exactly what I had earlier, how do I find out what’s taking up memory within Discourse? I can only see the task manager which shows that Ruby is taking up more memory with time (all the instances of Ruby are growing in memory consumption).

18

As root run ps aux repeat every few hours

19

Okay when it was taking up 71% memory the top 14 consumers (%mem) are:

PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
49458  0.3  8.0 938568 326016 ?       Sl   May15   4:16 unicorn worker[2] -E production -c config/unicorn.conf.rb
49418  0.6  8.0 1041604 324192 ?      SNl  May15   7:19 sidekiq 5.2.7 discourse [0 of 5 busy]
49448  0.3  7.9 938056 321148 ?       Sl   May15   4:22 unicorn worker[1] -E production -c config/unicorn.conf.rb
49504  0.3  7.9 943692 319948 ?       Sl   May15   4:16 unicorn worker[7] -E production -c config/unicorn.conf.rb
49495  0.3  7.9 928328 319480 ?       Sl   May15   4:21 unicorn worker[6] -E production -c config/unicorn.conf.rb
49476  0.3  7.9 933448 318464 ?       Sl   May15   4:20 unicorn worker[4] -E production -c config/unicorn.conf.rb
49486  0.3  7.8 946768 315236 ?       Sl   May15   4:07 unicorn worker[5] -E production -c config/unicorn.conf.rb
49467  0.3  7.8 928840 315108 ?       Sl   May15   4:05 unicorn worker[3] -E production -c config/unicorn.conf.rb
49439  0.3  7.7 928328 313640 ?       Sl   May15   4:14 unicorn worker[0] -E production -c config/unicorn.conf.rb
49317  0.1  4.8 485628 196588 ?       Sl   May15   2:03 unicorn master -E production -c config/unicorn.conf.rb
49311  0.0  2.4 1263836 96848 ?       Ss   May15   0:08 postgres: 10/main: checkpointer process   
49293  0.0  1.3 1263704 54864 ?       S    May15   0:11 /usr/lib/postgresql/10/bin/postmaster -D /etc/postgresql/10/main
1226  0.0  1.2 280508 49016 tty7     Ssl+ May15   0:21 /usr/lib/xorg/Xorg -core :0 -seat seat0 -auth /var/run/lightdm/root/:0 -nolisten tcp vt7 -novtswitch

After a restart and a couple of grace minutes, it’s showing 50% and the top memory consumers are

PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
17466 17.2  7.5 913964 304276 ?       Sl   16:47   0:09 unicorn worker[1] -E production -c config/unicorn.conf.rb
17494 18.5  7.5 917036 302308 ?       Sl   16:47   0:09 unicorn worker[4] -E production -c config/unicorn.conf.rb
17475 17.8  7.4 913964 301368 ?       Sl   16:47   0:09 unicorn worker[2] -E production -c config/unicorn.conf.rb
17457 15.7  7.3 909244 297984 ?       Sl   16:47   0:08 unicorn worker[0] -E production -c config/unicorn.conf.rb
17522 19.1  7.3 906168 297556 ?       Sl   16:47   0:09 unicorn worker[7] -E production -c config/unicorn.conf.rb
17484 16.7  7.3 906168 297244 ?       Sl   16:47   0:08 unicorn worker[3] -E production -c config/unicorn.conf.rb
17503 18.6  7.3 899000 294548 ?       Sl   16:47   0:09 unicorn worker[5] -E production -c config/unicorn.conf.rb
17512 18.4  7.2 896952 292200 ?       Sl   16:47   0:09 unicorn worker[6] -E production -c config/unicorn.conf.rb
17303 13.0  4.8 477436 194544 ?       Sl   16:46   0:13 unicorn master -E production -c config/unicorn.conf.rb
17435  0.9  4.5 554280 182640 ?       SNl  16:47   0:00 sidekiq 5.2.7 discourse [0 of 5 busy]
17267  0.0  1.4 1263704 57740 ?       S    16:46   0:00 /usr/lib/postgresql/10/bin/postmaster -D /etc/postgresql/10/main
1226  0.0  1.2 280508 48464 tty7     Ssl+ May15   0:22 /usr/lib/xorg/Xorg -core :0 -seat seat0 -auth /var/run/lightdm/root/:0 -nolisten tcp vt7 -novtswitch
1447  0.3  1.2 776896 48360 ?        Ssl  May15   5:57 /usr/bin/dockerd -H fd://

Looks like sidekiq, some of the unicorn workers and the postgres.

Let me know if you would like me to collect any other data.

20

You are running too many unicorns, those numbers look right to me 300-500 per worker is in the normal range

Cut unicorn count down by 3