High memory use during assets precompilation

Development

AI-generated summary

High memory use during assets precompilation

Rotonen reports high memory usage during assets precompilation using Bundler, which may be related to a VM object refcount issue or an artifact of Bundler or the Rails asset pipeline.

Rotonen shares a process list showing high memory usage by the bundle command and multiple ruby processes.

sam mentions that the issue is known and that upgrading to Sprockets 4 may worsen the problem. However, there are plans to move away from Sprockets altogether, which would resolve the issue.

Rotonen notes that adjusting Ruby garbage collection parameters can help mitigate the issue, and davidnmbond asks for progress on resolving the issue, reporting a case where 14GB of memory is allocated before the process runs out of memory.

Ed_S suggests that allocated memory may not be the issue, and that resident set size (RSS) is a more important metric. Ed_S also recommends using vmstat to monitor memory pressure and swap usage.

1

During upgrades Bundler uses a lot of memory in relation to the rest of the installation during the assets precompilation. This would hint at some VM object refcount thing not releasing the touched objects (files having been read in?) and thus the GC not firing and doing its job. Most likely this is an artefact of either Bundler itself or the Rails asset pipeline.

PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
3952 1000      20   0 3636888 651740  14836 R  86,8 31,8   2:23.24 bundle
3433 1000      20   0 1681992 190036   4776 S   0,0  9,3   0:05.55 ruby
3403 1000      20   0 1688156 172824   4812 S   0,0  8,4   0:05.95 ruby
3391 1000      20   0 1687128 143528   5100 S   0,0  7,0   0:05.95 ruby
3366 1000      20   0  510236 105780   4528 S   0,3  5,2   0:02.84 ruby
2920 1000      20   0  465132  92584   6804 S   0,0  4,5   0:11.59 ruby

For your consideration to maybe take a deeper look at this, as it is causing annoying shuffling for sysadmins with other processes on servers at upgrade times if the servers are provisioned for the runtime memory requirements.

2

@sgrif is painfully aware of this :slight_smile:

The bad news, at the moment stuff is looking even worse if we upgrade to sprockets 4 (we are on 3)

The good news, some time this year we plan to move off sprockets altogether and make this a non-issue.

3

I figured you would hit this bit of juggling on your hosted services and hoped you would have cooked up a pattern in-house.

If you have a public ticket on an open source component for that at some point, do cross post here - I’m willing to help tackle this (design-side, implementation-side, just general input, metrification, testing).

Otherwise may the UNIX pipe (or Goroutine or equivalent) be with you.

4

RUBY_GC_MALLOC_LIMIT_MAX=20971520 RUBY_GC_OLDMALLOC_LIMIT_MAX=20971520 RUBY_GC_HEAP_GROWTH_MAX_SLOTS=50000 RUBY_GC_HEAP_OLDOBJECT_LIMIT_FACTOR=0.9 bundle exec rake assets:precompile

Seems you fixed this via GC parametres at some point as of late, thank you.

5

Ci sono progressi su questo problema? Stiamo vedendo 14 GB di memoria allocata quando creiamo un’istanza VUOTA di 3.2.1 prima che esaurisca la memoria e muoia.

6

Potresti condividere l’output di vmstat 5 in questo caso? Se stai guardando ps o top potresti vedere qualcosa di diverso dalla memoria in uso. La memoria allocata non è così importante se non è in uso.

Ad esempio, nell’output di ps vedo VSZ e RSS. Mi interessa l’RSS.

# ps uax|egrep '[0-9].unicorn|VS[Z]'
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
ddiscou+  2192  0.0 13.4 558320 267540 ?       Sl   Jan10  33:21 unicorn master -E production -c config/unicorn.conf.rb
ddiscou+  2294  0.0 18.2 5126600 363432 ?      Sl   Jan10  90:59 unicorn worker[0] -E production -c config/unicorn.conf.rb
ddiscou+  2301  0.0 14.2 5149576 282892 ?      Sl   Jan10  38:40 unicorn worker[1] -E production -c config/unicorn.conf.rb

In top vedo la stessa cosa ma si chiama VIRT e RES

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
 2284 ddiscou+  25   5 5246644 327496   5640 S  1.7 16.4 872:50.70 ruby
 2294 ddiscou+  20   0 5126600 364284  14436 S  0.7 18.3  91:00.13 ruby

vmstat mi dice qual è la pressione della memoria in questo momento: quanta swap rimane, quanta memoria libera e buffer, e soprattutto se c’è qualche swap-in (la colonna si)

root@ubuntu-2gb-nbg1-1:~# vmstat 5 5
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0 239616 127080 185428 464448    0    0    14    32    4    2  2  1 97  0  0
 0  0 239616 124584 185432 464500    0    0     0    13  138  318  2  2 96  0  0
 0  0 239616 126120 185436 464500    0    0     0    13  116  264  1  0 99  0  0
 0  0 239616 120376 185436 464504    0    0     0    46  125  322  3  2 95  0  0
 0  0 239616 123448 185436 464504    0    0     0    73  129  306  1  1 98  0  0