Rotonen reports high memory usage during assets precompilation using Bundler, which may be related to a VM object refcount issue or an artifact of Bundler or the Rails asset pipeline.
Rotonen shares a process list showing high memory usage by the bundle command and multiple ruby processes.
sam mentions that the issue is known and that upgrading to Sprockets 4 may worsen the problem. However, there are plans to move away from Sprockets altogether, which would resolve the issue.
Rotonen notes that adjusting Ruby garbage collection parameters can help mitigate the issue, and davidnmbond asks for progress on resolving the issue, reporting a case where 14GB of memory is allocated before the process runs out of memory.
Ed_S suggests that allocated memory may not be the issue, and that resident set size (RSS) is a more important metric. Ed_S also recommends using vmstat to monitor memory pressure and swap usage.
During upgrades Bundler uses a lot of memory in relation to the rest of the installation during the assets precompilation. This would hint at some VM object refcount thing not releasing the touched objects (files having been read in?) and thus the GC not firing and doing its job. Most likely this is an artefact of either Bundler itself or the Rails asset pipeline.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3952 1000 20 0 3636888 651740 14836 R 86,8 31,8 2:23.24 bundle
3433 1000 20 0 1681992 190036 4776 S 0,0 9,3 0:05.55 ruby
3403 1000 20 0 1688156 172824 4812 S 0,0 8,4 0:05.95 ruby
3391 1000 20 0 1687128 143528 5100 S 0,0 7,0 0:05.95 ruby
3366 1000 20 0 510236 105780 4528 S 0,3 5,2 0:02.84 ruby
2920 1000 20 0 465132 92584 6804 S 0,0 4,5 0:11.59 ruby
For your consideration to maybe take a deeper look at this, as it is causing annoying shuffling for sysadmins with other processes on servers at upgrade times if the servers are provisioned for the runtime memory requirements.
I figured you would hit this bit of juggling on your hosted services and hoped you would have cooked up a pattern in-house.
If you have a public ticket on an open source component for that at some point, do cross post here - I’m willing to help tackle this (design-side, implementation-side, just general input, metrification, testing).
Otherwise may the UNIX pipe (or Goroutine or equivalent) be with you.
