It’s normal for one pitchfork process to take most of the load - other processes are only picked when the first one is busy. It’s also normal for memory to grow over time. Although it should eventually reach a steady state, once everything like caches/ruby-JIT are warmed up.
Here’s a graph from one of our production clusters, over a 7-day period. This tracks the single process with the most memory usage. Green-dotted lines indicate deploys (i.e. fresh starts of pitchfork):
So you can see, it does grow after initial launch, but then it settles at just under 1.4GB. This cluster is serving hundreds of sites, so perhaps not the best comparison in terms of numbers… but hopefully the visualisation of the growth is useful.
Looking at per-process RSS numbers doesn’t tell the whole story. Pitchfork is a “forking web server”. So it boots up the ‘mold’ process, and then all of the workers are forked from that with a copy-on-write memory pattern. So, while the RSS might show 500mb for worker[0], ~200mb of that is shared with the mold and all the other workers.
In the not-too-distant future we’re hoping to enable Pitchfork’s “reforking” feature. It periodically kills workers, and re-forks them from an already-warmed-up worker, so they share as much memory as possible over time. Some work needed to make sure that all our code is compatible with that kind of pattern though 