Does anyone else see this? It’s a pretty modest forum in size and traffic. There are four pitchfork worker processes, it looks like one of them handles the great majority of incoming requests and is using more memory than the others. There’s lots of swap in use, and by deliberately respawning pitchfork this reduces considerably. So I conclude the pitchfork processes have a lot of cold pages presently swapped out.
This is after 20 days of operation, on a machine with 4G RAM and 4G swap.
It wouldn’t be good to run out of swap. It wouldn’t be good to find all that swap being paged back in, as might possibly happen when pitchfork eventually does a garbage collect.
Anyone else see anything similar? Please share machine config!
The vmstat output shows that there’s not presently any memory pressure - no paging activity. My concern is that there’s a danger, not that my installation is presently working poorly.
I’d be interested to see the stats from a more busy forum. It seems that the cumulative number of requests is important, and mine is by no means very busy.
Yes, the RSS differences are relatively modest, but ranging from 300M to 500M there’s quite a difference in proportion.
More striking is the swap usage. Before recycling the pitchforks, I see
# free
total used free shared buff/cache available
Mem: 3904968 1888140 77308 679004 1939520 1147220
Swap: 4194288 1226344 2967944
and after
# free
total used free shared buff/cache available
Mem: 3904968 1486412 1422380 640744 996176 1587908
Swap: 4194288 309892 3884396
Looks to me like 1300M freed by recycling (difference of used+swap)
I don’t believe I saw this much swap needed when using unicorn: hence the title.
As I say, interested in seeing numbers from other instances.
It’s normal for one pitchfork process to take most of the load - other processes are only picked when the first one is busy. It’s also normal for memory to grow over time. Although it should eventually reach a steady state, once everything like caches/ruby-JIT are warmed up.
Here’s a graph from one of our production clusters, over a 7-day period. This tracks the single process with the most memory usage. Green-dotted lines indicate deploys (i.e. fresh starts of pitchfork):
So you can see, it does grow after initial launch, but then it settles at just under 1.4GB. This cluster is serving hundreds of sites, so perhaps not the best comparison in terms of numbers… but hopefully the visualisation of the growth is useful.
Looking at per-process RSS numbers doesn’t tell the whole story. Pitchfork is a “forking web server”. So it boots up the ‘mold’ process, and then all of the workers are forked from that with a copy-on-write memory pattern. So, while the RSS might show 500mb for worker[0], ~200mb of that is shared with the mold and all the other workers.
In the not-too-distant future we’re hoping to enable Pitchfork’s “reforking” feature. It periodically kills workers, and re-forks them from an already-warmed-up worker, so they share as much memory as possible over time. Some work needed to make sure that all our code is compatible with that kind of pattern though
Useful visualisation, thanks. I wouldn’t say that had settled, but the growth rate isn’t as high as it has been.
The periodic reforking sounds like a great thing to bring it - if you could, please note it here when that lands.
Can you say anything about the green lines? I’m interested in when (if ever) the pitchforks do a garbage collect, what causes it and what the effect is while it’s doing it.
The green lines are when we deploy updates. Essentially: ./launcher rebuild app.
Ruby (and JS, which we run inside the Ruby processes via MiniRacer/v8) are constantly garbage-collecting while they’re running, and in response to signals from the operating system. So I don’t think there’s any large benefit to be gained from doing anything manually.
Thanks. I’d be interested to see how things work out in the long run - which won’t be evident on a server that’s being updated relatively often. In my case, I had 20 days, then I chose to get pitchfork to restart, which told me something but also reset the clock.
