It looks like 2+2 may not be enough any more… I’m managing a fairly unassuming (no big/fancy plugins, etc) Discourse instance that, as of today, is failing to bootstrap because ember is chewing all the RAM it can find, and all the swap, and grinding the machine into unresponsiveness. Adding another 2GB of swap allowed the bootstrap to complete, with a peak swap usage of around 2.5GB.

I would quite like to do some controlled experiments, with clean installations and rebuilds, on a variety of setups, and in particular would like to see the difference (if any) of running with vm overcommit, but I’m afraid I’ve lacked the time. (Without overcommit, a large process which forks will have an instantaneous increase in memory footprint which might be fatal, and it won’t show up on a polled monitor. Even with overcommit, memory increase could be rapid enough not to show up on a poll, whether htop or vmstat or something else.) I don’t think I’ve ever seen anyone volunteer whether or not they are running with overcommit, although in my view it’s an important aspect of the host configuration.

[image] Ed_S: I don’t think I’ve ever seen anyone volunteer whether or not they are running with overcommit I bet most people don’t. I set it automatically on my installs. I still get that warning about it, though.

Overcommit is irrelevant here, because the problem isn’t processes being OOM-killed prematurely, it’s just trying to stuff ten pounds of allocated memory into a five pound sack. It’s not practically possible to run a Discourse rebuild with overcommit_memory=2, because ember (amongst other things, no doubt) pre-allocates masses of virtual memory (IIRC, 80GB or so is what I saw), so that’s always going to fall foul of any reasonable overcommit_ratio setting. Setting overcommit_memory=1 isn’t going to help, either, because, again, the problem isn’t an overzealous VMM killing processes, it’s hideously poor memory management from the ember compiler.

I’m not sure I entirely agree with your analysis! As I understand it, overcommit allows processes to allocate memory which they don’t touch. It’s not just about OOM-killer behaviour. But as I say, I’d like to run some controlled experiments, that’s a better way to see what does and doesn’t make a difference.

I have 4GB of RAM and many plugins (no swap file of what I am aware of). How many plugins do you have and do you think plain 4GB of RAM is enough?

Can you confirm that post @david’s changes memory requirements are down? We should be in a reasonable state now. The next big jump is going to be pre-compiling and distributed pre-compiled assets, it is a pretty big change to get there but it will erase a big bunch of work from the Internet once done.

[image] mpalmer: the problem being discussed in this topic is processes allocating memory which they do touch With respect, I’m not sure about that. I’ve seen log files showing failures in forking. We’re saying in this thread that it’s “memory requirements” but that does - in my view - include the kernel’s tactics for virtual memory. Clearly an experiment or three will show whether I’m right or not about overcommit.

That was a fresh build without any plugins. I can try another one with a few plugins enabled and maybe temporarily disable swap to confirm that the build goes through (it’ll probably take a few days until I have time, though).

Ok, next test completed. I built with the following plugins: https://github.com/discourse/docker_manager.git https://github.com/discourse/discourse-data-explorer https://github.com/communiteq/discourse-legal-compliance https://github.com/pfaffman/discourse-allow-pm-to-staff https://github.com/singerscreations/discourse-stopforumspam https://github.com/discourse/discourse-cakeday Swap was disabled, so only 4GiB/3.8GB RAM. Maximum memory usage during build was 3.4GB. Build time was 6m 48s.

High rebuild memory requirements: April 2025 edition

Product Support Self-hosting

mpalmer (Matt Palmer) April 7, 2025, 6:12am 17

Partially correct, but even so, it’s irrelevant, because the problem being discussed in this topic is processes allocating memory which they do touch, and touching more of it than the system has available, which causes customer visible outages.

Topic		Replies	Views
Precompiling assets takes 20 minutes Self-hosting server-resources	18	1788	January 31, 2024
Failed to upgrade discourse instance to Feb 15 2022 Self-hosting server-resources	29	3377	February 16, 2022
Suddenly boot strapping is failing Self-hosting two-container	25	1540	December 30, 2022
Upgrade from 3.2.0.beta3-dev to 3.2.0.beta3 failed due to out of memory Self-hosting server-resources	18	1866	February 6, 2024
Failed to bootstrap due to out of memory killer Bug	35	3130	April 29, 2021

High rebuild memory requirements: April 2025 edition

Related topics