It looks like 2+2 may not be enough any more… I’m managing a fairly unassuming (no big/fancy plugins, etc) Discourse instance that, as of today, is failing to bootstrap because ember is chewing all the RAM it can find, and all the swap, and grinding the machine into unresponsiveness. Adding another 2GB of swap allowed the bootstrap to complete, with a peak swap usage of around 2.5GB.

[image]Create a swapfile for your Linux server [image] sam: we do confirm that as it stands 2gb is enough for docker rebuilds, Like Jagster, 2gb RAM + 2gb swap is not, in fact, enough for my CLI-driven docker rebuild. Checking further, the only plugins on this install are docker_manager Indeed initial install failed in my recent install in 4c 4g instance. Ed suggested creating a swap file. Found the topic to create a swap and created a 4g swap. Now everything working as expected in web or cli update/upgrade Imho we may just need to accept discourse requires more ram than it used to.

We just landed this commit which should hopefully improve the situation. Please let us know how you get on! (it’ll hit tests-passed in the next ~30 mins) When testing with a memory-constrained docker container locally, I can now get a successful build with -m 1600m. Before this change, the minimum I could achieve was -m 3000m.

I did a test rebuild (fresh install), which went through without issue. Now that machine has 4GiB of RAM (Hetzner CAX11) and a swap file of the same size, so it’s certainly less constraint than the 2+2 GB setup mentioned above. However, swap use was minimal during the whole build and the maximum RAM usage I saw was ~3.1GB. And most of the time, it stayed around ~2GB, so it doesn’t look too bad (build time was more or less unchanged, i.e. about 8 minutes). [Maximum RAM usage during build (htop)]

I would quite like to do some controlled experiments, with clean installations and rebuilds, on a variety of setups, and in particular would like to see the difference (if any) of running with vm overcommit, but I’m afraid I’ve lacked the time. (Without overcommit, a large process which forks will have an instantaneous increase in memory footprint which might be fatal, and it won’t show up on a polled monitor. Even with overcommit, memory increase could be rapid enough not to show up on a poll, whether htop or vmstat or something else.) I don’t think I’ve ever seen anyone volunteer whether or not they are running with overcommit, although in my view it’s an important aspect of the host configuration.

[image] Ed_S: I don’t think I’ve ever seen anyone volunteer whether or not they are running with overcommit I bet most people don’t. I set it automatically on my installs. I still get that warning about it, though.

Overcommit is irrelevant here, because the problem isn’t processes being OOM-killed prematurely, it’s just trying to stuff ten pounds of allocated memory into a five pound sack. It’s not practically possible to run a Discourse rebuild with overcommit_memory=2, because ember (amongst other things, no doubt) pre-allocates masses of virtual memory (IIRC, 80GB or so is what I saw), so that’s always going to fall foul of any reasonable overcommit_ratio setting. Setting overcommit_memory=1 isn’t going to help, either, because, again, the problem isn’t an overzealous VMM killing processes, it’s hideously poor memory management from the ember compiler.

I’m not sure I entirely agree with your analysis! As I understand it, overcommit allows processes to allocate memory which they don’t touch. It’s not just about OOM-killer behaviour. But as I say, I’d like to run some controlled experiments, that’s a better way to see what does and doesn’t make a difference.

I have 4GB of RAM and many plugins (no swap file of what I am aware of). How many plugins do you have and do you think plain 4GB of RAM is enough?

[image] Ed_S: As I understand it, overcommit allows processes to allocate memory which they don’t touch. Partially correct, but even so, it’s irrelevant, because the problem being discussed in this topic is processes allocating memory which they do touch, and touching more of it than the system has available, which causes customer visible outages.

High rebuild memory requirements: April 2025 edition

Product Support Self-hosting

pfaffman (Jay Pfaffman) April 5, 2025, 10:28pm 13

I bet most people don’t.

I set it automatically on my installs. I still get that warning about it, though.

Topic		Replies	Views
Precompiling assets takes 20 minutes Self-hosting server-resources	18	1788	January 31, 2024
Failed to upgrade discourse instance to Feb 15 2022 Self-hosting server-resources	29	3377	February 16, 2022
Suddenly boot strapping is failing Self-hosting two-container	25	1540	December 30, 2022
Upgrade from 3.2.0.beta3-dev to 3.2.0.beta3 failed due to out of memory Self-hosting server-resources	18	1866	February 6, 2024
Failed to bootstrap due to out of memory killer Bug	35	3130	April 29, 2021

High rebuild memory requirements: April 2025 edition

Related topics