# Upgrade from 3.2.0.beta3-dev to 3.2.0.beta3 failed due to out of memory

**URL:** <https://meta.discourse.org/t/upgrade-from-3-2-0-beta3-dev-to-3-2-0-beta3-failed-due-to-out-of-memory/286643>\
**Category:** Self-hosting\
**Tags:** server-resources\
**Created:** [November 26, 2023, 6:00pm UTC](https://meta.discourse.org/t/upgrade-from-3-2-0-beta3-dev-to-3-2-0-beta3-failed-due-to-out-of-memory/286643 "2023-11-26T18:00:01Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![andreid](https://avatars.discourse-cdn.com/v4/letter/a/b4bc9f/32.png) [@andreid](https://meta.discourse.org/u/andreid)\
**Post date:** [November 26, 2023, 6:00pm UTC](https://meta.discourse.org/t/upgrade-from-3-2-0-beta3-dev-to-3-2-0-beta3-failed-due-to-out-of-memory/286643/1 "2023-11-26T18:00:01Z")

</div>

Hello,

Tried to upgrade on prompt from 3.2.0.beta3-dev to 3.2.0.beta3 and it broke my Discourse instance due to out of memory during ember build of assets. Tried `./launcher rebuild app` with same result.

```plaintext
FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
 1: 0xb83f50 node::Abort() [ember]
 2: 0xa94834 [ember]
 3: 0xd647c0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [ember]
 4: 0xd64b67 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [ember]
 5: 0xf42265 [ember]
 6: 0xf5474d v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [ember]
 7: 0xf2ee4e v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [ember]
 8: 0xf30217 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [ember]
 9: 0xf113ea v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [ember]
10: 0x12d674f v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [ember]
11: 0x17035b9 [ember]
Aborted (core dumped)
error Command failed with exit code 134.
I, [2023-11-26T17:19:26.345389 #1] INFO -- : yarn run v1.22.19
$ /var/www/discourse/app/assets/javascripts/node_modules/.bin/ember build
Environment: development
WARNING: ember-test-selectors: You are using an unsupported ember-cli-babel version. data-test properties are not automatically stripped from your JS code.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

```

Running on a DigitalOcean instance with 1GB for a non-profit, so I can’t afford to resize it with more memory. 1GB is the minimum size for discourse and previous versions used to run without issues. Any ideas on how to make it run again?

---

<div class="post-metadata">

**Author:** ![itsbhanusharma](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/itsbhanusharma/32/180717_2.png) [@itsbhanusharma](https://meta.discourse.org/u/itsbhanusharma)\
**Post date:** [November 26, 2023, 6:31pm UTC](https://meta.discourse.org/t/upgrade-from-3-2-0-beta3-dev-to-3-2-0-beta3-failed-due-to-out-of-memory/286643/2 "2023-11-26T18:31:11Z")

</div>

Do you have Swap?

What is the output of

```plaintext
free -h

```

---

<div class="post-metadata">

**Author:** ![andreid](https://avatars.discourse-cdn.com/v4/letter/a/b4bc9f/32.png) [@andreid](https://meta.discourse.org/u/andreid)\
**Post date:** [November 26, 2023, 6:40pm UTC](https://meta.discourse.org/t/upgrade-from-3-2-0-beta3-dev-to-3-2-0-beta3-failed-due-to-out-of-memory/286643/3 "2023-11-26T18:40:29Z")

</div>

```plaintext
               total used free shared buff/cache available
Mem: 952Mi 321Mi 414Mi 3.1Mi 374Mi 631Mi
Swap: 2.0Gi 75Mi 1.9Gi

```

---

<div class="post-metadata">

**Author:** ![RGJ](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/rgj/32/523185_2.png) [@RGJ](https://meta.discourse.org/u/RGJ)\
**Post date:** [November 26, 2023, 7:00pm UTC](https://meta.discourse.org/t/upgrade-from-3-2-0-beta3-dev-to-3-2-0-beta3-failed-due-to-out-of-memory/286643/4 "2023-11-26T19:00:09Z")

</div>

> [@andreid](#):
>
> so I can’t afford to resize it with more memory.

You would only need to resize it during the rebuild.

---

<div class="post-metadata">

**Author:** ![itsbhanusharma](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/itsbhanusharma/32/180717_2.png) [@itsbhanusharma](https://meta.discourse.org/u/itsbhanusharma)\
**Post date:** [November 26, 2023, 7:03pm UTC](https://meta.discourse.org/t/upgrade-from-3-2-0-beta3-dev-to-3-2-0-beta3-failed-due-to-out-of-memory/286643/5 "2023-11-26T19:03:54Z")

</div>

You might want to consider moving to Hetzner who offer competitive prices and 2 GB Ram on their base plan

---

<div class="post-metadata">

**Author:** ![JammyDodger](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/jammydodger/32/254611_2.png) [@JammyDodger](https://meta.discourse.org/u/JammyDodger)\
**Post date:** [November 26, 2023, 7:05pm UTC](https://meta.discourse.org/t/upgrade-from-3-2-0-beta3-dev-to-3-2-0-beta3-failed-due-to-out-of-memory/286643/6 "2023-11-26T19:05:06Z")

</div>

Hello and welcome @andreid 🙂

My 1GB DO test site has been struggling with memory issues during rebuilds recently too. I temporarily upgraded to a 2GB just to get it over the line.

---

<div class="post-metadata">

**Author:** ![itsbhanusharma](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/itsbhanusharma/32/180717_2.png) [@itsbhanusharma](https://meta.discourse.org/u/itsbhanusharma)\
**Post date:** [November 26, 2023, 7:07pm UTC](https://meta.discourse.org/t/upgrade-from-3-2-0-beta3-dev-to-3-2-0-beta3-failed-due-to-out-of-memory/286643/7 "2023-11-26T19:07:32Z")

</div>

It might be worth the effort to now update the minimum requirements in docs to 2GB RAM then?

---

<div class="post-metadata">

**Author:** ![JammyDodger](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/jammydodger/32/254611_2.png) [@JammyDodger](https://meta.discourse.org/u/JammyDodger)\
**Post date:** [November 26, 2023, 7:09pm UTC](https://meta.discourse.org/t/upgrade-from-3-2-0-beta3-dev-to-3-2-0-beta3-failed-due-to-out-of-memory/286643/9 "2023-11-26T19:09:04Z")

</div>

I remember it happening last year and some tweaks were made [JavaScript heap out of memory due to Ember CLI - #4 by JammyDodger](https://meta.discourse.org/t/javascript-heap-out-of-memory-due-to-ember-cli/217258/4). I’m not sure if something can be done this time too, but I’ll ask. 👍

---

<div class="post-metadata">

**Author:** ![andreid](https://avatars.discourse-cdn.com/v4/letter/a/b4bc9f/32.png) [@andreid](https://meta.discourse.org/u/andreid)\
**Post date:** [November 26, 2023, 8:39pm UTC](https://meta.discourse.org/t/upgrade-from-3-2-0-beta3-dev-to-3-2-0-beta3-failed-due-to-out-of-memory/286643/10 "2023-11-26T20:39:35Z")

</div>

Thank you @RGJ and @JammyDodger, temporarily resizing it did the trick.

---

<div class="post-metadata">

**Author:** ![Ed\_S](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/ed_s/32/134015_2.png) [@Ed\_S](https://meta.discourse.org/u/Ed_S)\
**Post date:** [November 29, 2023, 10:02pm UTC](https://meta.discourse.org/t/upgrade-from-3-2-0-beta3-dev-to-3-2-0-beta3-failed-due-to-out-of-memory/286643/11 "2023-11-29T22:02:10Z")

</div>

Adding 1G of swap should be functionally the same as adding 1G of RAM, if you have the disk space to do it. (It will probably take longer to run the upgrade, but that’s performance, rather than function. What you desire is to avoid the out of memory situation.)

---

<div class="post-metadata">

**Author:** ![Troid92](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/troid92/32/125272_2.png) [@Troid92](https://meta.discourse.org/u/Troid92)\
**Post date:** [December 11, 2023, 9:30pm UTC](https://meta.discourse.org/t/upgrade-from-3-2-0-beta3-dev-to-3-2-0-beta3-failed-due-to-out-of-memory/286643/13 "2023-12-11T21:30:16Z")

</div>

I have additional info in case it helps mitigate the issue from Discourse’s end. My instance (DigitalOcean ~1GB droplet w/ 2GB swap) recently began taking significantly longer to rebuild and reporting the same fatal error about 3 out of every 4 times (luck seems to improve after running `./launcher cleanup`, but I don’t have enough sample size to confirm this).

Shortly before the heap out of memory error, these lines are logged:

```plaintext
Node.js heap_size_limit (491.0) is less than 1024MB. Setting --max-old-space-size=1024.
Node.js heap_size_limit (491.0) is less than 2048MB. Disabling Webpack parallelization with JOBS=0 to conserve memory.

```

I am out of my domain here, so I apologize if I get something wrong. Some quick research indicates that ember-cli depends on node.js which is why I think this is relevant. The `--max-old-space-size` flag can potentially be set higher than the RAM (it would just go into swap space, which as mentioned is fine for this case), so perhaps 1024 is an artificial ceiling we’re hitting against that Discourse rebuilds can no longer be contained in.

_Side notes: apparently `--optimize-for-size` is a node.js flag which helps reduce memory usage (not sure if it’s being used by Discourse/ember, perhaps it already is), and there is an anecdote out there of the garbage collector not being turned on for certain node.js uses, which may be an issue._

If any of this is relevant and controllable from the Discourse side of ember/node.js usage, it might be worth someone looking into it. If not, no worries, I will do the temporary 2GB upgrade solution proposed above. 🙂

---

<div class="post-metadata">

**Author:** ![david](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/david/32/157490_2.png) [@david](https://meta.discourse.org/u/david)\
**Post date:** [December 11, 2023, 9:38pm UTC](https://meta.discourse.org/t/upgrade-from-3-2-0-beta3-dev-to-3-2-0-beta3-failed-due-to-out-of-memory/286643/14 "2023-12-11T21:38:03Z")

</div>

> [@Troid92](#):
>
> perhaps 1024 is an artificial ceiling we’re hitting against that Discourse rebuilds can no longer be contained in.

That is a very good point! Right now we up it to 1024mb on low-RAM machines [here](https://github.com/discourse/discourse/blob/3999f3e826c80e891794f2bc4c48b818301faedc/lib/tasks/assets.rake#L27C26-L27C66). We could certainly experiment with increasing that to 1500 or 2000 and see if it helps.

If you have the time/inclination to try it out yourself, you could configure it by adding a new variable to the `env:` section of your app.yml file:

Edit: ⚠ this is now the Discourse default. No need to configure yourself

```yml
  NODE_OPTIONS: "--max-old-space-size=2048"

```

---

<div class="post-metadata">

**Author:** ![Troid92](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/troid92/32/125272_2.png) [@Troid92](https://meta.discourse.org/u/Troid92)\
**Post date:** [December 11, 2023, 10:04pm UTC](https://meta.discourse.org/t/upgrade-from-3-2-0-beta3-dev-to-3-2-0-beta3-failed-due-to-out-of-memory/286643/15 "2023-12-11T22:04:59Z")

</div>

Ah, perfect! I went ahead and tried it out.

Since the fatal error doesn’t happen every time, and a rebuild takes about 25 minutes lately (up from 5-10), it may be some time before I know if increasing that number solves the memory issue for these server specs.

But, I can already confirm that the two `Node.js heap_size_limit` warnings no longer appear in the rebuild log, and my first rebuild was successful, so that’s promising.

EDIT: I’ve been able to rebuild several times now with no issues, thanks to the `NODE_OPTIONS` setting above in my app.yml. Yay!

EDIT2: This solution should probably make its way into Discourse by way of increasing [that magic number](https://github.com/discourse/discourse/blob/3999f3e826c80e891794f2bc4c48b818301faedc/lib/tasks/assets.rake#L27C26-L27C66) (link from David’s post) so that other low-RAM machines can continue to operate. If anyone reads this who knows how to do that, that’d be great. 🙂

---

<div class="post-metadata">

**Author:** ![francislavoie](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/francislavoie/32/200225_2.png) [@francislavoie](https://meta.discourse.org/u/francislavoie)\
**Post date:** [December 12, 2023, 8:23am UTC](https://meta.discourse.org/t/upgrade-from-3-2-0-beta3-dev-to-3-2-0-beta3-failed-due-to-out-of-memory/286643/16 "2023-12-12T08:23:37Z")

</div>

We ran into this as well on [https://caddy.community](https://caddy.community).

We ran `./launcher rebuild app` a few times and it failed with various problems.

First we had problems with `bundle install` complaining about `rbtrace` (finishing with `An error occurred while installing rbtrace (0.5.0), and Bundler cannot continue.`)

Then eventually we had this OOM issue:

```plaintext
I, [2023-12-12T07:50:59.497921 #1] INFO -- : > cd /var/www/discourse && su discourse -c 'bundle exec rake themes:update assets:precompile'
Node.js heap_size_limit (1010.0) is less than 1024MB. Setting --max-old-space-size=1024.
Node.js heap_size_limit (1010.0) is less than 2048MB. Disabling Webpack parallelization with JOBS=0 to conserve memory.

<--- Last few GCs --->

[3683:0x5dab130] 279104 ms: Scavenge 981.3 (1037.1) -> 974.5 (1037.1) MB, 8.3 / 0.0 ms (average mu = 0.699, current mu = 0.681) allocation failure; 
[3683:0x5dab130] 279136 ms: Scavenge 981.8 (1037.1) -> 975.0 (1037.1) MB, 8.0 / 0.0 ms (average mu = 0.699, current mu = 0.681) allocation failure; 
[3683:0x5dab130] 282606 ms: Mark-sweep 994.8 (1050.6) -> 987.7 (1048.9) MB, 3316.1 / 0.0 ms (average mu = 0.593, current mu = 0.501) allocation failure; GC in old space requested

<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
 1: 0xb83f50 node::Abort() [ember]
 2: 0xa94834 [ember]
 3: 0xd647c0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [ember]
 4: 0xd64b67 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [ember]
 5: 0xf42265 [ember]
 6: 0xf5474d v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, [snip]
Aborted (core dumped)
error Command failed with exit code 134.

```

And finally running it with `./discourse_doctor` is managed to get past that eventually (why though? more stuff in cache in subsequent runs which made it use less memory? 🤔)

```plaintext
I, [2023-12-12T08:02:50.556442 #1] INFO -- : > cd /var/www/discourse && su discourse -c 'bundle exec rake themes:update assets:precompile'
Node.js heap_size_limit (1010.0) is less than 1024MB. Setting --max-old-space-size=1024.
Node.js heap_size_limit (1010.0) is less than 2048MB. Disabling Webpack parallelization with JOBS=0 to conserve memory.
110:M 12 Dec 2023 08:07:50.026 * 100 changes in 300 seconds. Saving...
110:M 12 Dec 2023 08:07:50.030 * Background saving started by pid 3706
3706:C 12 Dec 2023 08:07:51.292 * DB saved on disk
3706:C 12 Dec 2023 08:07:51.294 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB
110:M 12 Dec 2023 08:07:51.334 * Background saving terminated with success
Purging temp files
Bundling assets

```

But this was friction we shouldn’t have had to run into. Hopefully this improves in the future.

FWIW:

```plaintext
# free -h
              total used free shared buff/cache available
Mem: 1.9Gi 1.3Gi 87Mi 138Mi 593Mi 394Mi
Swap: 2.0Gi 337Mi 1.7Gi

```

---

<div class="post-metadata">

**Author:** ![Falco](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/falco/32/179432_2.png) [@Falco](https://meta.discourse.org/u/Falco)\
**Post date:** [December 12, 2023, 4:21pm UTC](https://meta.discourse.org/t/upgrade-from-3-2-0-beta3-dev-to-3-2-0-beta3-failed-due-to-out-of-memory/286643/17 "2023-12-12T16:21:35Z")

</div>

> [@francislavoie](#):
>
> But this was friction we shouldn’t have had to run into.

Definitely, which is why we are gathering info here.

It appears that tweaking our NODE\_OPTIONS environment variable is all that is needed, so I’d guess that either a dependency of the app or a V8 change made our previous value there not work anymore.

@david how this looks?

[https://github.com/discourse/discourse/pull/24850](https://github.com/discourse/discourse/pull/24850)

---

<div class="post-metadata">

**Author:** ![david](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/david/32/157490_2.png) [@david](https://meta.discourse.org/u/david)\
**Post date:** [December 12, 2023, 5:06pm UTC](https://meta.discourse.org/t/upgrade-from-3-2-0-beta3-dev-to-3-2-0-beta3-failed-due-to-out-of-memory/286643/18 "2023-12-12T17:06:14Z")

</div>

Looks good to me! Obviously 30m+ rebuilds are still not ideal, so I hope we can improve things in the not-too-distant future. But this seems like a good solution to stop the bleeding.

---

<div class="post-metadata">

**Author:** ![volanar](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/volanar/32/318163_2.png) [@volanar](https://meta.discourse.org/u/volanar)\
**Post date:** [December 12, 2023, 5:50pm UTC](https://meta.discourse.org/t/upgrade-from-3-2-0-beta3-dev-to-3-2-0-beta3-failed-due-to-out-of-memory/286643/19 "2023-12-12T17:50:44Z")

</div>

It is worth noting that the increase in postgres version 16 compared to version 13 consumes less space and is much better optimized. This can reduce the total amount of server memory consumed.

> **[Postgres versions 11, 12, 13, 14, 15, and 16 vs sysbench with a medium server](https://smalldatum.blogspot.com/2023/10/postgres-versions-11-12-13-14-15-and-16.html)**
>
> This provides additional results for Postgres versions 11 through 16 vs Sysbench on a medium server. My previous post is here . The goal is ...

---

<div class="post-metadata">

**Author:** ![nathank](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/nathank/32/290039_2.png) [@nathank](https://meta.discourse.org/u/nathank)\
**Post date:** [January 8, 2024, 10:44pm UTC](https://meta.discourse.org/t/upgrade-from-3-2-0-beta3-dev-to-3-2-0-beta3-failed-due-to-out-of-memory/286643/20 "2024-01-08T22:44:01Z")

</div>

I’ve run into a similar rebuild problem today (two container) with a 2GB + 2GB swap setup, for a small site.

Expanding it to 2GB + 4GB swap has gotten it over the line this time.

---

<div class="post-metadata">

**Author:** ![david](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/david/32/157490_2.png) [@david](https://meta.discourse.org/u/david)\
**Post date:** [February 5, 2024, 1:17pm UTC](https://meta.discourse.org/t/upgrade-from-3-2-0-beta3-dev-to-3-2-0-beta3-failed-due-to-out-of-memory/286643/21 "2024-02-05T13:17:47Z")

</div>

2 posts were split to a new topic: [Rebuild is showing “Environment: development” during ember-cli build](https://meta.discourse.org/t/rebuild-is-showing-environment-development-during-ember-cli-build/294183)

---

<div class="post-metadata">

**Author:** ![tophee](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/tophee/32/73406_2.png) [@tophee](https://meta.discourse.org/u/tophee)\
**Post date:** [February 6, 2024, 8:27pm UTC](https://meta.discourse.org/t/upgrade-from-3-2-0-beta3-dev-to-3-2-0-beta3-failed-due-to-out-of-memory/286643/22 "2024-02-06T20:27:51Z")

</div>

FWIW, in my case, adding

> [@david](#):
>
> ` NODE_OPTIONS: "--max-old-space-size=2048"`

to the `app.yml` didn’t help. What helped was simply

```plaintext

sudo apt update
sudo apt upgrade

```

[Next page](https://meta.discourse.org/t/upgrade-from-3-2-0-beta3-dev-to-3-2-0-beta3-failed-due-to-out-of-memory/286643.md?page=2)