Failed to bootstrap due to out of memory killer

Okay, but can you tell us what’s the reason for that new behavior? It’s working for more than one year with that setup. What makes Discourse different now than it has been in the past?

1 Like

thanks for the recommendation. yea slightly strange i’ve usually gotten away with default 2GB swap file on a $5 Digital ocean droplet. will keep an eye out on if this becomes more common with latest updates or something.

anyways i went ahead and added a lot more swap (4 G) in a separate file

but the upgrade still failed. maybe more RAM is non-negotiable. just unexpected because the instance has 1 topic and 1 user at the moment. also curious if i need to do anything to make sure discourse knows to use the swap or if it is just accessible by default?

my new free -m output…

              total        used        free      shared  buff/cache   available
Mem:            981         138         576           0         266         703
Swap:          6143         109        6034
1 Like

Yep, I’m on a Digital Ocean Droplet with 1 vCPU and 1GB vRAM as well :slight_smile:

2 Likes

I have increased the Swap memory onto 3GB. It’s still not working with same error.

1 Like

I’m able to rebuild my test instance with only 2.5GiB of RAM+swap. It’s possible that your instance is requiring more than that, however.

Do you have any plugins installed? I suspect that one of them is causing massive memory use during compilation.

Can you rebuild without plugins and see if that fixes the problem?

Thanks for chiming in-

Out of curiosity what is the break down of RAM vs swap? and are you only counting what is “free” space on both of those or the total swap file size + total RAM of instance?

Oh right of course - i had forget to mention i was hoping to install the Discourse OpenID Connect Authentication plugin

Currently also have the Data Explorer plugin already installed.


  • Tried again with only Data Explorer + Docker Manager and no luck, same stacktrace as shared before.
  • Tried again with no plugins (just Docker Manager) and the rebuild still didn’t work.

will keep looking around since outside of trying to add the ConnectID plugin i haven’t changed anything since original install

I am having a problem that might be related over at Trouble with `tests/test_helper`? - #2 by Falco.

I tried to rebuild app without any plugins. No change. Same error.

I don’t understand this, but it looks like a bug. I’m trying to do a bootstrap on this site. No non-standard plugins. I just moved assets from one bucket to another and it’s all working. I was doing one more rebuild to add DISCOURSE_S3_UPLOAD_BUCKET to the ENV so that it won’t show up in the UX. When this failed the first time, I commented out that line and tried again with the same config that worked 3 days ago.


Done compressing embed-application-9cef8308c816fc1d83137e63d6c556c6cc2b68fe2b6e5ce16cca6766ba2c0ae4.js : 0.17 secs

844614.350963717 Compressing: discourse/tests/test_helper-8590b31b8e73c4172aeea4a4a6bd1930ccbce2547a20d831a30d457ba092a631.js
terser '/var/www/discourse/public/assets/discourse/tests/_test_helper-8590b31b8e73c4172aeea4a4a6bd1930ccbce2547a20d831a30d457ba092a631.js' -m -c -o '/var/www/discourse/public/assets/discourse/tests/test_helper-8590b31b8e73c4172aeea4a4a6bd1930ccbce2547a20d831a30d457ba092a631.js' --source-map "base='/var/www/discourse/public/assets/discourse/tests',root='/assets/discourse/tests',url='https://CORRECT_CDN_ADDRESS.b-cdn.net/assets/discourse/tests/test_helper-8590b31b8e73c4172aeea4a4a6bd1930ccbce2547a20d831a30d457ba092a631.js.map'"
Killed
rake aborted!
Errno::ENOENT: No such file or directory @ rb_file_s_size - /var/www/discourse/public/assets/discourse/tests/test_helper-8590b31b8e73c4172aeea4a4a6bd1930ccbce2547a20d831a30d457ba092a631.js
/var/www/discourse/lib/tasks/assets.rake:290:in `size'
/var/www/discourse/lib/tasks/assets.rake:290:in `block (4 levels) in <main>'
/var/www/discourse/lib/tasks/assets.rake:181:in `block in concurrent?'
/var/www/discourse/lib/tasks/assets.rake:281:in `block (3 levels) in <main>'
/var/www/discourse/lib/tasks/assets.rake:272:in `each'
/var/www/discourse/lib/tasks/assets.rake:272:in `block (2 levels) in <main>'
/var/www/discourse/lib/tasks/assets.rake:181:in `concurrent?'
/var/www/discourse/lib/tasks/assets.rake:269:in `block in <main>'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/rake-13.0.3/exe/rake:27:in `<top (required)>'
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'
Tasks: TOP => assets:precompile
(See full trace by running task with --trace)
I, [2021-04-26T18:38:36.072881 #1]  INFO -- : Updating Discourse Loading Slider...
Downloading MaxMindDB...
Compressing Javascript and Generating Source Maps

I wondered if there was some problem with the CDN url, but all the lines above included it and worked fine.

Does this mean there’s faulty CSS in their theme? If so, ouch. There’s only some CSS in their master theme. Also these components: https://github.com/discourse/DiscoTOC.git and https://github.com/davidtaylorhq/discourse-loading-slider.git

Is this a minimum size droplet? Looks like that file is kinda hard for terser to compress as it causes a lot of memory pressure.

Oh. It is surprisingly small. It’s a site that I think you guys set up some years ago (when you’d still run a site not on your infrastructure).

root@community:/var/discourse# free -h
              total        used        free      shared  buff/cache   available
Mem:           1.9G        1.2G        101M        259M        655M        354M
Swap:          2.0G        1.2G        773M

Ah. OK. It’ s 2GB DO droplet, and I have access to their control panel. I’ll tell them that we need to upgrade to 4GB and move them to an AMD.

EDIT: But if this is just to compress that one file, shouldn’t a 2GB droplet be enough?

This test-helper file is hard on the compression.

  • UglifyJS uses 1.5GB RAM to compress it.

  • Terser uses a little over 1GB. Takes 40s. For comparison the same server takes 8s on Ember+jQuery :scream:

@eviltrout should we even have this file in production?

Ohh looks like it comes from this change from @Osama

https://github.com/discourse/discourse/pull/12815

-rw-r--r-- 1 discourse discourse  14M Apr 26 19:13 _test_helper-f4c4b5bf0657eab910d85b9a65b4bddbbbe2ce2ba603b17fe11b3d633d324e34.js
-rw-r--r-- 1 discourse discourse 6.6M Apr 26 19:14 test_helper-f4c4b5bf0657eab910d85b9a65b4bddbbbe2ce2ba603b17fe11b3d633d324e34.js
-rw-r--r-- 1 discourse discourse 1.1M Apr 26 19:14 test_helper-f4c4b5bf0657eab910d85b9a65b4bddbbbe2ce2ba603b17fe11b3d633d324e34.js.br
-rw-r--r-- 1 discourse discourse 1.5M Apr 26 19:14 test_helper-f4c4b5bf0657eab910d85b9a65b4bddbbbe2ce2ba603b17fe11b3d633d324e34.js.gz
-rw-r--r-- 1 discourse discourse 5.7M Apr 26 19:14 test_helper-f4c4b5bf0657eab910d85b9a65b4bddbbbe2ce2ba603b17fe11b3d633d324e34.js.map
8 Likes

Just to add another data point - still experiencing the rebuild failures after removing the two theme components. So just using the default Light theme.

also where is this output from - looking to verify/debug a bit on my end as well. is this like a verbose option on the ./launcher rebuild app?

Fixing this properly is going to take a little while, so I’m going to revert that change for now.

9 Likes

It may be a good idea to put it behind a feature flag afterwards so we can iterate on it internally.

3 Likes

Ok it’s been reverted and the commit is in tests-passed now:

https://github.com/discourse/discourse/commit/a169dc6832fb90bcec664b0e8bf52026ce0dae3f

Thanks for letting us know and sorry for the inconvenience.

7 Likes

I’d be interested to hear if this revert has also solved the OP’s problem. The OP (@devnull ) reported seeing

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

which wasn’t reported by @pfaffman

Also I’d like to comment on this:

Adding RAM can sometimes improve performance, but for actually running out of memory, what counts is RAM+swap. If adding swap doesn’t help with an out of memory condition, then adding RAM won’t either.

As @weallwegot reports the failure persists even after adding 4G of swap, something is (or was) very hungry indeed.

2 Likes

It’s the same error. Reading the backtrace shows a reference to OOM on the fourth line.

3 Likes

All of that is true. He has < 1GB of ram, which isn’t much. So even if he didn’t have this problem, I’d still recommend that he add more RAM. For that matter, I mostly recommend that anyone with <2GB of ram increase their RAM.

1 Like

The problem is resolved for me. Rebuild App works. Thank you very much for your effort!

3 Likes