Failed to bootstrap due to out of memory killer

I am having a problem that might be related over at Trouble with `tests/test_helper`? - #2 by Falco.

I tried to rebuild app without any plugins. No change. Same error.

I don’t understand this, but it looks like a bug. I’m trying to do a bootstrap on this site. No non-standard plugins. I just moved assets from one bucket to another and it’s all working. I was doing one more rebuild to add DISCOURSE_S3_UPLOAD_BUCKET to the ENV so that it won’t show up in the UX. When this failed the first time, I commented out that line and tried again with the same config that worked 3 days ago.


Done compressing embed-application-9cef8308c816fc1d83137e63d6c556c6cc2b68fe2b6e5ce16cca6766ba2c0ae4.js : 0.17 secs

844614.350963717 Compressing: discourse/tests/test_helper-8590b31b8e73c4172aeea4a4a6bd1930ccbce2547a20d831a30d457ba092a631.js
terser '/var/www/discourse/public/assets/discourse/tests/_test_helper-8590b31b8e73c4172aeea4a4a6bd1930ccbce2547a20d831a30d457ba092a631.js' -m -c -o '/var/www/discourse/public/assets/discourse/tests/test_helper-8590b31b8e73c4172aeea4a4a6bd1930ccbce2547a20d831a30d457ba092a631.js' --source-map "base='/var/www/discourse/public/assets/discourse/tests',root='/assets/discourse/tests',url='https://CORRECT_CDN_ADDRESS.b-cdn.net/assets/discourse/tests/test_helper-8590b31b8e73c4172aeea4a4a6bd1930ccbce2547a20d831a30d457ba092a631.js.map'"
Killed
rake aborted!
Errno::ENOENT: No such file or directory @ rb_file_s_size - /var/www/discourse/public/assets/discourse/tests/test_helper-8590b31b8e73c4172aeea4a4a6bd1930ccbce2547a20d831a30d457ba092a631.js
/var/www/discourse/lib/tasks/assets.rake:290:in `size'
/var/www/discourse/lib/tasks/assets.rake:290:in `block (4 levels) in <main>'
/var/www/discourse/lib/tasks/assets.rake:181:in `block in concurrent?'
/var/www/discourse/lib/tasks/assets.rake:281:in `block (3 levels) in <main>'
/var/www/discourse/lib/tasks/assets.rake:272:in `each'
/var/www/discourse/lib/tasks/assets.rake:272:in `block (2 levels) in <main>'
/var/www/discourse/lib/tasks/assets.rake:181:in `concurrent?'
/var/www/discourse/lib/tasks/assets.rake:269:in `block in <main>'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/rake-13.0.3/exe/rake:27:in `<top (required)>'
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'
Tasks: TOP => assets:precompile
(See full trace by running task with --trace)
I, [2021-04-26T18:38:36.072881 #1]  INFO -- : Updating Discourse Loading Slider...
Downloading MaxMindDB...
Compressing Javascript and Generating Source Maps

I wondered if there was some problem with the CDN url, but all the lines above included it and worked fine.

Does this mean there’s faulty CSS in their theme? If so, ouch. There’s only some CSS in their master theme. Also these components: https://github.com/discourse/DiscoTOC.git and https://github.com/davidtaylorhq/discourse-loading-slider.git

Is this a minimum size droplet? Looks like that file is kinda hard for terser to compress as it causes a lot of memory pressure.

Oh. It is surprisingly small. It’s a site that I think you guys set up some years ago (when you’d still run a site not on your infrastructure).

root@community:/var/discourse# free -h
              total        used        free      shared  buff/cache   available
Mem:           1.9G        1.2G        101M        259M        655M        354M
Swap:          2.0G        1.2G        773M

Ah. OK. It’ s 2GB DO droplet, and I have access to their control panel. I’ll tell them that we need to upgrade to 4GB and move them to an AMD.

EDIT: But if this is just to compress that one file, shouldn’t a 2GB droplet be enough?

This test-helper file is hard on the compression.

  • UglifyJS uses 1.5GB RAM to compress it.

  • Terser uses a little over 1GB. Takes 40s. For comparison the same server takes 8s on Ember+jQuery :scream:

@eviltrout should we even have this file in production?

Ohh looks like it comes from this change from @Osama

https://github.com/discourse/discourse/pull/12815

-rw-r--r-- 1 discourse discourse  14M Apr 26 19:13 _test_helper-f4c4b5bf0657eab910d85b9a65b4bddbbbe2ce2ba603b17fe11b3d633d324e34.js
-rw-r--r-- 1 discourse discourse 6.6M Apr 26 19:14 test_helper-f4c4b5bf0657eab910d85b9a65b4bddbbbe2ce2ba603b17fe11b3d633d324e34.js
-rw-r--r-- 1 discourse discourse 1.1M Apr 26 19:14 test_helper-f4c4b5bf0657eab910d85b9a65b4bddbbbe2ce2ba603b17fe11b3d633d324e34.js.br
-rw-r--r-- 1 discourse discourse 1.5M Apr 26 19:14 test_helper-f4c4b5bf0657eab910d85b9a65b4bddbbbe2ce2ba603b17fe11b3d633d324e34.js.gz
-rw-r--r-- 1 discourse discourse 5.7M Apr 26 19:14 test_helper-f4c4b5bf0657eab910d85b9a65b4bddbbbe2ce2ba603b17fe11b3d633d324e34.js.map
8 Likes

Just to add another data point - still experiencing the rebuild failures after removing the two theme components. So just using the default Light theme.

also where is this output from - looking to verify/debug a bit on my end as well. is this like a verbose option on the ./launcher rebuild app?

Fixing this properly is going to take a little while, so I’m going to revert that change for now.

9 Likes

It may be a good idea to put it behind a feature flag afterwards so we can iterate on it internally.

3 Likes

Ok it’s been reverted and the commit is in tests-passed now:

https://github.com/discourse/discourse/commit/a169dc6832fb90bcec664b0e8bf52026ce0dae3f

Thanks for letting us know and sorry for the inconvenience.

7 Likes

I’d be interested to hear if this revert has also solved the OP’s problem. The OP (@devnull ) reported seeing

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

which wasn’t reported by @pfaffman

Also I’d like to comment on this:

Adding RAM can sometimes improve performance, but for actually running out of memory, what counts is RAM+swap. If adding swap doesn’t help with an out of memory condition, then adding RAM won’t either.

As @weallwegot reports the failure persists even after adding 4G of swap, something is (or was) very hungry indeed.

2 Likes

It’s the same error. Reading the backtrace shows a reference to OOM on the fourth line.

3 Likes

All of that is true. He has < 1GB of ram, which isn’t much. So even if he didn’t have this problem, I’d still recommend that he add more RAM. For that matter, I mostly recommend that anyone with <2GB of ram increase their RAM.

1 Like

The problem is resolved for me. Rebuild App works. Thank you very much for your effort!

3 Likes

(Absolutely, I understand the advice, it’s the logic which seemed like it might mislead.)

2 Likes

Was able to rebuild with both of those plugins with no issue by the way. Although now i have 6G of swap so not quite apples to apples lol.

4 Likes

Thanks! I’m back in business.

6 Likes

FYI I’ve just merged that change again but this time it should not break bootstrap. Please let me know if this causes any problems.

2 Likes

I just now had the same type of failure in bootstrap that @pfaffman described… parse error at theme_test_helper. Let me know if you need more info.

1 Like

Can you copy and paste the error here please? Also the output lines around compressing theme_test_helper and theme_test_vendor, they should look like this:

12825804.271282336 Compressing: discourse/tests/theme_test_helper-790dafad7d3cb8a853fd3127fa3d99022120baf968cbb297066d166808ad3ae1.js
gzip -f -c -9 /var/www/discourse/public/assets/discourse/tests/theme_test_helper-790dafad7d3cb8a853fd3127fa3d99022120baf968cbb297066d166808ad3ae1.js > /var/www/discourse/public/assets/discourse/tests/theme_test_helper-790dafad7d3cb8a853fd3127fa3d99022120baf968cbb297066d166808ad3ae1.js.gz

brotli -f --quality=11 /var/www/discourse/public/assets/discourse/tests/theme_test_helper-790dafad7d3cb8a853fd3127fa3d99022120baf968cbb297066d166808ad3ae1.js --output=/var/www/discourse/public/assets/discourse/tests/theme_test_helper-790dafad7d3cb8a853fd3127fa3d99022120baf968cbb297066d166808ad3ae1.js.br
Done compressing discourse/tests/theme_test_helper-790dafad7d3cb8a853fd3127fa3d99022120baf968cbb297066d166808ad3ae1.js : 6.85 secs