Failed to bootstrap due to out of memory killer

This test-helper file is hard on the compression.

  • UglifyJS uses 1.5GB RAM to compress it.

  • Terser uses a little over 1GB. Takes 40s. For comparison the same server takes 8s on Ember+jQuery :scream:

@eviltrout should we even have this file in production?

Ohh looks like it comes from this change from @Osama

-rw-r--r-- 1 discourse discourse  14M Apr 26 19:13 _test_helper-f4c4b5bf0657eab910d85b9a65b4bddbbbe2ce2ba603b17fe11b3d633d324e34.js
-rw-r--r-- 1 discourse discourse 6.6M Apr 26 19:14 test_helper-f4c4b5bf0657eab910d85b9a65b4bddbbbe2ce2ba603b17fe11b3d633d324e34.js
-rw-r--r-- 1 discourse discourse 1.1M Apr 26 19:14
-rw-r--r-- 1 discourse discourse 1.5M Apr 26 19:14 test_helper-f4c4b5bf0657eab910d85b9a65b4bddbbbe2ce2ba603b17fe11b3d633d324e34.js.gz
-rw-r--r-- 1 discourse discourse 5.7M Apr 26 19:14

Just to add another data point - still experiencing the rebuild failures after removing the two theme components. So just using the default Light theme.

also where is this output from - looking to verify/debug a bit on my end as well. is this like a verbose option on the ./launcher rebuild app?

Fixing this properly is going to take a little while, so I’m going to revert that change for now.


It may be a good idea to put it behind a feature flag afterwards so we can iterate on it internally.


Ok it’s been reverted and the commit is in tests-passed now:

Thanks for letting us know and sorry for the inconvenience.


I’d be interested to hear if this revert has also solved the OP’s problem. The OP (@devnull ) reported seeing

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

which wasn’t reported by @pfaffman

Also I’d like to comment on this:

Adding RAM can sometimes improve performance, but for actually running out of memory, what counts is RAM+swap. If adding swap doesn’t help with an out of memory condition, then adding RAM won’t either.

As @weallwegot reports the failure persists even after adding 4G of swap, something is (or was) very hungry indeed.


It’s the same error. Reading the backtrace shows a reference to OOM on the fourth line.


All of that is true. He has < 1GB of ram, which isn’t much. So even if he didn’t have this problem, I’d still recommend that he add more RAM. For that matter, I mostly recommend that anyone with <2GB of ram increase their RAM.

1 Like

The problem is resolved for me. Rebuild App works. Thank you very much for your effort!


(Absolutely, I understand the advice, it’s the logic which seemed like it might mislead.)


Was able to rebuild with both of those plugins with no issue by the way. Although now i have 6G of swap so not quite apples to apples lol.


Thanks! I’m back in business.


FYI I’ve just merged that change again but this time it should not break bootstrap. Please let me know if this causes any problems.


I just now had the same type of failure in bootstrap that @pfaffman described… parse error at theme_test_helper. Let me know if you need more info.

1 Like

Can you copy and paste the error here please? Also the output lines around compressing theme_test_helper and theme_test_vendor, they should look like this:

12825804.271282336 Compressing: discourse/tests/theme_test_helper-790dafad7d3cb8a853fd3127fa3d99022120baf968cbb297066d166808ad3ae1.js
gzip -f -c -9 /var/www/discourse/public/assets/discourse/tests/theme_test_helper-790dafad7d3cb8a853fd3127fa3d99022120baf968cbb297066d166808ad3ae1.js > /var/www/discourse/public/assets/discourse/tests/theme_test_helper-790dafad7d3cb8a853fd3127fa3d99022120baf968cbb297066d166808ad3ae1.js.gz

brotli -f --quality=11 /var/www/discourse/public/assets/discourse/tests/theme_test_helper-790dafad7d3cb8a853fd3127fa3d99022120baf968cbb297066d166808ad3ae1.js --output=/var/www/discourse/public/assets/discourse/tests/
Done compressing discourse/tests/theme_test_helper-790dafad7d3cb8a853fd3127fa3d99022120baf968cbb297066d166808ad3ae1.js : 6.85 secs
6061451.556507351 Compressing: discourse/tests/theme_test_helper-790dafad7d3cb8a853fd3127fa3d99022120baf968cbb297066d166808ad3ae1.js
uglifyjs '/var/www/discourse/public/assets/discourse/tests/_theme_test_helper-790dafad7d3cb8a853fd3127fa3d99022120baf968cbb297066d166808ad3ae1.js' -m -c -o '/var/www/discourse/public/assets/discourse/tests/theme_test_helper-790dafad7d3cb8a853fd3127fa3d99022120baf968cbb297066d166808ad3ae1.js' --source-map "base='/var/www/discourse/public/assets/discourse/tests',root='/assets/discourse/tests',url='/assets/discourse/tests/'"
Parse error at _theme_test_helper-790dafad7d3cb8a853fd3127fa3d99022120baf968cbb297066d166808ad3ae1.js:27883,17
define("sinon", () => {
ERROR: Unexpected token: punc «)»
    at JS_Parse_Error.get (eval at <anonymous> (/usr/lib/node_modules/uglify-js/tools/node.js:18:1), <anonymous>:71:23)
    at fatal (/usr/lib/node_modules/uglify-js/bin/uglifyjs:409:27)
    at run (/usr/lib/node_modules/uglify-js/bin/uglifyjs:347:9)
    at Object.<anonymous> (/usr/lib/node_modules/uglify-js/bin/uglifyjs:261:5)
    at Module._compile (internal/modules/cjs/loader.js:778:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
    at Module.load (internal/modules/cjs/loader.js:653:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
    at Function.Module._load (internal/modules/cjs/loader.js:585:3)
    at Function.Module.runMain (internal/modules/cjs/loader.js:831:12)
rake aborted!
Errno::ENOENT: No such file or directory @ rb_file_s_size - /var/www/discourse/public/assets/discourse/tests/theme_test_helper-790dafad7d3cb8a853fd3127fa3d99022120baf968cbb297066d166808ad3ae1.js
/var/www/discourse/lib/tasks/assets.rake:290:in `size'
/var/www/discourse/lib/tasks/assets.rake:290:in `block (4 levels) in <main>'
/var/www/discourse/lib/tasks/assets.rake:181:in `block in concurrent?'
/var/www/discourse/lib/tasks/assets.rake:281:in `block (3 levels) in <main>'
/var/www/discourse/lib/tasks/assets.rake:272:in `each'
/var/www/discourse/lib/tasks/assets.rake:272:in `block (2 levels) in <main>'
/var/www/discourse/lib/tasks/assets.rake:181:in `concurrent?'
/var/www/discourse/lib/tasks/assets.rake:269:in `block in <main>'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/rake-13.0.3/exe/rake:27:in `<top (required)>'
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'
Tasks: TOP => assets:precompile
(See full trace by running task with --trace)

and then of course after running through the rest of the plugins, theme components, etc.:

Pups::ExecError: cd /var/www/discourse && su discourse -c 'bundle exec rake themes:update assets:precompile' failed with return #<Process::Status: pid 5128 exit 1>
Location of failure: /pups/lib/pups/exec_command.rb:112:in `spawn'
exec failed with the params {"cd"=>"$home", "hook"=>"assets_precompile", "cmd"=>["su discourse -c 'bundle exec rake themes:update assets:precompile'"]}
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.

Can you please enter the container and see what version uglifyjs is on?

./launcher enter app
uglifyjs --version

And also discourse_docker in your host machine:

git rev-parse HEAD

@Osama @downey,

we just bumped the base image a few hours ago and are now forcing a new base image so we can properly compile Javascript from 2015. Can you please do a git pull and then try a rebuild @downey ? It should use terser instead of uglify now.

cc @pmusaraj


Thanks all. I was away for a few evening hours with some family committments but was pleasantly surprised to see this update.

Good news: I did git pull and rebuilt, and everything seemed to work fine. (Although I did get a notice about less than 5GB available so needed to clean up some old Docker images and try it a second time.)

But all is well now, as far as I can tell. :tada:

Let me know if you need any further info from my side.