Failed to bootstrap due to out of memory killer

bug
#17

Oh. It is surprisingly small. It’s a site that I think you guys set up some years ago (when you’d still run a site not on your infrastructure).

root@community:/var/discourse# free -h
              total        used        free      shared  buff/cache   available
Mem:           1.9G        1.2G        101M        259M        655M        354M
Swap:          2.0G        1.2G        773M

Ah. OK. It’ s 2GB DO droplet, and I have access to their control panel. I’ll tell them that we need to upgrade to 4GB and move them to an AMD.

EDIT: But if this is just to compress that one file, shouldn’t a 2GB droplet be enough?

#18

This test-helper file is hard on the compression.

  • UglifyJS uses 1.5GB RAM to compress it.

  • Terser uses a little over 1GB. Takes 40s. For comparison the same server takes 8s on Ember+jQuery :scream:

@eviltrout should we even have this file in production?

Ohh looks like it comes from this change from @Osama

https://github.com/discourse/discourse/pull/12815

-rw-r--r-- 1 discourse discourse  14M Apr 26 19:13 _test_helper-f4c4b5bf0657eab910d85b9a65b4bddbbbe2ce2ba603b17fe11b3d633d324e34.js
-rw-r--r-- 1 discourse discourse 6.6M Apr 26 19:14 test_helper-f4c4b5bf0657eab910d85b9a65b4bddbbbe2ce2ba603b17fe11b3d633d324e34.js
-rw-r--r-- 1 discourse discourse 1.1M Apr 26 19:14 test_helper-f4c4b5bf0657eab910d85b9a65b4bddbbbe2ce2ba603b17fe11b3d633d324e34.js.br
-rw-r--r-- 1 discourse discourse 1.5M Apr 26 19:14 test_helper-f4c4b5bf0657eab910d85b9a65b4bddbbbe2ce2ba603b17fe11b3d633d324e34.js.gz
-rw-r--r-- 1 discourse discourse 5.7M Apr 26 19:14 test_helper-f4c4b5bf0657eab910d85b9a65b4bddbbbe2ce2ba603b17fe11b3d633d324e34.js.map
8 Likes
#20

Just to add another data point - still experiencing the rebuild failures after removing the two theme components. So just using the default Light theme.

also where is this output from - looking to verify/debug a bit on my end as well. is this like a verbose option on the ./launcher rebuild app?

#21

Fixing this properly is going to take a little while, so I’m going to revert that change for now.

9 Likes
#22

It may be a good idea to put it behind a feature flag afterwards so we can iterate on it internally.

3 Likes
#23

Ok it’s been reverted and the commit is in tests-passed now:

https://github.com/discourse/discourse/commit/a169dc6832fb90bcec664b0e8bf52026ce0dae3f

Thanks for letting us know and sorry for the inconvenience.

7 Likes
#24

I’d be interested to hear if this revert has also solved the OP’s problem. The OP (@devnull ) reported seeing

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

which wasn’t reported by @pfaffman

Also I’d like to comment on this:

Adding RAM can sometimes improve performance, but for actually running out of memory, what counts is RAM+swap. If adding swap doesn’t help with an out of memory condition, then adding RAM won’t either.

As @weallwegot reports the failure persists even after adding 4G of swap, something is (or was) very hungry indeed.

2 Likes
#25

It’s the same error. Reading the backtrace shows a reference to OOM on the fourth line.

3 Likes
#26

All of that is true. He has < 1GB of ram, which isn’t much. So even if he didn’t have this problem, I’d still recommend that he add more RAM. For that matter, I mostly recommend that anyone with <2GB of ram increase their RAM.

1 Like
#27

The problem is resolved for me. Rebuild App works. Thank you very much for your effort!

3 Likes
#28

(Absolutely, I understand the advice, it’s the logic which seemed like it might mislead.)

2 Likes
#29

Was able to rebuild with both of those plugins with no issue by the way. Although now i have 6G of swap so not quite apples to apples lol.

4 Likes
#30

Thanks! I’m back in business.

6 Likes
#31

FYI I’ve just merged that change again but this time it should not break bootstrap. Please let me know if this causes any problems.

2 Likes
#32

I just now had the same type of failure in bootstrap that @pfaffman described… parse error at theme_test_helper. Let me know if you need more info.

1 Like
#33

Can you copy and paste the error here please? Also the output lines around compressing theme_test_helper and theme_test_vendor, they should look like this:

12825804.271282336 Compressing: discourse/tests/theme_test_helper-790dafad7d3cb8a853fd3127fa3d99022120baf968cbb297066d166808ad3ae1.js

gzip -f -c -9 /var/www/discourse/public/assets/discourse/tests/theme_test_helper-790dafad7d3cb8a853fd3127fa3d99022120baf968cbb297066d166808ad3ae1.js > /var/www/discourse/public/assets/discourse/tests/theme_test_helper-790dafad7d3cb8a853fd3127fa3d99022120baf968cbb297066d166808ad3ae1.js.gz

brotli -f --quality=11 /var/www/discourse/public/assets/discourse/tests/theme_test_helper-790dafad7d3cb8a853fd3127fa3d99022120baf968cbb297066d166808ad3ae1.js --output=/var/www/discourse/public/assets/discourse/tests/theme_test_helper-790dafad7d3cb8a853fd3127fa3d99022120baf968cbb297066d166808ad3ae1.js.br

Done compressing discourse/tests/theme_test_helper-790dafad7d3cb8a853fd3127fa3d99022120baf968cbb297066d166808ad3ae1.js : 6.85 secs
#34 
6061451.556507351 Compressing: discourse/tests/theme_test_helper-790dafad7d3cb8a853fd3127fa3d99022120baf968cbb297066d166808ad3ae1.js
uglifyjs '/var/www/discourse/public/assets/discourse/tests/_theme_test_helper-790dafad7d3cb8a853fd3127fa3d99022120baf968cbb297066d166808ad3ae1.js' -m -c -o '/var/www/discourse/public/assets/discourse/tests/theme_test_helper-790dafad7d3cb8a853fd3127fa3d99022120baf968cbb297066d166808ad3ae1.js' --source-map "base='/var/www/discourse/public/assets/discourse/tests',root='/assets/discourse/tests',url='/assets/discourse/tests/theme_test_helper-790dafad7d3cb8a853fd3127fa3d99022120baf968cbb297066d166808ad3ae1.js.map'"
Parse error at _theme_test_helper-790dafad7d3cb8a853fd3127fa3d99022120baf968cbb297066d166808ad3ae1.js:27883,17
define("sinon", () => {
                 ^
ERROR: Unexpected token: punc «)»
    at JS_Parse_Error.get (eval at <anonymous> (/usr/lib/node_modules/uglify-js/tools/node.js:18:1), <anonymous>:71:23)
    at fatal (/usr/lib/node_modules/uglify-js/bin/uglifyjs:409:27)
    at run (/usr/lib/node_modules/uglify-js/bin/uglifyjs:347:9)
    at Object.<anonymous> (/usr/lib/node_modules/uglify-js/bin/uglifyjs:261:5)
    at Module._compile (internal/modules/cjs/loader.js:778:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
    at Module.load (internal/modules/cjs/loader.js:653:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
    at Function.Module._load (internal/modules/cjs/loader.js:585:3)
    at Function.Module.runMain (internal/modules/cjs/loader.js:831:12)
rake aborted!
Errno::ENOENT: No such file or directory @ rb_file_s_size - /var/www/discourse/public/assets/discourse/tests/theme_test_helper-790dafad7d3cb8a853fd3127fa3d99022120baf968cbb297066d166808ad3ae1.js
/var/www/discourse/lib/tasks/assets.rake:290:in `size'
/var/www/discourse/lib/tasks/assets.rake:290:in `block (4 levels) in <main>'
/var/www/discourse/lib/tasks/assets.rake:181:in `block in concurrent?'
/var/www/discourse/lib/tasks/assets.rake:281:in `block (3 levels) in <main>'
/var/www/discourse/lib/tasks/assets.rake:272:in `each'
/var/www/discourse/lib/tasks/assets.rake:272:in `block (2 levels) in <main>'
/var/www/discourse/lib/tasks/assets.rake:181:in `concurrent?'
/var/www/discourse/lib/tasks/assets.rake:269:in `block in <main>'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/rake-13.0.3/exe/rake:27:in `<top (required)>'
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'
Tasks: TOP => assets:precompile
(See full trace by running task with --trace)

and then of course after running through the rest of the plugins, theme components, etc.:

FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && su discourse -c 'bundle exec rake themes:update assets:precompile' failed with return #<Process::Status: pid 5128 exit 1>
Location of failure: /pups/lib/pups/exec_command.rb:112:in `spawn'
exec failed with the params {"cd"=>"$home", "hook"=>"assets_precompile", "cmd"=>["su discourse -c 'bundle exec rake themes:update assets:precompile'"]}
29b7ee9f8297bdac0cc17dae6117ebe5aa94dcdf3f5dbfed4bf03b0bd7751cc0
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.
4 Likes
#35

Can you please enter the container and see what version uglifyjs is on?

./launcher enter app
uglifyjs --version

And also discourse_docker in your host machine:

git rev-parse HEAD
5 Likes
#36

@Osama @downey,

we just bumped the base image a few hours ago and are now forcing a new base image so we can properly compile Javascript from 2015. Can you please do a git pull and then try a rebuild @downey ? It should use terser instead of uglify now.

cc @pmusaraj

6 Likes
#37

Thanks all. I was away for a few evening hours with some family committments but was pleasantly surprised to see this update.

Good news: I did git pull and rebuilt, and everything seemed to work fine. (Although I did get a notice about less than 5GB available so needed to clean up some old Docker images and try it a second time.)

But all is well now, as far as I can tell. :tada:

Let me know if you need any further info from my side.

8 Likes