Upgrading cannot allocate memory

(Michael Downey) #1

Continuing the discussion from Memory is running out and Discourse stops working:

I just ran our mostly-daily update this morning, and received the following error above the black status window:

Sorry, there was an error upgrading Discourse. Please check the logs below.

Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

/usr/local/bin/ruby -r ./siteconf20150216-7453-5i1l39.rb extconf.rb
Cannot allocate memory - /usr/local/bin/ruby -r ./siteconf20150216-7453-5i1l39.rb extconf.rb  2>&1

Gem files will remain installed in /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/eventmachine-1.0.7 for inspection.
Results logged to /var/www/discourse/vendor/bundle/ruby/2.0.0/extensions/x86_64-linux/2.0.0-static/eventmachine-1.0.7/gem_make.out
An error occurred while installing eventmachine (1.0.7), and Bundler cannot
Make sure that `gem install eventmachine -v '1.0.7'` succeeds before bundling.

Should I try to restart Discourse to free up memory or is there a bigger issue here?

             total       used       free     shared    buffers     cached
Mem:          2.0G       1.8G       184M        54M        15M       168M 
-/+ buffers/cache:       1.6G       369M
Swap:         1.0G       925M        98M

502 Bad Gateway after failed upgrade
(Jens Maier) #2

This is most likely because your Discourse workers, which are still affected by the leak, are consuming so much memory that gcc runs out of memory while trying to compile the C extension for the EventMachine gem.

So yeah. First reclaim your leaked memory, then try the update again.

To reclaim your memory, you can either restart Discourse or manually kill -QUIT your unicorn worker processes. The workers will spin down gracefully and the master will very quickly replace each worker with a fresh copy of itself that hasn’t yet leaked as much RAM. This should give your system enough breating room to upgrade Discourse without interrupting service with a full restart.

(Michael Downey) #3

That worked; thanks. Was just a bit worried maybe something larger underlying the update process.

(Régis Hanol) #4