25GB of "cannot allocate memory" logs

support
1

I installed Discourse a few days ago following the installation guide on DigitalOcean with 1GB RAM with a Swap file.

Now I get lots of log messages like

> E, [2015-09-21T07:56:41.518010 #21097] ERROR -- : master loop error: Cannot allocate memory - ps -eo rss,args | grep sidekiq | grep -v grep | awk '{print $1}' (Errno::ENOMEM)
> E, [2015-09-21T07:56:41.518119 #21097] ERROR -- : /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.10/lib/active_support/core_ext/kernel/agnostics.rb:7:in ``'
> E, [2015-09-21T07:56:41.518162 #21097] ERROR -- : /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.10/lib/active_support/core_ext/kernel/agnostics.rb:7:in ``'
> E, [2015-09-21T07:56:41.518198 #21097] ERROR -- : config/unicorn.conf.rb:84:in `max_rss'
> E, [2015-09-21T07:56:41.518222 #21097] ERROR -- : config/unicorn.conf.rb:99:in `out_of_memory?'
> E, [2015-09-21T07:56:41.518242 #21097] ERROR -- : config/unicorn.conf.rb:122:in `check_sidekiq_heartbeat'
> E, [2015-09-21T07:56:41.518268 #21097] ERROR -- : config/unicorn.conf.rb:146:in `master_sleep'
> E, [2015-09-21T07:56:41.518292 #21097] ERROR -- : /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-4.8.3/lib/unicorn/http_server.rb:295:in `join'
> E, [2015-09-21T07:56:41.518313 #21097] ERROR -- : /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-4.8.3/bin/unicorn:126:in `<top (required)>'
> E, [2015-09-21T07:56:41.518333 #21097] ERROR -- : /var/www/discourse/vendor/bundle/ruby/2.0.0/bin/unicorn:23:in `load'
> E, [2015-09-21T07:56:41.518365 #21097] ERROR -- : /var/www/discourse/vendor/bundle/ruby/2.0.0/bin/unicorn:23:in `<main>'

Between 2015-09-21T07:35:07 and 2015-09-21T10:29:57 I got 17GB in unicorn.stderr.log-20150921 and another 6GB in unicorn.stderr.log, which caused my server to quit working.

This happens every now and then.

As Discourse seems to work fine while those logs are heavily written, the problem is not the low memory but the disk filled by the logs.

What cat I do to get Discourse working?

2

What does free on the box return?

3

Odd, we definitely do not see this happen on the many customer $99 installs, with 1GB RAM and swap out there.

Any plugins or other unusual configuration? Did you deviate from the install guide in any way?

4

Thanks for your replies.

root@www:/var/discourse/shared/standalone/log/rails# free -h
              total      used       free     shared    buffers     cached
Mem:          994M       929M        64M       7.7M       4.6M        36M
-/+ buffers/cache:       888M       105M
Swap:         255M       191M        64M

I do not have any plugins installed. It’s still the very default Discourse from your installation guide. The only thing that I can mention is that I at first tried the 512MB server from DigitalOcean and increased to 1GB when installation failed and then started once more.

5

Swap is tiny and below our recommendation.

2 Likes
6

Thank you for this hint. I created the swap file by following a howto of DigitalOcean. Now I increased the size of the swap file and will see if it works.