25GB of "cannot allocate memory" logs

Product Support

AI-generated summary

michaelgiesler reports encountering repeated “Cannot allocate memory” errors in Discourse logs, generating 25GB of logs and filling the disk despite the site appearing functional. The errors originate from memory-checking scripts in unicorn.conf.rb, likely triggered by low available RAM and insufficient swap space. sam asks for memory stats, and michaelgiesler shares free -h output showing 994MB total RAM with 888MB used and only 255MB swap, of which 191MB is used. codinghorror notes this issue is unusual for standard 1GB installs and asks about plugins or config deviations — michaelgiesler confirms no plugins and a default install, though he upgraded from a 512MB droplet. sam identifies the swap size as below recommendation. michaelgiesler acknowledges the hint and increases swap size to test if the issue resolves. The core problem appears to be insufficient swap space causing memory-related errors that flood logs, not a Discourse bug or misconfiguration.

1

I installed Discourse a few days ago following the installation guide on DigitalOcean with 1GB RAM with a Swap file.

Now I get lots of log messages like

> E, [2015-09-21T07:56:41.518010 #21097] ERROR -- : master loop error: Cannot allocate memory - ps -eo rss,args | grep sidekiq | grep -v grep | awk '{print $1}' (Errno::ENOMEM)
> E, [2015-09-21T07:56:41.518119 #21097] ERROR -- : /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.10/lib/active_support/core_ext/kernel/agnostics.rb:7:in ``'
> E, [2015-09-21T07:56:41.518162 #21097] ERROR -- : /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.10/lib/active_support/core_ext/kernel/agnostics.rb:7:in ``'
> E, [2015-09-21T07:56:41.518198 #21097] ERROR -- : config/unicorn.conf.rb:84:in `max_rss'
> E, [2015-09-21T07:56:41.518222 #21097] ERROR -- : config/unicorn.conf.rb:99:in `out_of_memory?'
> E, [2015-09-21T07:56:41.518242 #21097] ERROR -- : config/unicorn.conf.rb:122:in `check_sidekiq_heartbeat'
> E, [2015-09-21T07:56:41.518268 #21097] ERROR -- : config/unicorn.conf.rb:146:in `master_sleep'
> E, [2015-09-21T07:56:41.518292 #21097] ERROR -- : /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-4.8.3/lib/unicorn/http_server.rb:295:in `join'
> E, [2015-09-21T07:56:41.518313 #21097] ERROR -- : /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-4.8.3/bin/unicorn:126:in `<top (required)>'
> E, [2015-09-21T07:56:41.518333 #21097] ERROR -- : /var/www/discourse/vendor/bundle/ruby/2.0.0/bin/unicorn:23:in `load'
> E, [2015-09-21T07:56:41.518365 #21097] ERROR -- : /var/www/discourse/vendor/bundle/ruby/2.0.0/bin/unicorn:23:in `<main>'

Between 2015-09-21T07:35:07 and 2015-09-21T10:29:57 I got 17GB in unicorn.stderr.log-20150921 and another 6GB in unicorn.stderr.log, which caused my server to quit working.

This happens every now and then.

As Discourse seems to work fine while those logs are heavily written, the problem is not the low memory but the disk filled by the logs.

What cat I do to get Discourse working?

2

What does free on the box return?

3

Odd, we definitely do not see this happen on the many customer $99 installs, with 1GB RAM and swap out there.

Any plugins or other unusual configuration? Did you deviate from the install guide in any way?

4

Thanks for your replies.

root@www:/var/discourse/shared/standalone/log/rails# free -h
              total      used       free     shared    buffers     cached
Mem:          994M       929M        64M       7.7M       4.6M        36M
-/+ buffers/cache:       888M       105M
Swap:         255M       191M        64M

I do not have any plugins installed. It’s still the very default Discourse from your installation guide. The only thing that I can mention is that I at first tried the 512MB server from DigitalOcean and increased to 1GB when installation failed and then started once more.

5

Swap is tiny and below our recommendation.

6

Thank you for this hint. I created the swap file by following a howto of DigitalOcean. Now I increased the size of the swap file and will see if it works.