Memory is running out and Discourse stops working

I like many existing users I have been experiencing a combination of these issues:

  • Low memory errors
  • ImageMagicK message in admin panel.
  • White pages / instability

This post / topic is a dump of available information before I restart the instance - so it might help track down the problem.

###System
2GB Digital Ocean instance with 2GB Swap - single site running in Docker.

###History
Earlier this week I restarted my Discourse instance using ./launcher restart app to resolve my then availability issues.

Current version: Discourse 1.2.0.beta6 - https://github.com/discourse/discourse version 8689c85b54efccbccfa216c800522e5ba1a79dd1

###Possible related reports
These are only noted here for there similarity and may not be the the same issue!

https://meta.discourse.org/t/memory-usage-and-blank-pages/24511

###Digital Ocean graphs

###Pingdom uptime graphs

Still collecting information and will continue to post in this thread.

4 Likes

I believe ruby / some processed restarted not long after I logged into the shell and the site became “more available” displaying pages:

%Cpu(s): 33.4 us,  9.0 sy,  0.0 ni, 38.9 id, 18.6 wa,  0.2 hi,  0.0 si,  0.0 st
KiB Mem:   2049988 total,  1983064 used,    66924 free,      136 buffers
KiB Swap:  2097148 total,  2082940 used,    14208 free.   349796 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
20572 message+  20   0  413312 277384 257608 R  68.7 13.5   0:17.45 postmaster
17653 1000      20   0 1126156 178964   2996 S   9.3  8.7   0:15.72 ruby
   36 root      20   0       0      0      0 S   3.3  0.0  80:30.07 kswapd0
28638 1000      20   0 1685276 305940   3064 S   1.0 14.9  40:48.71 ruby
19441 root      20   0   24956    648    364 R   0.7  0.0   0:01.91 top
28547 landsca+  20   0  435592  55272    688 S   0.3  2.7  20:40.51 redis-server
28569 message+  20   0  386336 107708 107172 S   0.3  5.3   0:46.53 postmaster
28572 message+  20   0  103908    800    248 S   0.3  0.0   0:46.54 postmaster
    1 root      20   0   33492    272      0 S   0.0  0.0   1:54.08 init

Note that postmaster is using a lot of CPU - this normally only happens when starting for me.

My current top looks like this:

Tasks: 116 total,   2 running, 114 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.5 us,  0.3 sy,  0.0 ni, 99.2 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:   2049988 total,  1980548 used,    69440 free,     1208 bufferss-server
KiB Swap:  2097148 total,  2074824 used,    22324 free.   307812 cached Memter
   36 root      20   0       0      0      0 S   3.3  0.0  80:30.26 kswapd0
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
    7 root      20   0       0      0      0 S   0.3  0.0  23:09.09 rcu_sched
17653 1000      20   0 1132308 182752   3936 S   0.3  8.9   0:22.23 ruby
18099 root      20   0  105632    260    160 S   0.3  0.0   0:00.90 sshd
21762 root      20   0   24952   1592   1096 R   0.3  0.1   0:00.02 top
28547 landsca+  20   0  435592  56044    712 S   0.3  2.7  20:44.53 redis-server
28656 1000      20   0 1664796 288848   2432 S   0.3 14.1  40:53.90 ruby
    1 root      20   0   33492    264      0 S   0.0  0.0   1:54.10 init
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.70 kthreadd
    3 root      20   0       0      0      0 S   0.0  0.0   1:36.78 ksoftirqd/0
    5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0H
    8 root      20   0       0      0      0 R   0.0  0.0  11:06.96 rcuos/0

Current login system displays high swap / memory usage

  System information as of Mon Feb  9 15:04:46 EST 2015

  System load:  0.17               Processes:              119
  Usage of /:   63.7% of 39.25GB   Users logged in:        0
  Memory usage: 81%                IP address for eth0:    
  Swap usage:   95%                IP address for docker0:

8 hours ago the following noteworthy items appeared in the logs

##Log


###Error 1
####Info
Errno::ENOMEM (Cannot allocate memory - convert public/uploads/letter_avatars/2/I/142_140_188/240.png[0] -background transparent -gravity center -thumbnail 40x40^ -extent 40x40 -interpolate bicubic -unsharp 2x0.5+0.7+0 -quality 98 public/uploads/letter_avatars/2/I/142_140_188/40.png)

####Backtrace
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/core_ext/kernel/agnostics.rb:7:in ' /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/core_ext/kernel/agnostics.rb:7:in ’
/var/www/discourse/app/models/optimized_image.rb:108:in resize' /var/www/discourse/lib/letter_avatar.rb:44:in generate’
/var/www/discourse/app/controllers/user_avatars_controller.rb:29:in show_letter' /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/action_controller/metal/implicit_render.rb:4:in send_action’
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/abstract_controller/base.rb:189:in process_action' /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/action_controller/metal/rendering.rb:10:in process_action’
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/abstract_controller/callbacks.rb:20:in block in process_action' /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:113:in call’
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:113:in call' /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:149:in block in halting_and_conditional’
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:149:in call' /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:149:in block in halting_and_conditional’
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:166:in call' /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:166:in block in halting’
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:166:in call' /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:166:in block in halting’
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:166:in call' /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:166:in block in halting’
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:166:in call' /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:166:in block in halting’
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:166:in call' /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:166:in block in halting’
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:166:in call' /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:166:in block in halting’
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:166:in call' /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:166:in block in halting’
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:166:in call' /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:166:in block in halting’
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:229:in call' /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:229:in block in halting’
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:149:in call' /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:149:in block in halting_and_conditional’
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:86:in call' /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:86:in run_callbacks’
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/abstract_controller/callbacks.rb:19:in process_action' /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/action_controller/metal/rescue.rb:29:in process_action’
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/action_controller/metal/instrumentation.rb:31:in block in process_action' /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/notifications.rb:159:in block in instrument’
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/notifications/instrumenter.rb:20:in instrument' /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/notifications.rb:159:in instrument’
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/action_controller/metal/instrumentation.rb:30:in process_action' /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/action_controller/metal/params_wrapper.rb:250:in process_action’
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.1.8/lib/active_record/railties/controller_runtime.rb:18:in process_action' /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/abstract_controller/base.rb:136:in process’
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionview-4.1.8/lib/action_view/rendering.rb:30:in process' /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-mini-profiler-0.9.2/lib/mini_profiler/profiling_methods.rb:79:in block in profile_method’
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/action_controller/metal.rb:196:in dispatch' /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/action_controller/metal/rack_delegation.rb:13:in dispatch’
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/action_controller/metal.rb:232:in block in action' /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/action_dispatch/routing/route_set.rb:82:in call’
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/action_dispatch/routing/route_set.rb:82:in dispatch' /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/action_dispatch/routing/route_set.rb:50:in call’
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/action_dispatch/journey/router.rb:73:in block in call' /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/action_dispatch/journey/router.rb:59:in each’
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/action_dispatch/journey/router.rb:59:in call' /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/action_dispatch/routing/route_set.rb:678:in call’
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in call' /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.2.2/lib/omniauth/strategy.rb:186:in call!’
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.2.2/lib/omniauth/strategy.rb:164:in call' /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.2.2/lib/omniauth/strategy.rb:186:in call!’
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.2.2/lib/omniauth/strategy.rb:164:in call' /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.2.2/lib/omniauth/strategy.rb:186:in call!’
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.2.2/lib/omniauth/strategy.rb:164:in call' /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.2.2/lib/omniauth/strategy.rb:186:in call!’
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.2.2/lib/omniauth/strategy.rb:164:in call' /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.2.2/lib/omniauth/strategy.rb:186:in call!’
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.2.2/lib/omniauth/strategy.rb:164:in call' /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.2.2/lib/omniauth/strategy.rb:186:in call!’
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.2.2/lib/omniauth/strategy.rb:164:in call' /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.2.2/lib/omniauth/builder.rb:59:in call’
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.5.2/lib/rack/conditionalget.rb:25:in call' /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.5.2/lib/rack/head.rb:11:in call’
/var/www/discourse/lib/middleware/anonymous_cache.rb:119:in call' /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/action_dispatch/middleware/params_parser.rb:27:in call’
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/action_dispatch/middleware/flash.rb:254:in call' /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/message_bus-1.0.6/lib/message_bus/rack/middleware.rb:55:in call’
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.5.2/lib/rack/session/abstract/id.rb:225:in context' /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.5.2/lib/rack/session/abstract/id.rb:220:in call’
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/action_dispatch/middleware/cookies.rb:560:in call' /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.1.8/lib/active_record/query_cache.rb:36:in call’
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.1.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:621:in call' /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/action_dispatch/middleware/callbacks.rb:29:in block in call’
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:82:in run_callbacks' /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/action_dispatch/middleware/callbacks.rb:27:in call’
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/action_dispatch/middleware/remote_ip.rb:76:in call' /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/action_dispatch/middleware/debug_exceptions.rb:17:in call’
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/action_dispatch/middleware/show_exceptions.rb:30:in call' /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/logster-0.1.6/lib/logster/middleware/reporter.rb:23:in call’
/var/www/discourse/config/initializers/silence_logger.rb:24:in call' /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/action_dispatch/middleware/request_id.rb:21:in call’
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.5.2/lib/rack/methodoverride.rb:21:in call' /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.5.2/lib/rack/runtime.rb:17:in call’
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.5.2/lib/rack/sendfile.rb:112:in call' /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-mini-profiler-0.9.2/lib/mini_profiler/profiler.rb:193:in call’
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.1.8/lib/rails/engine.rb:514:in call' /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.1.8/lib/rails/application.rb:144:in call’
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.1.8/lib/rails/railtie.rb:194:in public_send' /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.1.8/lib/rails/railtie.rb:194:in method_missing’
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.5.2/lib/rack/builder.rb:138:in call' /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.5.2/lib/rack/urlmap.rb:65:in block in call’
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.5.2/lib/rack/urlmap.rb:50:in each' /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.5.2/lib/rack/urlmap.rb:50:in call’
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-4.8.3/lib/unicorn/http_server.rb:576:in process_client' /var/www/discourse/lib/middleware/unicorn_oobgc.rb:95:in process_client’
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-4.8.3/lib/unicorn/http_server.rb:670:in worker_loop' /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-4.8.3/lib/unicorn/http_server.rb:525:in spawn_missing_workers’
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-4.8.3/lib/unicorn/http_server.rb:140:in start' /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-4.8.3/bin/unicorn:126:in <top (required)>’
/var/www/discourse/vendor/bundle/ruby/2.0.0/bin/unicorn:23:in load' /var/www/discourse/vendor/bundle/ruby/2.0.0/bin/unicorn:23:in ’

####Env

HTTP_HOST: example.com
REQUEST_URI: /letter_avatar/isaac/40/2.png
REQUEST_METHOD: GET
HTTP_USER_AGENT: Mozilla/5.0 (iPad; CPU OS 8_1_2 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B440 Safari/600.1.4
HTTP_ACCEPT: */*
HTTP_REFERER: http://example.com/t/topic/36090/5

###Error 2
####Info

Errno::ENOMEM (Cannot allocate memory - convert)

####Backtrace

/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/core_ext/kernel/agnostics.rb:7:in ``'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/core_ext/kernel/agnostics.rb:7:in ``'
/var/www/discourse/app/models/upload.rb:134:in `fix_image_orientation'
/var/www/discourse/app/models/upload.rb:106:in `resize_image'
/var/www/discourse/app/models/upload.rb:77:in `create_for'
/var/www/discourse/app/controllers/uploads_controller.rb:8:in `create'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/action_controller/metal/implicit_render.rb:4:in `send_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/abstract_controller/base.rb:189:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/action_controller/metal/rendering.rb:10:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/abstract_controller/callbacks.rb:20:in `block in process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:113:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:113:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:149:in `block in halting_and_conditional'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:166:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:166:in `block in halting'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:149:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:149:in `block in halting_and_conditional'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:166:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:166:in `block in halting'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:166:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:166:in `block in halting'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:166:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:166:in `block in halting'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:166:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:166:in `block in halting'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:166:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:166:in `block in halting'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:166:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:166:in `block in halting'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:166:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:166:in `block in halting'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:166:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:166:in `block in halting'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:229:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:229:in `block in halting'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:166:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:166:in `block in halting'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:86:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:86:in `run_callbacks'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/abstract_controller/callbacks.rb:19:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/action_controller/metal/rescue.rb:29:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/action_controller/metal/instrumentation.rb:31:in `block in process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/notifications.rb:159:in `block in instrument'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/notifications.rb:159:in `instrument'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/action_controller/metal/instrumentation.rb:30:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/action_controller/metal/params_wrapper.rb:250:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.1.8/lib/active_record/railties/controller_runtime.rb:18:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/abstract_controller/base.rb:136:in `process'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionview-4.1.8/lib/action_view/rendering.rb:30:in `process'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-mini-profiler-0.9.2/lib/mini_profiler/profiling_methods.rb:79:in `block in profile_method'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/action_controller/metal.rb:196:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/action_controller/metal/rack_delegation.rb:13:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/action_controller/metal.rb:232:in `block in action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/action_dispatch/routing/route_set.rb:82:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/action_dispatch/routing/route_set.rb:82:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/action_dispatch/routing/route_set.rb:50:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/action_dispatch/journey/router.rb:73:in `block in call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/action_dispatch/journey/router.rb:59:in `each'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/action_dispatch/journey/router.rb:59:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/action_dispatch/routing/route_set.rb:678:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.2.2/lib/omniauth/strategy.rb:186:in `call!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.2.2/lib/omniauth/strategy.rb:164:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.2.2/lib/omniauth/strategy.rb:186:in `call!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.2.2/lib/omniauth/strategy.rb:164:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.2.2/lib/omniauth/strategy.rb:186:in `call!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.2.2/lib/omniauth/strategy.rb:164:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.2.2/lib/omniauth/strategy.rb:186:in `call!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.2.2/lib/omniauth/strategy.rb:164:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.2.2/lib/omniauth/strategy.rb:186:in `call!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.2.2/lib/omniauth/strategy.rb:164:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.2.2/lib/omniauth/strategy.rb:186:in `call!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.2.2/lib/omniauth/strategy.rb:164:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.2.2/lib/omniauth/builder.rb:59:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.5.2/lib/rack/conditionalget.rb:35:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.5.2/lib/rack/head.rb:11:in `call'
/var/www/discourse/lib/middleware/anonymous_cache.rb:119:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/action_dispatch/middleware/params_parser.rb:27:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/action_dispatch/middleware/flash.rb:254:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/message_bus-1.0.6/lib/message_bus/rack/middleware.rb:55:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.5.2/lib/rack/session/abstract/id.rb:225:in `context'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.5.2/lib/rack/session/abstract/id.rb:220:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/action_dispatch/middleware/cookies.rb:560:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.1.8/lib/active_record/query_cache.rb:36:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.1.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:621:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/action_dispatch/middleware/callbacks.rb:29:in `block in call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.8/lib/active_support/callbacks.rb:82:in `run_callbacks'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/action_dispatch/middleware/callbacks.rb:27:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/action_dispatch/middleware/remote_ip.rb:76:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/action_dispatch/middleware/debug_exceptions.rb:17:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/action_dispatch/middleware/show_exceptions.rb:30:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/logster-0.1.6/lib/logster/middleware/reporter.rb:23:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.1.8/lib/rails/rack/logger.rb:38:in `call_app'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.1.8/lib/rails/rack/logger.rb:22:in `call'
/var/www/discourse/config/initializers/quiet_logger.rb:10:in `call_with_quiet_assets'
/var/www/discourse/config/initializers/silence_logger.rb:26:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.8/lib/action_dispatch/middleware/request_id.rb:21:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.5.2/lib/rack/methodoverride.rb:21:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.5.2/lib/rack/runtime.rb:17:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.5.2/lib/rack/sendfile.rb:112:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-mini-profiler-0.9.2/lib/mini_profiler/profiler.rb:193:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.1.8/lib/rails/engine.rb:514:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.1.8/lib/rails/application.rb:144:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.1.8/lib/rails/railtie.rb:194:in `public_send'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.1.8/lib/rails/railtie.rb:194:in `method_missing'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.5.2/lib/rack/builder.rb:138:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.5.2/lib/rack/urlmap.rb:65:in `block in call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.5.2/lib/rack/urlmap.rb:50:in `each'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.5.2/lib/rack/urlmap.rb:50:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-4.8.3/lib/unicorn/http_server.rb:576:in `process_client'
/var/www/discourse/lib/middleware/unicorn_oobgc.rb:95:in `process_client'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-4.8.3/lib/unicorn/http_server.rb:670:in `worker_loop'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-4.8.3/lib/unicorn/http_server.rb:525:in `spawn_missing_workers'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-4.8.3/lib/unicorn/http_server.rb:140:in `start'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-4.8.3/bin/unicorn:126:in `<top (required)>'
/var/www/discourse/vendor/bundle/ruby/2.0.0/bin/unicorn:23:in `load'
/var/www/discourse/vendor/bundle/ruby/2.0.0/bin/unicorn:23:in `<main>'

####Env

REQUEST_URI: /uploads.json?authenticity_token=emx0%2B3NI6Vl2QP%2BpO2TcGnBZxxxxxxxxxxxxxx%3D
REQUEST_METHOD: POST
HTTP_USER_AGENT: Mozilla/5.0 (Windows NT 6.1; rv:35.0) Gecko/20100101 Firefox/35.0
HTTP_ACCEPT: application/json, text/javascript, */*; q=0.01
HTTP_REFERER: http://example.com/t/topic/66341
username: example

params:
  authenticity_token: emx0+3xxxxl2QP+pO2TcGnBZxxxxxxxxxxxxxx
  files: #<ActionDispatch::Http::UploadedFile:0x007fea29205010>

Okay, this is multiple ENOMEM reports in a day. Something with our memory usage has gone up.

Sidekiq looks normal

@riking Are there any specific commands you would like me to run / logs to check at this time before I restart?

Unstable again

From 36 minutes ago - ~20:26

###Error 3
####Info

Failed to log exception MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error. -1025848994072858232
Reason: Redis::CommandError EXECABORT Transaction discarded because of previous errors.
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-3.2.0/lib/redis/pipeline.rb:79:in `finish'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-3.2.0/lib/redis/client.rb:146:in `block in call_pipeline'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-3.2.0/lib/redis/client.rb:273:in `with_reconnect'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-3.2.0/lib/redis/client.rb:144:in `call_pipeline'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-3.2.0/lib/redis.rb:2147:in `block in multi'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-3.2.0/lib/redis.rb:37:in `block in synchronize'
/usr/local/lib/ruby/2.0.0/monitor.rb:211:in `mon_synchronize'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-3.2.0/lib/redis.rb:37:in `synchronize'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-3.2.0/lib/redis.rb:2139:in `multi'
/var/www/discourse/lib/discourse_redis.rb:40:in `method_missing'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/logster-0.1.6/lib/logster/redis_store.rb:17:in `save'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/logster-0.1.6/lib/logster/base_store.rb:70:in `report'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/logster-0.1.6/lib/logster/logger.rb:47:in `add'
/usr/local/lib/ruby/2.0.0/logger.rb:454:in `error'
/var/www/discourse/config/initializers/sidekiq.rb:60:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sidekiq-3.3.1/lib/sidekiq/exception_handler.rb:20:in `block in handle_exception'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sidekiq-3.3.1/lib/sidekiq/exception_handler.rb:18:in `each'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sidekiq-3.3.1/lib/sidekiq/exception_handler.rb:18:in `handle_exception'
/var/www/discourse/lib/discourse.rb:23:in `handle_exception'
/var/www/discourse/config/initializers/sidekiq.rb:36:in `rescue in block (2 levels) in <top (required)>'
/var/www/discourse/config/initializers/sidekiq.rb:32:in `block (2 levels) in <top (required)>'

####Backtrace

/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/logster-0.1.6/lib/logster/logger.rb:47:in `add'
/usr/local/lib/ruby/2.0.0/logger.rb:463:in `fatal'
/var/www/discourse/config/initializers/sidekiq.rb:62:in `rescue in call'
/var/www/discourse/config/initializers/sidekiq.rb:65:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sidekiq-3.3.1/lib/sidekiq/exception_handler.rb:20:in `block in handle_exception'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sidekiq-3.3.1/lib/sidekiq/exception_handler.rb:18:in `each'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sidekiq-3.3.1/lib/sidekiq/exception_handler.rb:18:in `handle_exception'
/var/www/discourse/lib/discourse.rb:23:in `handle_exception'
/var/www/discourse/config/initializers/sidekiq.rb:36:in `rescue in block (2 levels) in <top (required)>'
/var/www/discourse/config/initializers/sidekiq.rb:32:in `block (2 levels) in <top (required)>'

####Env

current_db: default
message: While ticking scheduling manager

confirmed, will find the leak and plug it.

4 Likes

Yep, memory errors. @sam is investigating.

Perhaps we need a memory bench, to prevent this in the future?

1 Like

@sam Would you like any further information at this time? - It’s currently occurring for me.

This is a leak, we would need to run for multiple days to catch it.

1 Like

Top processes using memory.

root@forum:~# ps -eo pmem,pcpu,vsize,pid,cmd | sort -k 1 -nr | head -10
15.1  0.8 1685276 28638 unicorn worker[0] -E production -c config/unicorn.conf.rb
14.7  0.8 1670916 28648 unicorn worker[1] -E production -c config/unicorn.conf.rb
14.3  0.8 1664796 28656 unicorn worker[2] -E production -c config/unicorn.conf.rb
11.6  0.0 386440 28568 postgres: checkpointer process
10.5  0.3 420496 16637 postgres: discourse discourse [local] idle
 8.9  0.9 1148692 17653 sidekiq 3.3.1 discourse [0 of 5 busy]
 7.7  0.3 406816 20262 postgres: discourse discourse [local] idle
 7.6  0.3 406768 20340 postgres: discourse discourse [local] idle
 7.2  0.3 409860 20606 postgres: discourse discourse [local] idle
 5.3  0.2 418868 28573 unicorn master -E production -c config/unicorn.conf.rb

Seems this looks like it has been confirmed as a bug - I’ll change the category.

Feel free to change it back.

Possible earlier reports of issues:

2 Likes

During / after restarting memory usage from top processes.

root@forum:~# cd /var/discourse/
root@forum:/var/discourse#
root@forum:/var/discourse# ./launcher restart app
WARNING: No swap limit support
a33abd1122692e1672bd0892f13e28d792146dd2a519c8b7a07cc0012195ab74
cid found, ensuring container is started
a33abd1122692e1672bd0892f13e28d792146dd2a519c8b7a07cc0012195ab74


root@forum:/var/discourse# ps -eo pmem,pcpu,vsize,pid,cmd | sort -k 1 -nr | head -10
17.8  104 394636 25707 /usr/bin/redis-server *:6379
 2.3 88.8 142816 25738 ruby /var/www/discourse/vendor/bundle/ruby/2.0.0/bin/unicorn -E production -c config/unicorn.conf.rb
 1.1  1.8 386204 25710 /usr/lib/postgresql/9.3/bin/postmaster -D /etc/postgresql/9.3/main
 0.6  0.0 692804   640 /usr/bin/docker -d
 0.3  0.2 188332 25650 docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 80 -container-ip 172.17.0.36 -container-port 80
 0.2  1.0  20036 25745 sleep 1
 0.2  0.7 121152 25706 nginx: master process /usr/sbin/nginx
 0.2  0.2 188332 25641 docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 2222 -container-ip 172.17.0.36 -container-port 22
 0.1 18.5  29748 25705 /bin/bash config/unicorn_launcher -E production -c config/unicorn.conf.rb
 0.1  0.4  61364 25712 /usr/sbin/sshd -D -e


root@forum:/var/discourse# ps -eo pmem,pcpu,vsize,pid,cmd | sort -k 1 -nr | head -10
19.1 19.1 423304 25707 /usr/bin/redis-server *:6379
11.9 17.9 497800 25796 sidekiq 3.3.1 discourse [0 of 5 busy]
10.7 21.2 457864 25806 unicorn worker[0] -E production -c config/unicorn.conf.rb
10.3 62.8 439344 25738 unicorn master -E production -c config/unicorn.conf.rb
10.3 12.3 457864 25832 unicorn worker[1] -E production -c config/unicorn.conf.rb
10.2 12.3 455728 25842 unicorn worker[2] -E production -c config/unicorn.conf.rb
 9.4 57.0 398572 25810 postgres: discourse discourse [local] SELECT
 7.0 16.7 405736 25848 postgres: discourse discourse [local] idle
 6.0  8.4 396424 25841 postgres: discourse discourse [local] idle
 4.2 16.0 397380 25818 postgres: discourse discourse [local] idle


root@forum:/var/discourse# ps -eo pmem,pcpu,vsize,pid,cmd | sort -k 1 -nr | head -10
19.1 11.9 423304 25707 /usr/bin/redis-server *:6379
13.4 61.6 400348 25810 postgres: discourse discourse [local] SELECT
12.0  7.4 501896 25796 sidekiq 3.3.1 discourse [0 of 5 busy]
10.7 12.3 457864 25806 unicorn worker[0] -E production -c config/unicorn.conf.rb
10.3  8.2 457892 25842 unicorn worker[2] -E production -c config/unicorn.conf.rb
10.2  6.6 457864 25832 unicorn worker[1] -E production -c config/unicorn.conf.rb
10.2 38.0 439344 25738 unicorn master -E production -c config/unicorn.conf.rb
 8.0  6.7 407784 25848 postgres: discourse discourse [local] idle
 6.5  2.9 401448 25841 postgres: discourse discourse [local] idle
 4.8  2.1 397328 25829 postgres: discourse discourse [local] idle
root@forum:/var/discourse#

Hm, it seems redis is taking up a higher percentage of real memory?

Perhaps Redis is getting extra keys gunking it up? That don’t expire properly.

What’s the general theme of the output of this command?

./launcher enter app
redis-cli INFO
redis-cli KEYS '*' | awk '{print $1}' | sort
redis-cli KEYS '*' | awk '{print $1}' | wc -l

Obviously I have restarted the instance now so not sure how useful that info will be.

No longer at console. Will run when can, atleast 4 hours time.