Hi,
We get this message in the logs about twice a day since the update to Discourse 1.6.0:
Sidekiq is consuming too much memory (using: 532.09M) for
forum.detendusdupad.com, restarting
Is it a normal behaviour, a misconfiguration or is Sidekiq actually leaking?
The UNICORN_WORKERS value is set to 4 on a 16GB install. The 1.6.0 update was applied by doing a clean rebuild of our split containers setup.
Thank you.
Any ideas here @sam?
I see it a lot in our logs, will be investigating. Something about Ruby 2.3 means that in some cases our Sidekiq process gets more memory hungry.
Slightly off topics, when developing my own projects, Sidekiq with Sidetiq consumses memory too much (heap increase 100mb per hour). Same configuration, ruby 2.3.1, rails 4.x, latest sidekiq. Seems like this is the problem with Ruby 2.3.1.
Some people recommend me to just switch to jRuby because of JVM does GC better, but i don’t really know.
FYI we removed sidetiq quite a while back. I will look at sorting out the current issue, just need to gather lots of metrics for a while.
Any new insights on this problem? We see this warning every hour in our logs.
We are also experiencing the same problem, would it alleviate the problem if we bump up memory from 2GB to 4GB?
It never hurts to have more memory, I wouldn’t do it based on this message alone, though. You should correlate memory to traffic and usage.
I am also getting this on my machine with 64gb of ram. Running 8 sidekiqs, and 16gb allocated to shared db.
Machine also runs apache2 serving up a wordpress site.
I am also getting periodic slow downs on the forums, dropping them to a crawl (a minute or so to load a thread). I have a feeling it is the daily backup. But havn’t really pinned it down yet.
Do you mean 8 unicorns?
Yes, sorry.
db_shared_buffers: "16GB"
db_work_mem: "40MB"
UNICORN_WORKERS: 8
I have reduced the workers to 4 for the time being. Seams I mistook my hardware configuration, and thought the server had dual 4 Core Xeons, but it only has one. We will see how it goes.
Edit: Seems it is an apache2 ProxyPass issue. Skipping the proxy makes it nice and fast.
any new insights @sam?
I have added some GC tuning to help mitigate some of the memory issues, noticing it less in the logs these days.
https://github.com/discourse/discourse_docker/blob/master/templates/web.template.yml#L10-L12
Try:
cd /var/discourse
git pull
./launcher rebuild app
I’m running 1.8.0 Beta6 and seeing this in my logs. Already did a rebuild of the app since I changed the domain name of the forum on Saturday and everything appears (front end) to be running fine. I have ample memory 22GB available.
Sidekiq is consuming too much memory (using: 540.01M) for 'photographic.community', restarting
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/logster-1.2.7/lib/logster/logger.rb:93:in `add_with_opts'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/logster-1.2.7/lib/logster/logger.rb:50:in `add'
/usr/local/lib/ruby/2.3.0/logger.rb:498:in `warn'
config/unicorn.conf.rb:123:in `check_sidekiq_heartbeat'
config/unicorn.conf.rb:146:in `master_sleep'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/unicorn-5.2.0/lib/unicorn/http_server.rb:284:in `join'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/unicorn-5.2.0/bin/unicorn:126:in `<top (required)>'
/var/www/discourse/vendor/bundle/ruby/2.3.0/bin/unicorn:22:in `load'
/var/www/discourse/vendor/bundle/ruby/2.3.0/bin/unicorn:22:in `<main>'
My app.yml settings are:
db_shared_buffers: "5GB"
db_work_mem: "40MB"
UNICORN_WORKERS: 6
and also seeing this error in my logs
console' is undefined Url: https://photographic.community/assets/ember_jquery-60dedd1dca1d8b8ce48b6d0087db3b96f3149b9b5af2d0c7c6357967d29031fb.js Line: 20 Window Location: https://photographic.community
params
message 'console' is undefined Url: https://photographic.community/assets/ember_jquery-60dedd1dca1d8b8ce48b6d
url https://photographic.community/assets/ember_jquery-60dedd1dca1d8b8ce48b6d0087db3b96f3149b9b5af2d0c7c6
line 20
window_location https://photographic.community/
It is not unusual to see this on high volume sites from time to time.
That’s the problem @codinghorror it’s brand new and not really high volume yet. Any suggestions on changing any of my app.yml settings as you can see I have 22Gb available in total allocated to the box.
By chance did you import from another forum into Discourse? If so, can you make a backup, and the restore said backup to improve Postgres performance.
Oh, other random thoughts, is Discourse the only thing running on that server, or are there other apps on it too?
Nope it was a clean install. Only two small sites running on Nginx, neither of which are public only git wordpress sites I use for versioning control while developing. Other than that Discourse can have the entire server to itself.
I’ve also seen this error for a couple of days now on my small and low activity instance with 2GB RAM.
Do I understand this thread correctly in that this is currently just something that happens sometimes and we don’t quite know why?
Yep. Seeing the exact same thing too. I have three identical Discourse servers. They are dedicated (no other application is served off of them, and have never received any imports from another Forum). Right now, the logs are only writing about the issue on one of them (this one, however, does get the most hits, and posts though). The write to the log, however, followed a complete degredation of all services (app stopped responding, discourse site stopped drawing, and I could not ssh into the box). Had to cycle power to get back into it. 
I’ve applied the band-aid of increasing memory on this one server from 4Gb to 8GB, and have not seen recurrence in the last 24 hours.
ruby 2.3.1p112
sidekiq 5.0.3
discourse 1.9.0.beta7
ubuntu 16.04.3 LTS
