Sidekiq is consuming too much memory

Product Support Self-hosting

AI-generated summary

Users are reporting that Sidekiq is consuming too much memory, triggering restarts despite having sufficient RAM (e.g., 8GB with 60% usage). davidkingham initially saw this after updating and tried resetting settings via discourse-setup, which configured db_shared_buffers = 2048MB and UNICORN_WORKERS = 4. Don suggested increasing Sidekiq’s memory limit, which resolved the issue for davidkingham. merefield confirmed experiencing the same issue and considered whether this reflects insufficient memory headroom or a potential memory leak. davidkingham later reported stable memory usage around 80% (up from 60%) with no further errors. Jagster questioned if the architecture has changed, noting the warning may appear even on non-busy, non-multisite instances. Don linked the issue to enabling YJIT, which increases memory usage, citing a note that self-hosted instances should monitor memory if enabling it. merefield noted the Sidekiq dashboard shows Redis memory but not Sidekiq’s own usage. davidkingham shared a screenshot confirming memory usage stabilized after the fix, and other performance issues also resolved.

1

Since updating to latest yesterday I’ve been seeing this error in the logs constantly. My server has 8gb of ram and only 60% used. I tried running discourse-setup to reset my settings, which gave me this:

setting db_shared_buffers = 2048MB
setting UNICORN_WORKERS = 4

Logs:

Message (781 copies reported)

Sidekiq is consuming too much memory (using: 510.87M) for 'community.naturephotographers.network', restarting

Backtrace

config/unicorn.conf.rb:164:in `check_sidekiq_heartbeat'
config/unicorn.conf.rb:244:in `master_sleep'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:295:in `join'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/bin/unicorn:128:in `<top (required)>'
/var/www/discourse/vendor/bundle/ruby/3.2.0/bin/unicorn:25:in `load'
/var/www/discourse/vendor/bundle/ruby/3.2.0/bin/unicorn:25:in `<main>'

Any ideas what may be going on here?

2

Increase the Sidekiq memory fixed it for me.

3

If it’s any consolation I get this too, fairly regularly.

I’ll investigate @Don’s suggestion. I did wonder! Now I know!

4

That seems to have done the trick, thank you Don!

5

Is this a case of “not enough headroom” or do we think this is a memory leak of some kind?

If the latter then surely this error will pop up anyway and increasing the memory allocated will just reduce the error frequency …

6

FWIW I still haven’t seen the error and my memory usage went up about 400 MB after changing this setting.

7

I’m a wondering this:

I get same error ( or should it count more as informative?) and mine is not multisite nor busy.

Is achitecture changed over the times and we are near old topic do we need more menory than told?

8

This warning appeared in my /logs after I actived YJIT but there is a note it is increase memory usage. However I think it can also happen busy sites with lots of activity or busy shared servers with shared memory etc…

9

the Sidekiq dashboard appears to show memory use for Redis, shame it doesn’t do the same for Sidekiq itself?:

image
image1052×211 31 KB

10

Just a little update here, since making the change my memory usage jumped from around 60% to 80% (8 GB) and holding steady. I haven’t seen another error in the logs and some other issues have cleared up as well.

image
image1012×304 30 KB