The discussion revolves around an Out of memory errors with custom plugin issue on a Discourse instance. The instance is running with 8GB of RAM and 2GB of swap, and the system is experiencing sporadic kernel “out of memory” messages, increased swapping, and degraded response times.
The user pelcami initially asked for instructions on how to configure memory allocation, as they are using all default memory configurations. Falco pointed out that the number of Ruby unicorns is set in the /var/discourse/containers/app.yml file and that 4 unicorns should use around 2GB of memory, leaving 6 GB for Postgres and Redis.
However, Serge investigated the issue further and found that Ruby is the process consuming all the memory, causing kernel out-of-memory events. Falco asked if any plugins are running on the instance, and Serge revealed that they are running a custom Scheduled Digest plugin, which works fine on other instances.
Falco requested the plugin’s source code repository, but Serge was unable to share it due to it being a paid development. Falco suggested setting up the Prometheus exporter plugin to follow metrics on the instance.
Benjamin_D noted that the issue might be related to the Ruby version (2.3.1-2~ubuntu16.04.14) and pointed out that the Discourse readme recommends Ruby 2.6+. Serge confirmed that the other instances are running the same Ruby version and plans to test an update. sam clarified that the Ruby version is not relevant as long as the official Docker image is being used.
Would someone provide instructions to configure memory allocation?
Given the sporadic kernel “out of memory”/dump messages, 3x increased swapping, and increasing application degraded response, it seems our app is sometimes running out of memory. Our system currently has 8GB mem and 2GB swap. Details below.
I have reviewed the instructions for adding more physical mem and swap at ("Cannot allocate memory" when upgrading) but have not been able to find details about how to allocate it.
When it comes to mem configs, we are using all defaults. The system recovers after a few minutes, but with increased usage - we think it’s time to get educated about how to improve performance. I am, however, unsure of where and how to configure this. Increase memory allocated for the docker instance, or for number of ruby unicorns (or both)?
I’m a sysadmin with no ruby and limited docker experience - so pointing me in direction of the config file, and syntax to use will help greatly.
Did you re-run discourse-setup after you upped your RAM? It will tweak the memory settings accordingly. You can also read the comments in app.yml and adjust them.
the plugin was paid for development by our organization, so unfortunately I am not allowed to expose the source code here. Also since the plugin works on other instances without an issue, this makes me thinking of increasing server memory resources for this particular one. The machine is a VM so I can easily double the memory amount and see if this will fix it.
Are the other instances also running Ruby 2.3.1-2~ubuntu16.04.14 ?
maybe nothing relevant but:
so this was clearly a Ruby bug. We tested across multiple versions of Ruby and determined that only rubies 2.3.x and 2.4.x were exhibiting the leak (apparently this was fixed in Ruby 2.5.0****).
