Excessive consumption of memory due to precompiling assets

Support
1

Hi folks,

We have been running our own instances of Discourse in OpenShift for the last years, and in the last months (as of January 2026 approx and more or less aligned with the new approach explained in Introducing pre-compiled JS assets for self-hosters and Introducing a new build system for plugins ), we have observed the following scenario:

When precompiling assets in build time (bundle exec rake assets:precompile:build), this operation now explodes and consumes more than 20Gb:

...
gem install prometheus_exporter -v 2.2.0 -i /var/www/discourse/plugins/discourse-prometheus/gems/3.4.7 --no-document --ignore-dependencies --no-user-install
Successfully installed prometheus_exporter-2.2.0
1 gem installed
Plugin name is 'msgraph-polling', but plugin directory is named 'msgraph-poll-discourse-plugin'
[assemble_ember_build] No existing build info file found.
Fetching and extracting https://get.discourse.org/discourse-assets/2026.5.0-latest-03484cbd/production.tar.gz...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 20.1M  100 20.1M    0     0  22.5M      0 --:--:-- --:--:-- --:--:-- 22.5M
Prebuilt assets downloaded and extracted successfully.
[assemble_ember_build] Reusing existing core ember build. All done.
Plugin name is 'msgraph-polling', but plugin directory is named 'msgraph-poll-discourse-plugin'
[Plugin::JsManager] Compiling 49 plugins...

# stucks here for long

looking at the memory consumption, we see:

Every 1.0s: free -h                             webapp-test-discourse-689b5fcb4d-fd2dp-debug-b7nn2: Mon May  4 14:15:57 2026

               total        used        free      shared  buff/cache   available
Mem:            28Gi        26Gi       596Mi       524Mi       2.1Gi       1.7Gi
Swap:             0B          0B          0B

while before was just rather quick with moderated memory consumption.

We have tried setting environment variables likeCI=1 and NODE_OPTIONS=”-–max-old-space-size=X”, and nothing seems to help containing this memory consumption.

Is anybody facing the same issue and if so, how did you solve it?

Many thanks!

Ismael

2

Are you able to share a list of plugins which you have installed?

3

Is your total Memory 4g? If so do you have a swap setup?

Can you share server specs?

4

Hi David,

The list of extra plugins are:

#   - Trade buttons (used on Marketplace)
          - git clone --depth=1 https://github.com/jannolii/discourse-topic-trade-buttons.git
          #   - Saved Searches
          - git clone --depth=1 https://github.com/discourse/discourse-saved-searches.git
          #   - Discourse Askimet
          - git clone --depth=1 https://github.com/discourse/discourse-akismet.git
          #   - Prometheus
          - git clone --depth=1 https://github.com/discourse/discourse-prometheus.git
          #   - Discourse docs
          - git clone --depth=1 https://github.com/discourse/discourse-docs.git
          #   - MSGraph poll
          - git clone --depth=1 https://github.com/CERN/msgraph-poll-discourse-plugin.git

Cheers,

Ismael

5

Hi Heliosurge,

Nodes are 8 cpu / 30 GiB ram. In normal conditions, a forum used to consume 1 cpu and 2-3 Gb max of ram (including precompiling).

No swap configured. I understood that swap is used here when there are memory constraints, which should not be the case here. However, I’m more “worried” about the amount of memory consumed, when this was never the case.

Cheers,

Ismael

6

Well your server spec shouldn’t need swap. The team member David here likely able to best help

7

Does the memory usage subside as soon as the assets:build task is done?

8

Hi @david ,

Does the memory usage subside as soon as the assets:build task is done?

Nope. I digged further and there is something strange.

Before precompiling:build, this is the list of plugins:

/var/www/discourse$ ls plugins/
automation           discourse-akismet           discourse-data-explorer  discourse-hcaptcha           discourse-microsoft-auth  discourse-post-voting  discourse-saved-searches       discourse-user-notes           styleguide
chat                 discourse-apple-auth        discourse-details        discourse-lazy-videos        discourse-narrative-bot   discourse-presence     discourse-solved               discourse-zendesk-plugin
checklist            discourse-assign            discourse-docs           discourse-local-dates        discourse-oauth2-basic    discourse-prometheus   discourse-subscriptions        footnote
discourse-adplugin   discourse-cakeday           discourse-gamification   discourse-login-with-amazon  discourse-openid-connect  discourse-reactions    discourse-templates            msgraph-poll-discourse-plugin
discourse-affiliate  discourse-calendar          discourse-github         discourse-lti                discourse-patreon         discourse-rewind       discourse-topic-trade-buttons  poll
discourse-ai         discourse-chat-integration  discourse-graphviz       discourse-math               discourse-policy          discourse-rss-polling  discourse-topic-voting         spoiler-alert

I’ve noticed the following behaviour after debugging the code.

/var/www/discourse$ script/rails runner "AssetProcessor.ember_version"
Plugin name is 'msgraph-polling', but plugin directory is named 'msgraph-poll-discourse-plugin'

# hangs here forever

The AssetProcessor.ember_version corresponds to the line discourse/lib/plugin/js_manager.rb at latest · discourse/discourse · GitHub

So I did some modifications to this file (attached) , basically to print where it stucks while processing, and to remove the AssetProcessor.ember_version in discourse/lib/plugin/js_manager.rb at latest · discourse/discourse · GitHub , just putting 5 to continue the hex generation.

Then, I reduced parallelism to 1 to ease things in discourse/lib/plugin/js_manager.rb at latest · discourse/discourse · GitHub (parallel_count = [Etc.nprocessors, 1].min).

After this, I executed bundle exec rake assets:precompile:build, resulting in the following:

/var/www/discourse$ bundle exec rake assets:precompile:build
Plugin name is 'msgraph-polling', but plugin directory is named 'msgraph-poll-discourse-plugin'
[assemble_ember_build] Reusing existing core ember build. All done.
Plugin name is 'msgraph-polling', but plugin directory is named 'msgraph-poll-discourse-plugin'
[Plugin::JsManager] Compiling 49 plugins...
Compiling automation...
end of files.sort
end of files.sort
        hex_digest 103dc9ebebb80a7065cb8dd41fb3356b30f151f7
########### recursive

# hang here forever, consuming the full memory

I have my suspicious that it may be related to the ulimit value (that we have set to unlimited instead to something like ulimit -n 1048576;), given the fact that you are opening quite a lot number of files and storing its content in memory (via the recursive calls).

Let me know if this rings you a bell, or you have any other hint about what may be the issue.

Cheers,

Ismael

js_manager.rb.txt (7.7 KB)

9

It’s always advisable to have swap. It’s a very good idea to enable the kernel to overcommit. It can substantially reduce your peak memory needs.

I’d sort out those two things and then try again. For overcommit, see

In terms of diagnosis, it might be helpful to check dmesg for OOM events, which you could do afterwards, and also to run vmstat at the time of the stall.

vmstat 5 5

Here’s my general diagnostic advice:

10

Hi @Ed_S ,

Thank you for your message.

I did diagnose and this is what I’m getting:

vmstat 5 200
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 3  0      0 19595924    104 3919416    0    0  4173    32  439 1040  5  1 93  0  0
 1  0      0 19595924    104 3919416    0    0     0   154 4249 6449  1  1 98  0  0
 1  0      0 19595924    104 3919416    0    0     0    39 4399 6778  1  1 98  0  0
 1  0      0 19595924    104 3919416    0    0    12    75 5414 8640  1  1 98  0  0
 1  0      0 19595924    104 3919416    0    0    51    69 4248 6637  1  1 99  0  0
 1  0      0 19595924    104 3919416    0    0     0    83 4441 6784  1  1 98  0  0
 1  0      0 19595924    104 3919416    0    0     9    53 6111 9254  2  1 97  0  0
 1  0      0 19595924    104 3919416    0    0     0   887 4854 7373  1  1 98  0  0
 1  0      0 19595924    104 3919416    0    0     0    40 4705 7319  1  1 98  0  0
 1  0      0 19595924    104 3919416    0    0     0    37 4701 7305  1  1 98  0  0
# we start precompiling...
 3  0      0 19595924    104 3919416    0    0   124   902 8292 10254 19  5 75  0  0
 2  0      0 19595924    104 3919416    0    0 43073  6829 13702 16200 11  4 82  4  0
 2  0      0 19595924    104 3919416    0    0 19624   815 12340 15581 10  4 83  3  0
 2  0      0 19595924    104 3919416    0    0  1818  3953 7554 9248 13  3 84  0  0
 2  0      0 19595924    104 3919416    0    0     0    99 7475 8661 16  2 82  0  0
 2  0      0 19595924    104 3919416    0    0     0    52 7634 9084 13  2 84  0  0
 2  0      0 19595924    104 3919416    0    0   115   585 6843 8121 13  2 85  0  0
 2  0      0 19595924    104 3919416    0    0     0 13139 7254 8444 13  2 84  0  0
 2  0      0 19595924    104 3919416    0    0     3  1305 8740 11091 14  2 83  0  0
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 5  0      0 19595924    104 3919416    0    0   465  9798 8403 9279 13  2 85  0  0
 3  0      0 19595924    104 3919416    0    0     6    99 7264 8993 13  2 85  0  0
 2  0      0 19595924    104 3919416    0    0     0    96 7190 8627 13  2 85  0  0
 2  0      0 19595924    104 3919416    0    0     0    66 6869 8299 13  2 85  0  0
 2  0      0 19595924    104 3919416    0    0     0   109 7075 8521 13  2 85  0  0
 2  0      0 19595924    104 3919416    0    0     3    78 8763 11295 14  2 83  0  0
 2  0      0 19595924    104 3919416    0    0     0  3075 7337 8358 13  2 85  0  0
 4  0      0 19595924    104 3919416    0    0     6   133 7016 8697 13  2 85  0  0
 3  0      0 19595924    104 3919416    0    0     0    45 7005 8370 13  2 85  0  0
 2  0      0 19595924    104 3919416    0    0     0   134 7330 9011 13  2 85  0  0
 2  0      0 19595924    104 3919416    0    0    26    86 7239 8747 13  2 85  0  0
 2  0      0 19595924    104 3919416    0    0     0   127 8809 11618 15  3 83  0  0
 2  0      0 19595924    104 3919416    0    0     6  1473 7142 8352 13  2 85  0  0
 2  0      0 19595924    104 3919416    0    0  2021   136 8041 10138 13  3 84  0  0
 2  0      0 19595924    104 3919416    0    0  4457   664 6913 7927 12  3 84  0  0

Memory consumption skyrocketed:

               total        used        free      shared  buff/cache   available
Mem:            28Gi        26Gi       460Mi       518Mi       2.3Gi       1.8Gi
Swap:             0B          0B          0B

And stuck while compiling plugins:

/var/www/discourse$ bundle exec rake assets:precompile:build

gem install prometheus_exporter -v 2.2.0 -i /var/www/discourse/plugins/discourse-prometheus/gems/3.4.7 --no-document --ignore-dependencies --no-user-install
Successfully installed prometheus_exporter-2.2.0
1 gem installed

A new release of RubyGems is available: 3.6.9 → 4.0.11!
Run `gem update --system 4.0.11` to update your installation.

Plugin name is 'msgraph-polling', but plugin directory is named 'msgraph-poll-discourse-plugin'
[assemble_ember_build] No existing build info file found.
Fetching and extracting https://get.discourse.org/discourse-assets/2026.5.0-latest-6b98fe35/production.tar.gz...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--  0     0    0     0    0     0      0      0 --:--:-- --:--  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 20.1M  100 20.1M    0     0  20.8M      0 --:--:-- --:--:-- --:--:-- 20.8M
Prebuilt assets downloaded and extracted successfully.
[assemble_ember_build] Reusing existing core ember build. All done.
Plugin name is 'msgraph-polling', but plugin directory is named 'msgraph-poll-discourse-plugin'
[Plugin::JsManager] Compiling 49 plugins...

# hangs here till giving an OOMKilled

Are you able to modify ulimit and see if the above behaviour is produced?

Cheers,

Ismael

11

Ah, you saw an OOM, great. That’s definitive. ulimit has nothing to do with this.

Add swap. There’s no reason not to, other than lack of disk space. Add 8G or 16G, try again. You want to get to a working state. Then you can try to measure what process is blowing up, if you wish.

Set up overcommit. It’s good practice, it reduces peak memory problems. You don’t need to understand it or justify it, just do it. It’s part of a good Linux setup. Check first. It’s this easy:

# uname -a
Linux ubuntu-4gb-hel1-1 6.8.0-110-generic #110-Ubuntu SMP PREEMPT_DYNAMIC
 Thu Mar 19 17:16:23 UTC 2026 aarch64 aarch64 aarch64 GNU/Linux
# cat /proc/sys/vm/overcommit_memory
1
12

Hi guys,

Thank you a lot for all the tips you have provided us, very much appreciated. We believe we’ve identified the root cause of the recent memory issues.

Previously, running bundle exec rake assets:precompile:build at build-time (as root), did not require having redis nor connection to the database. This behavior has changed (ref: Introducing pre-compiled JS assets for self-hosters and Introducing a new build system for plugins ).

To accommodate this, we moved the bundle exec rake assets:precompile:build step to an init container at runtime (prior to execute db:migrate, etc). This allows it to run as the discourse user with the necessary service access to both redis and the database.

However, during execution, the process hits a loop in lib/plugin/js_manager.rb. Looking at ps -fe, we see pnpm repeatedly attempting to add itself, which leads to memory saturation:

...
discour+     704     688  5 11:00 pts/0    00:00:00 node /usr/bin/pnpm -C=frontend/asset-processor node build.js
discour+     718     704  5 11:00 pts/0    00:00:00 node /usr/bin/pnpm add pnpm@10.28.0 --loglevel=error --allow-build=@pnpm
discour+     729     718  6 11:00 pts/0    00:00:00 node /usr/bin/pnpm add pnpm@10.28.0 --loglevel=error --allow-build=@pnpm
discour+     740     729  6 11:00 pts/0    00:00:00 node /usr/bin/pnpm add pnpm@10.28.0 --loglevel=error --allow-build=@pnpm
discour+     754     740  7 11:00 pts/0    00:00:00 node /usr/bin/pnpm add pnpm@10.28.0 --loglevel=error --allow-build=@pnpm
...
# and the list starts growing and goes on and on, provoking the memory saturation

In our tests, we found that running the init container as root instead, and running npm uninstall -g pnpm followed by npm install -g pnpm@10.28.0, resolves the loop and allows the plugin compilation to finish successfully:

...
[Plugin::JsManager] Compiling 49 plugins...
[Plugin::JsManager] Finished initial compilation of plugins in 5.82s

So before overengineering our infra and probably change our design, I think this question is more for @david : are there plans to restore the previous behavior for assets:precompile:build so that it can run without Redis or a DB connection (similar to what you are doing with the DISCOURSE_DOWNLOAD_PRE_BUILT_ASSETS: 0 flow)?

On a side note, and out of curiosity: why does running node process as a non-root user trigger this recursive pnpm installation loop, whereas running as root seems to avoid it?

Cheers,
Ismael