Problem with Upgrade [error 137]

For standard install site have this error log for most recent upgrade:


********************************************************
*** Please be patient, next steps might take a while ***
********************************************************
Cycling Unicorn, to free up memory
Restarting unicorn pid: 545
Waiting for Unicorn to reload.
Waiting for Unicorn to reload..
Waiting for Unicorn to reload...
Waiting for Unicorn to reload....
Waiting for Unicorn to reload.....
Waiting for Unicorn to reload......
Waiting for Unicorn to reload.......
Waiting for Unicorn to reload........
Waiting for Unicorn to reload.........
Waiting for Unicorn to reload..........
Waiting for Unicorn to reload...........
Waiting for Unicorn to reload............
Waiting for Unicorn to reload.............
Waiting for Unicorn to reload..............
Waiting for Unicorn to reload...............
Waiting for Unicorn to reload................
Waiting for Unicorn to reload.................
Waiting for Unicorn to reload..................
Waiting for Unicorn to reload...................
Waiting for Unicorn to reload....................
Waiting for Unicorn to reload.....................
Waiting for Unicorn to reload......................
Waiting for Unicorn to reload.......................
Using libv8-node 18.16.0.0 (x86_64-linux)
Using method_source 1.0.0
Using thor 1.3.0
Using zeitwerk 2.6.12
Using railties 7.0.7
Using request_store 1.5.1
Using lograge 0.14.0
Using logstash-event 1.2.02
Using logstash-logger 0.26.1
Using logster 2.13.1
Using lru_redux 1.1.0
Using lz4-ruby 0.3.3
Using maxminddb 0.1.22
Using memory_profiler 1.0.1
Using message_bus 4.3.8
Using mini_racer 0.8.0
Using redis 4.8.1
Using sidekiq 6.5.12
Using mini_scheduler 0.16.0
Using mini_sql 1.5.0
Using mini_suffix 0.3.3
Using multi_json 1.15.0
Using multi_xml 0.6.0
Using mustache 1.1.1
Using uri 0.13.0
Using net-http 0.4.0
Using nio4r 2.7.0
Using version_gem 1.1.3
Using oauth-tty 1.0.5
Using snaky_hash 2.0.1
Using oauth 1.1.0
Using oauth2 1.4.11
Using oj 3.16.3
Using omniauth 1.9.2
Using omniauth-oauth2 1.7.3
Using omniauth-facebook 9.0.0
Using omniauth-github 1.4.0
Using omniauth-google-oauth2 0.8.2
Using omniauth-oauth 1.2.0
Using omniauth-twitter 1.4.0
Using optimist 3.1.0
Using pg 1.5.4
Using pry 0.14.2
Using pry-byebug 3.10.1
Using pry-rails 0.3.9
Using puma 6.4.0
Using rack-mini-profiler 3.3.0
Using rack-protection 3.1.0
Using rails_failover 2.0.1
Using rails_multisite 5.0.0
Using raindrops 0.20.1
Using rbtrace 0.5.1
Using rchardet 1.8.0
Using redis-namespace 1.11.0
Using rexml 3.2.6
Using rinku 2.0.6
Using rotp 6.3.0
Using rqrcode_core 1.2.0
Using rqrcode 2.2.0
Using rss 0.3.0
Using rtlcss 0.2.1
Using ruby-readability 0.7.0
Using rubyzip 2.3.2
Using sanitize 6.1.0
Using sass-embedded 1.69.5 (x86_64-linux-gnu)
Using sassc-embedded 1.68.6
Using sprockets 3.7.2 from https://github.com/rails/sprockets (at 3.x@f4d3dae)
Using sprockets-rails 3.4.2
Using sshkey 3.0.0
Using stackprof 0.2.25
Using tzinfo-data 1.2023.4
Using uglifier 4.2.0
Using unicorn 6.1.0
Using web-push 3.0.1
Bundle complete! 138 Gemfile dependencies, 171 gems now installed.
Gems in the groups 'development' and 'test' were not installed.
Bundled gems are installed into `./vendor/bundle`
1 installed gem you directly depend on is looking for funding.
  Run `bundle fund` for details
$ yarn install
yarn install v1.22.19
[1/5] Validating package.json...
[2/5] Resolving packages...
success Already up-to-date.
$ yarn --cwd app/assets/javascripts $(node -e 'const argv = JSON.parse(process.env.npm_config_argv).original; const passthrough = [`--frozen-lockfile`, `-s`].filter(arg => argv.includes(arg)); console.log(passthrough.join(` `));')
yarn install v1.22.19
[1/4] Resolving packages...
warning Resolution field "unset-value@2.0.1" is incompatible with requested version "unset-value@^1.0.0"
success Already up-to-date.
$ ./run-patch-package
patch-package 8.0.0
Applying patches...
babel-plugin-debug-macros@0.3.4 ✔
content-tag@1.2.2 ✔
ember-cli@5.0.0 ✔
ember-this-fallback@0.4.0 (1 deprecation-name) ✔
ember-this-fallback@0.4.0 (2 themes) ✔
virtual-dom@2.1.1 ✔
Done in 4.79s.
Done in 7.25s.
$ LOAD_PLUGINS=0 bundle exec rake plugin:pull_compatible_all
docker_manager is already at latest compatible version
$ SKIP_POST_DEPLOYMENT_MIGRATIONS=1 bundle exec rake multisite:migrate
Multisite migrator is running using 1 threads

Migrating default
Seeding default
*** Bundling assets. This will take a while *** 
$ bundle exec rake themes:update assets:precompile
Checking 'Air Theme' for 'default'... updating from b9d44745 to 85dc24d6
Checking 'Modern Category + Group Boxes' for 'default'... up to date
Checking 'Discourse Clickable Topic' for 'default'... up to date
Checking 'discourse-search-banner' for 'default'... updating from 934e0d35 to 6ba0e9d0
Node.js heap_size_limit (488.25) is less than 2048MB. Setting --max-old-space-size=2048.
yarn run v1.22.19
$ /var/www/discourse/app/assets/javascripts/node_modules/.bin/ember build
Building
Environment: development
WARNING: ember-test-selectors: You are using an unsupported ember-cli-babel version. data-test properties are not automatically stripped from your JS code.
building...
...[ConfigLoader]
...[Babel: @embroider/macros > applyPatches]
...[Babel: discourse-widget-hbs > applyPatches]
...[Babel: ember-source > applyPatches]
...[ember.js]
...[Babel: discourse-common > applyPatches]
...[Babel: truth-helpers > applyPatches]
...[Babel: ember-tracked-storage-polyfill > applyPatches]
...[Babel: @ember/legacy-built-in-components > applyPatches]
...[Babel: @ember/render-modifiers > applyPatches]
...[Babel: @ember/test-helpers > applyPatches]
...[Babel: @ember/test-waiters > applyPatches]
...[Babel: @embroider/util > applyPatches]
...[Babel: @glimmer/component > applyPatches]
...[Babel: dialog-holder > applyPatches]
...[Babel: ember-this-fallback > applyPatches]
...[Babel: ember-buffered-proxy > applyPatches]
...[Babel: ember-cached-decorator-polyfill > applyPatches]
...[Babel: ember-exam > applyPatches]
...[Babel: ember-functions-as-helper-polyfill > applyPatches]
...[Babel: ember-load-initializers > applyPatches]
...[Babel: ember-on-resize-modifier > applyPatches]
...[Babel: ember-resize-observer-service > applyPatches]
...[Babel: ember-router-service-refresh-polyfill > applyPatches]
...[Babel: float-kit > applyPatches]
...[Babel: select-kit > applyPatches]
...[@embroider/compat/app]
...[@embroider/webpack]
...[@embroider/webpack]
...[@embroider/webpack]
...[@embroider/webpack]
...[@embroider/webpack]
...[@embroider/webpack]
...[@embroider/webpack]
...[@embroider/webpack]
...[@embroider/webpack]
...[@embroider/webpack]
...[@embroider/webpack]
...[@embroider/webpack]
...[@embroider/webpack]
...[@embroider/webpack]
...[@embroider/webpack]
...[@embroider/webpack]
...[@embroider/webpack]
Killed
error Command failed with exit code 137.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
Docker Manager: FAILED TO UPGRADE
#<RuntimeError: RuntimeError>
/var/www/discourse/plugins/docker_manager/lib/docker_manager/upgrader.rb:210:in `run'
/var/www/discourse/plugins/docker_manager/lib/docker_manager/upgrader.rb:111:in `upgrade'
/var/www/discourse/plugins/docker_manager/scripts/docker_manager_upgrade.rb:19:in `block in <main>'
/var/www/discourse/plugins/docker_manager/scripts/docker_manager_upgrade.rb:6:in `fork'
/var/www/discourse/plugins/docker_manager/scripts/docker_manager_upgrade.rb:6:in `<main>'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/railties-7.0.7/lib/rails/commands/runner/runner_command.rb:43:in `load'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/railties-7.0.7/lib/rails/commands/runner/runner_command.rb:43:in `perform'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/thor-1.3.0/lib/thor/command.rb:28:in `run'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/thor-1.3.0/lib/thor/invocation.rb:127:in `invoke_command'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/thor-1.3.0/lib/thor.rb:527:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/railties-7.0.7/lib/rails/command/base.rb:87:in `perform'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/railties-7.0.7/lib/rails/command.rb:48:in `invoke'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/railties-7.0.7/lib/rails/commands.rb:18:in `<main>'
<internal:/usr/local/lib/ruby/site_ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
<internal:/usr/local/lib/ruby/site_ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.17.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
bin/rails:18:in `<main>'
Spinning up 1 Unicorn worker(s) that were stopped initially

Seems to still be working to process upgrade not sure if this is any kind of a serious problem or what can be done for this.

Out of memory.

You’ll need to add swap or ram.

But also you can try again and it might work.

If you’re updating from the UX you should try a command line rebuild.

5 Likes

Ran into the same issue with 1 GB of RAM and 2 GBs of swap (upgrading from the web interface). Bumped swap to 3 GBs, so far it looks like it’s going to work this time (upgrading from CLI).

3 Likes

Thanks, yeah this is a 1GB memory server seems like that isn’t ideal for discourse anymore needs 2 to work properly. Update install may be working now after re-start will se if it completes.

No, failed again. This was from UX will try command line next.

How much RAM and swap do you have?

free -h

Online updates at least 4GB nowadays in my experience (might get away with 3 …)

Says 952 Mi Memory and 2.0 Gi swap. Don’t know what is swap?

Upgrades seemed to work from command line apt upgrade

upgrade complete

By that do you mean to be able to run updates through the UX you need 4 GB ideally for that to work?

Don’t know why that takes up more memory than command line.

Have two standard install sites on digital ocean total cost for both of these is $16.32 a month, this is a lot less that paying the $100 a month for one standard hosted site but so far haven’t had any sign ups for those so this is a waste unless people want to join there may shut those sites down.

This does not seem correct. What is it you’re trying to do?

In console logged in as root, I type the command “apt upgrade” and this initiates the upgrade install.

Thanks to @Arkshine for mentioning this here:

Don’t know what the “Apt” means, but that seems to be necessary for this to work.

Just to check, are you under the assumption that is upgrading your Discourse or your server?

Well had thought that did upgrade discourse, but perhaps that was the Ubuntu server instead that was being upgraded, not sure.

I think you may benefit from reading some more topics on the subject. If you have a search of the documentation, or just a search in general, I think you’ll find plenty of information that should help fill in the gaps in your knowledge. There is a substantial amount of advice that covers the basics that you can learn from. :+1:

2 Likes

Ok, much to learn for this.

2 Likes

Missed that you wrote rebuild can rebuild the app.

Here is the official guide:

Because running the website (albeit with fewer Unicorns) + performing a build at the same time is costly in memory?

I tried this out on my server (3 VCPU, 4GB):

Online:

2.21 before upgrade → 1.7 as it frees up memory → 3.5 peak during build (GBs)

Swap also went up ~200MB to peak ~800MB

Command line (not apples to apples as just after the above)

1.7 before upgrade → drops to ~250MB! → climbs and climbs to peak 3.25 during assets compilation, but generally much lower.

Swap usage was very low during almost the entire process.

Caveat: all figures observed in htop so may be inaccurate, probably best to graph it.

I was surprised how “peaky” offline rebuilds were. Assets precompile definitely uses a lot of memory - perhaps more the more VCPUs you have as it might do some of that in parallel?

I was also suprised how little difference the peak was in my case, though swap usage was significantly lower and generally, memory use was much lower for an offline rebuild, remaining in the 100’s of MB during 95% of the process, whereas an online upgrade was minimum 1.7GB.

2 Likes

That seems correct, no need for a question mark there!

So rebuilding the app then automatically updates to most recent version it seems?

Is there a supported or non-supported way to keep previous versions of discourse running, or do new installs for previous versions?

No, but you can slow things down by changing to stable.

Unfortunately you just missed the boat this time as you can’t go backwards.

Ok, may want to change to that stable setting for only stable tested updates not the developer/beta ones. Looking to figure out how to do that.

1 Like