Failed to upgrade discourse instance to Feb 15 2022

balupton · February 15, 2022, 6:10am

Upgrading from Commits · discourse/discourse · GitHub to the latest has caused the site to go down, here is notable segments of the logs:

I, [2022-02-15T05:27:48.995821 #1]  INFO -- : Replacing (?-mix:^host.*all.*all.*127.*$) with host all all 0.0.0.0/0 md5 in /etc/postgresql/13/main/pg_hba.conf
I, [2022-02-15T05:27:48.997476 #1]  INFO -- : Replacing (?-mix:^host.*all.*all.*::1\/128.*$) with host all all ::/0 md5 in /etc/postgresql/13/main/pg_hba.conf
I, [2022-02-15T05:27:48.998553 #1]  INFO -- : > HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main
I, [2022-02-15T05:27:49.003918 #1]  INFO -- : > sleep 5
2022-02-15 05:27:49.151 UTC [41] LOG:  starting PostgreSQL 13.5 (Debian 13.5-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
2022-02-15 05:27:49.152 UTC [41] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2022-02-15 05:27:49.152 UTC [41] LOG:  listening on IPv6 address "::", port 5432
2022-02-15 05:27:49.162 UTC [41] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2022-02-15 05:27:49.184 UTC [44] LOG:  database system was shut down at 2022-02-15 00:12:32 UTC
2022-02-15 05:27:49.209 UTC [41] LOG:  database system is ready to accept connections
I, [2022-02-15T05:27:54.013563 #1]  INFO -- : 
I, [2022-02-15T05:27:54.015179 #1]  INFO -- : > su postgres -c 'createdb discourse' || true
2022-02-15 05:27:54.259 UTC [54] postgres@postgres ERROR:  database "discourse" already exists
2022-02-15 05:27:54.259 UTC [54] postgres@postgres STATEMENT:  CREATE DATABASE discourse;
createdb: error: database creation failed: ERROR:  database "discourse" already exists
I, [2022-02-15T05:27:54.270626 #1]  INFO -- : 
I, [2022-02-15T05:27:54.272627 #1]  INFO -- : > su postgres -c 'psql discourse -c "create user discourse;"' || true
2022-02-15 05:27:54.425 UTC [58] postgres@discourse ERROR:  role "discourse" already exists
2022-02-15 05:27:54.425 UTC [58] postgres@discourse STATEMENT:  create user discourse;
ERROR:  role "discourse" already exists
I, [2022-02-15T05:27:54.431376 #1]  INFO -- : 
I, [2022-02-15T05:27:54.431915 #1]  INFO -- : > su postgres -c 'psql discourse -c "grant all privileges on database discourse to discourse;"' || true
I, [2022-02-15T05:27:54.533010 #1]  INFO -- : GRANT

I, [2022-02-15T05:27:55.303411 #1]  INFO -- : > echo redis installed
I, [2022-02-15T05:27:55.310858 #1]  INFO -- : redis installed

I, [2022-02-15T05:27:55.312551 #1]  INFO -- : > cat /etc/redis/redis.conf | grep logfile
I, [2022-02-15T05:27:55.323276 #1]  INFO -- : logfile ""

I, [2022-02-15T05:27:55.324362 #1]  INFO -- : > exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf
I, [2022-02-15T05:27:55.329352 #1]  INFO -- : > sleep 10
102:C 15 Feb 2022 05:27:55.343 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
102:C 15 Feb 2022 05:27:55.343 # Redis version=6.2.6, bits=64, commit=00000000, modified=0, pid=102, just started
102:C 15 Feb 2022 05:27:55.343 # Configuration loaded
102:M 15 Feb 2022 05:27:55.346 * monotonic clock: POSIX clock_gettime
102:M 15 Feb 2022 05:27:55.349 * Running mode=standalone, port=6379.
102:M 15 Feb 2022 05:27:55.350 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
102:M 15 Feb 2022 05:27:55.350 # Server initialized
102:M 15 Feb 2022 05:27:55.350 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
102:M 15 Feb 2022 05:27:55.352 * Loading RDB produced by version 6.2.6
102:M 15 Feb 2022 05:27:55.352 * RDB age 18923 seconds
102:M 15 Feb 2022 05:27:55.352 * RDB memory usage when created 133.51 Mb
102:M 15 Feb 2022 05:27:55.912 # Done loading RDB, keys loaded: 7294, keys expired: 127.
102:M 15 Feb 2022 05:27:55.912 * DB loaded from disk: 0.560 seconds
102:M 15 Feb 2022 05:27:55.912 * Ready to accept connections
I, [2022-02-15T05:28:05.336548 #1]  INFO -- : 
I, [2022-02-15T05:28:05.337696 #1]  INFO -- : > thpoff echo "thpoff is installed!"
I, [2022-02-15T05:28:05.349534 #1]  INFO -- : thpoff is installed!

I, [2022-02-15T05:28:05.349959 #1]  INFO -- : > /usr/local/bin/ruby -e 'if ENV["DISCOURSE_SMTP_ADDRESS"] == "smtp.example.com"; puts "Aborting! Mail is not configured!"; exit 1; end'
I, [2022-02-15T05:28:05.496038 #1]  INFO -- : 
I, [2022-02-15T05:28:05.496604 #1]  INFO -- : > /usr/local/bin/ruby -e 'if ENV["DISCOURSE_HOSTNAME"] == "discourse.example.com"; puts "Aborting! Domain is not configured!"; exit 1; end'
I, [2022-02-15T05:28:05.617847 #1]  INFO -- : 
I, [2022-02-15T05:28:05.618268 #1]  INFO -- : > /usr/local/bin/ruby -e 'if (ENV["DISCOURSE_CDN_URL"] || "")[0..1] == "//"; puts "Aborting! CDN must have a protocol specified. Once fixed you should rebake your posts now to correct all posts."; exit 1; end'
I, [2022-02-15T05:28:05.754370 #1]  INFO -- : 
I, [2022-02-15T05:28:05.755864 #1]  INFO -- : > rm -f /etc/cron.d/anacron
I, [2022-02-15T05:28:05.763371 #1]  INFO -- : 
I, [2022-02-15T05:28:05.773372 #1]  INFO -- : File > /etc/cron.d/anacron  chmod:   chown: 
I, [2022-02-15T05:28:05.785161 #1]  INFO -- : File > /etc/runit/1.d/copy-env  chmod: +x  chown: 
I, [2022-02-15T05:28:05.801026 #1]  INFO -- : File > /etc/service/unicorn/run  chmod: +x  chown: 
I, [2022-02-15T05:28:05.816469 #1]  INFO -- : File > /etc/service/nginx/run  chmod: +x  chown: 
I, [2022-02-15T05:28:05.828667 #1]  INFO -- : File > /etc/runit/3.d/01-nginx  chmod: +x  chown: 
I, [2022-02-15T05:28:05.841176 #1]  INFO -- : File > /etc/runit/3.d/02-unicorn  chmod: +x  chown: 
I, [2022-02-15T05:28:05.841464 #1]  INFO -- : Replacing # postgres with sv start postgres || exit 1 in /etc/service/unicorn/run
I, [2022-02-15T05:28:05.842241 #1]  INFO -- : > exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf
I, [2022-02-15T05:28:05.847862 #1]  INFO -- : > cd /var/www/discourse && git reset --hard
130:C 15 Feb 2022 05:28:05.853 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
130:C 15 Feb 2022 05:28:05.854 # Redis version=6.2.6, bits=64, commit=00000000, modified=0, pid=130, just started
130:C 15 Feb 2022 05:28:05.855 # Configuration loaded
130:M 15 Feb 2022 05:28:05.857 * monotonic clock: POSIX clock_gettime
130:M 15 Feb 2022 05:28:05.858 # Warning: Could not create server TCP listening socket *:6379: bind: Address already in use
130:M 15 Feb 2022 05:28:05.859 # Failed listening on port 6379 (TCP), aborting.
Updating files: 100% (30947/30947), done.
I, [2022-02-15T05:28:13.624544 #1]  INFO -- : HEAD is now at e1367cde I18N: Improve copy on blocked words error messages (#15741)

I, [2022-02-15T05:29:36.205328 #1]  INFO -- : > cd /var/www/discourse && [ ! -d 'node_modules' ] || su discourse -c 'yarn install --production && yarn cache clean'
warning Resolution field "lodash@4.17.21" is incompatible with requested version "lodash@4.17.15"
warning " > @mixer/parallel-prettier@2.0.1" has unmet peer dependency "prettier@^2.0.0".
I, [2022-02-15T05:33:01.692668 #1]  INFO -- : yarn install v1.22.17
[1/5] Validating package.json...
[2/5] Resolving packages...
[3/5] Fetching packages...
[4/5] Linking dependencies...
[5/5] Building fresh packages...
$ yarn --cwd app/assets/javascripts/discourse
yarn install v1.22.17
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
success Saved lockfile.
Done in 133.99s.
Done in 195.02s.
yarn cache v1.22.17
success Cleared cache.
Done in 9.35s.

I, [2022-02-15T05:33:01.697905 #1]  INFO -- : > cd /var/www/discourse && su discourse -c 'LOAD_PLUGINS=0 bundle exec rake plugin:pull_compatible_all'
I, [2022-02-15T05:33:12.324517 #1]  INFO -- : discourse-github is already at latest compatible version
discourse-solved is already at latest compatible version
discourse-chat is already at latest compatible version

I, [2022-02-15T05:33:12.325695 #1]  INFO -- : > cd /var/www/discourse && su discourse -c 'bundle exec rake db:migrate'
102:M 15 Feb 2022 05:33:37.418 * 100 changes in 300 seconds. Saving...
102:M 15 Feb 2022 05:33:37.423 * Background saving started by pid 1143
1143:C 15 Feb 2022 05:33:38.349 * DB saved on disk
1143:C 15 Feb 2022 05:33:38.352 * RDB: 2 MB of memory used by copy-on-write
102:M 15 Feb 2022 05:33:38.429 * Background saving terminated with success
I, [2022-02-15T05:33:38.994811 #1]  INFO -- : gem install sawyer -v 0.8.2 -i /var/www/discourse/plugins/discourse-github/gems/2.7.5 --no-document --ignore-dependencies --no-user-install
Successfully installed sawyer-0.8.2
1 gem installed
gem install octokit -v 4.21.0 -i /var/www/discourse/plugins/discourse-github/gems/2.7.5 --no-document --ignore-dependencies --no-user-install
Successfully installed octokit-4.21.0
1 gem installed
== 20220214224506 ResetCustomEmojiPostBakesVersionSecureFix: migrating ========
== 20220214224506 ResetCustomEmojiPostBakesVersionSecureFix: migrated (0.0099s) 

== 20220214233625 AddBounceErrorCodeToEmailLog: migrating =====================
-- add_column(:email_logs, :bounce_error_code, :string, {:null=>true})
   -> 0.0103s
== 20220214233625 AddBounceErrorCodeToEmailLog: migrated (0.0121s) ============


I, [2022-02-15T05:33:38.996639 #1]  INFO -- : > cd /var/www/discourse && su discourse -c 'bundle exec rake themes:update assets:precompile'
- Building
- Building
- building... 
102:M 15 Feb 2022 05:38:39.069 * 100 changes in 300 seconds. Saving...
102:M 15 Feb 2022 05:38:39.080 * Background saving started by pid 1213
1213:C 15 Feb 2022 05:38:39.863 * DB saved on disk
1213:C 15 Feb 2022 05:38:39.867 * RDB: 2 MB of memory used by copy-on-write
102:M 15 Feb 2022 05:38:39.883 * Background saving terminated with success
[WARN] (broccoli-terser-sourcemap) Minifying "assets/discourse.js" took: 35001ms (more than 20,000ms)
Killed
error Command failed with exit code 137.
I, [2022-02-15T05:52:50.058283 #1]  INFO -- : yarn run v1.22.17
$ /var/www/discourse/app/assets/javascripts/node_modules/.bin/ember build -prod
WARNING: Node v16.13.2 is not tested against Ember CLI on your platform. We recommend that you use the most-recent "Active LTS" version of Node.js. See https://git.io/v7S5n for details.
DEPRECATION: The integration of jQuery into Ember has been deprecated and will be removed with Ember 4.0. You can either opt-out of using jQuery, or install the `@ember/jquery` addon to provide the jQuery integration. Please consult the deprecation guide for further details: https://emberjs.com/deprecations/v3.x#toc_jquery-apis
Environment: production
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

I, [2022-02-15T05:52:50.072549 #1]  INFO -- : Terminating async processes
I, [2022-02-15T05:52:50.072816 #1]  INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main pid: 41
I, [2022-02-15T05:52:50.073275 #1]  INFO -- : Sending TERM to exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf pid: 102
102:signal-handler (1644904370) Received SIGTERM scheduling shutdown...
2022-02-15 05:52:50.074 UTC [41] LOG:  received fast shutdown request
2022-02-15 05:52:50.090 UTC [41] LOG:  aborting any active transactions
2022-02-15 05:52:50.096 UTC [41] LOG:  background worker "logical replication launcher" (PID 50) exited with exit code 1
2022-02-15 05:52:50.102 UTC [45] LOG:  shutting down
102:M 15 Feb 2022 05:52:50.118 # User requested shutdown...
102:M 15 Feb 2022 05:52:50.120 * Saving the final RDB snapshot before exiting.
2022-02-15 05:52:50.248 UTC [41] LOG:  database system is shut down
102:M 15 Feb 2022 05:52:51.075 * DB saved on disk
102:M 15 Feb 2022 05:52:51.076 # Redis is now ready to exit, bye bye...


FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && su discourse -c 'bundle exec rake themes:update assets:precompile' failed with return #<Process::Status: pid 1144 exit 137>
Location of failure: /usr/local/lib/ruby/gems/2.7.0/gems/pups-1.1.1/lib/pups/exec_command.rb:117:in `spawn'
exec failed with the params {"cd"=>"$home", "hook"=>"assets_precompile", "cmd"=>["su discourse -c 'bundle exec rake themes:update assets:precompile'"]}
1716fd9590e21cdfdaccea0ac6faabd133a6423e1a24ef3b3b5be0e68ce422c9
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.

Running discourse-doctor now, and gathering more detailed logs.

This is the upgrade script I made and have been using successfully for a year or more:

github.com

balupton/dotfiles/blob/master/commands/discourse-upgrader

#!/usr/bin/env bash
# this runs on the disource machine, do not depend on anything related to dorothy
set -Eeuo pipefail

# Receive arguments from caller
if test -z "${ssh_file_internal-}" -o -z "${ssh_pass_internal-}"; then
	echo 'For automatic SSH password insertion, you must pass the SSH file and password.' >/dev/stderr
fi

# Check discourse is already setup
if ! test -d /var/discourse; then
	echo 'Run this command on the remote discourse server'
	exit 1
fi

# Upgrading the host, which may solve SSL issues
# https://stackoverflow.com/a/61853542/130638
# https://meta.discourse.org/t/regenerating-letsencrypt-keys-from-behind-nginx/95884/3?u=balupton
# https://meta.discourse.org/t/ssl-letsencrypt-renewal-not-working-due-to-an-extra-reverse-proxy-on-the-outside/164236/2?u=balupton
apt-add-repository -y -r ppa:certbot/certbot

This file has been truncated. show original

JammyDodger · February 15, 2022, 6:24am

There should be a more specific error above ‘FAILED’ if you scroll up.

It looks like it doesn’t like a theme/component though. You may want to ./launcher restart app to bring back the pre-rebuild version so you can use the regular UI to disable the one in question before attempting another upgrade. Did scrolling up give you a clue as to which one?

balupton · February 15, 2022, 6:47am

The only siginficant log from brew doctor was these:

I, [2022-02-15T06:14:34.823778 #1]  INFO -- : > cd /var/www/discourse && su discourse -c 'bundle exec rake themes:update assets:precompile'
- Building
- Building
- building... 
Killed
error Command failed with exit code 137.
I, [2022-02-15T06:25:24.689000 #1]  INFO -- : yarn run v1.22.17
$ /var/www/discourse/app/assets/javascripts/node_modules/.bin/ember build -prod
WARNING: Node v16.13.2 is not tested against Ember CLI on your platform. We recommend that you use the most-recent "Active LTS" version of Node.js. See https://git.io/v7S5n for details.
DEPRECATION: The integration of jQuery into Ember has been deprecated and will be removed with Ember 4.0. You can either opt-out of using jQuery, or install the `@ember/jquery` addon to provide the jQuery integration. Please consult the deprecation guide for further details: https://emberjs.com/deprecations/v3.x#toc_jquery-apis
Environment: production
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

I, [2022-02-15T06:25:24.701625 #1]  INFO -- : Terminating async processes
I, [2022-02-15T06:25:24.702927 #1]  INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main pid: 42
I, [2022-02-15T06:25:24.703765 #1]  INFO -- : Sending TERM to exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf pid: 103
103:signal-handler (1644906324) Received SIGTERM scheduling shutdown...
2022-02-15 06:25:24.710 UTC [42] LOG:  received fast shutdown request
2022-02-15 06:25:24.741 UTC [42] LOG:  aborting any active transactions
103:M 15 Feb 2022 06:25:24.777 # User requested shutdown...
103:M 15 Feb 2022 06:25:24.777 * Saving the final RDB snapshot before exiting.
2022-02-15 06:25:24.796 UTC [42] LOG:  background worker "logical replication launcher" (PID 51) exited with exit code 1
2022-02-15 06:25:24.797 UTC [46] LOG:  shutting down
2022-02-15 06:25:25.023 UTC [42] LOG:  database system is shut down
103:M 15 Feb 2022 06:25:26.383 * DB saved on disk
103:M 15 Feb 2022 06:25:26.385 # Redis is now ready to exit, bye bye...


FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && su discourse -c 'bundle exec rake themes:update assets:precompile' failed with return #<Process::Status: pid 1146 exit 137>
Location of failure: /usr/local/lib/ruby/gems/2.7.0/gems/pups-1.1.1/lib/pups/exec_command.rb:117:in `spawn'
exec failed with the params {"cd"=>"$home", "hook"=>"assets_precompile", "cmd"=>["su discourse -c 'bundle exec rake themes:update assets:precompile'"]}
a1a2ad7dab631a87852f269d0de01bee80aea384be8a94ba4eb555f1fa5d9094
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.
==================== END REBUILD LOG ====================
Failed to rebuild app.

Checking your domain name . . .

Connection to discuss.bevry.me succeeded.
You should probably remove any non-standard plugins and rebuild.
Attempting to restart existing container. . . 
x86_64 arch detected.

starting up existing container
+ /usr/bin/docker start app
Error response from daemon: driver failed programming external connectivity on endpoint app (924d10c5f09f04b43e96bd6d1a4a13945cd86ebbb12821a4943914538821b856): Error starting userland proxy: listen tcp 0.0.0.0:443: bind: address already in use
Error: failed to start containers: app
Failed to restart the container.

I’ve updated the OP to include more of the original logs.

Rerunning discourse-doctor again, as it failed because wget was missing. Will post the full log once that is done.

itsbhanusharma · February 15, 2022, 7:28am

I can confirm this, I’ve tried this on 3 Different instances and builds are getting failed after reaching the below state.

I, [2022-02-15T07:09:36.714521 #1]  INFO -- : > cd /var/www/discourse && su discourse -c 'bundle exec rake db:migrate'
I, [2022-02-15T07:09:46.598212 #1]  INFO -- :
I, [2022-02-15T07:09:46.599983 #1]  INFO -- : > cd /var/www/discourse && su discourse -c 'bundle exec rake themes:update assets:precompile'
- Building
- Building
- building...
Killed
error Command failed with exit code 137.
I, [2022-02-15T07:30:11.622116 #1]  INFO -- : Checking 'Easy Footer' for 'default'... up to date
Checking 'Discourse Loading Slider' for 'default'... up to date
Checking 'discourse-search-banner' for 'sandbox'... up to date
Checking 'discourse-category-sidebars' for 'sandbox'... up to date
Checking 'Hide Users Column' for 'sandbox'... up to date
Checking 'discourse-sidebar-nav' for 'sandbox'... up to date
Checking 'Dracula Theme' for 'sandbox'... up to date
Checking 'discourse-buttons' for 'tekduke'... yarn run v1.22.17
$ /var/www/discourse/app/assets/javascripts/node_modules/.bin/ember build -prod
WARNING: Node v16.13.2 is not tested against Ember CLI on your platform. We recommend that you use the most-recent "Active LTS" version of Node.js. See https://git.io/v7S5n for details.
DEPRECATION: The integration of jQuery into Ember has been deprecated and will be removed with Ember 4.0. You can either opt-out of using jQuery, or install the `@ember/jquery` addon to provide the jQuery integration. Please consult the deprecation guide for further details: https://emberjs.com/deprecations/v3.x#toc_jquery-apis
Environment: production
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.



FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && su discourse -c 'bundle exec rake themes:update assets:precompile' failed with return #<Process::Status: pid 1034 exit 137>
Location of failure: /usr/local/lib/ruby/gems/2.7.0/gems/pups-1.1.1/lib/pups/exec_command.rb:117:in `spawn'
exec failed with the params {"cd"=>"$home", "hook"=>"assets_precompile", "cmd"=>["su discourse -c 'bundle exec rake themes:update assets:precompile'"]}
49ed46d025f5fe0a3de022b073b507cc3d94f74f96a60fe2d55042d413def454
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.

it looks like ember CLI related issue.

itsbhanusharma · February 15, 2022, 7:31am

hey @balupton could you try changing EMBER_CLI_PROD_ASSETS: 0 and try rebuilding again? I have at least 1 successful rebuild after the above change.

balupton · February 15, 2022, 7:54am

These seem to be the only relevant logs that indicate anything is wrong:

I, [2022-02-15T07:15:48.531001 #1]  INFO -- : > cd /var/www/discourse && [ ! -d 'node_modules' ] || su discourse -c 'yarn install --production && yarn cache clean'
warning Resolution field "lodash@4.17.21" is incompatible with requested version "lodash@4.17.15"
warning " > @mixer/parallel-prettier@2.0.1" has unmet peer dependency "prettier@^2.0.0".

I, [2022-02-15T07:19:45.453691 #1]  INFO -- : > cd /var/www/discourse && su discourse -c 'bundle exec rake themes:update assets:precompile'
- Building
- Building
- building... 
Killed
error Command failed with exit code 137.
I, [2022-02-15T07:31:39.549100 #1]  INFO -- : yarn run v1.22.17
$ /var/www/discourse/app/assets/javascripts/node_modules/.bin/ember build -prod
WARNING: Node v16.13.2 is not tested against Ember CLI on your platform. We recommend that you use the most-recent "Active LTS" version of Node.js. See https://git.io/v7S5n for details.
DEPRECATION: The integration of jQuery into Ember has been deprecated and will be removed with Ember 4.0. You can either opt-out of using jQuery, or install the `@ember/jquery` addon to provide the jQuery integration. Please consult the deprecation guide for further details: https://emberjs.com/deprecations/v3.x#toc_jquery-apis
Environment: production
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && su discourse -c 'bundle exec rake themes:update assets:precompile' failed with return #<Process::Status: pid 1145 exit 137>
Location of failure: /usr/local/lib/ruby/gems/2.7.0/gems/pups-1.1.1/lib/pups/exec_command.rb:117:in `spawn'
exec failed with the params {"cd"=>"$home", "hook"=>"assets_precompile", "cmd"=>["su discourse -c 'bundle exec rake themes:update assets:precompile'"]}
a262c461f2ae24390cb1397df309c44738cdc2688483d5fb7d52a3fde3155409
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.

balupton · February 15, 2022, 7:54am

Will try that, how do I apply it?

Nevermind, found it:

Will give it a go now.

balupton · February 15, 2022, 8:02am

Looking at the changelog:

I wonder if it is this:

github.com/discourse/discourse

PERF: Update ember-auto-import and webpack

discourse:main ← discourse:webpack-5

opened 11:31AM - 12 Feb 22 UTC

davidtaylorhq

+1740 -123

This makes a small improvement to 'cold cache' ember-cli build times, and a larg…e improvement to 'warm cache' build times The ember-auto-import update means that vendor is now split into multiple files for efficiency. These are named `chunk.*`, and should be included immediately after the `vendor.js` file. This commit also updates the rails app to render script tags for these chunks. This change was previously merged, and caused memory-related errors on RAM-constrained machines. This was because Webpack 5 switches from multiple worker processes to a single multi-threaded process. This meant that it was hitting node's default heap size limit (~500mb on a 1GB RAM server). Discourse's standard install procedure recommends adding 2GB swap to 1GB-RAM machines, so we can afford to override's Node's default via the `--max-old-space-size` flag.

As one of the logs was:

103:M 15 Feb 2022 06:54:18.720 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.

RGJ · February 15, 2022, 8:10am

Out of memory.

Make sure you have enough (virtual) memory. 1GB of memory and 2GB of swap is probably too low nowadays for a rebuild.

balupton · February 15, 2022, 8:19am

Yep, it worked.

vim /var/discourse/containers/app.yml

env:
  # ...
  ## Workaround for: https://meta.discourse.org/t/failed-to-upgrade-discourse-instance-to-feb-15-2022/218204/7?u=balupton
  EMBER_CLI_PROD_ASSETS: 0

balupton · February 15, 2022, 8:24am

with discourse now successfully running, top is currently returning:

KiB Mem : **2041324** total, **287808** free, **1033300** used, **720216** buff/cache
KiB Swap: **0** total, **0** free, **0** used. **581308** avail Mem

It is the VC1S server from Scaleway:

Would doing the suggestion in:

103:M 15 Feb 2022 06:54:18.720 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.

Also solve it?

itsbhanusharma · February 15, 2022, 8:24am

The VPS has 4GB RAM and 2GB Swap.

Ed_S · February 15, 2022, 9:26am

You need to add some swap. Looks like you have plenty of disk, so you could add 2G or even 3G for safety.

The discourse standard setup should, I think, have already set up the swap for you. Perhaps see Create a swapfile for your Linux server

Ed_S · February 15, 2022, 9:28am

And yet you see an out of memory - that’s worrying. Please paste your output of the free command.

balupton:

Would doing the suggestion in:

103:M 15 Feb 2022 06:54:18.720 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.

Also solve it?

I notice that in both my installations (both have just 1G of RAM and 2G of swap) I have at some point followed that advice and set up vm.overcommit_memory=1 in /etc/sysctl.conf. (That’s not to say it would solve this current problem.)

david · February 15, 2022, 9:39am

We’ll be removing the ability to turn off Ember CLI assets very soon, so please make sure to figure out the problem with your setup soon. Otherwise, it will break again in a future update.

You’ll need a minimum of 1GB RAM + 2GB swap (per our standard installation script)

That is indeed concerning - please let us know what you see in the free output @itsbhanusharma

itsbhanusharma · February 15, 2022, 3:25pm

Hey @david
Here is free

root@discourse:~# free
              total        used        free      shared  buff/cache   available
Mem:        3932016     3747708       87928       12812       96380        9960
Swap:       2097148     1095852     1001296

david · February 15, 2022, 3:42pm

Do you have anything else running on the server? The ‘used’ numbers there seem quite high for a standard Discourse install

RGJ · February 15, 2022, 3:51pm

available 9960

That’s not much

How many unicorns are you running?

itsbhanusharma · February 15, 2022, 3:53pm

It is a multisite discourse installation. There are a few additional containers (e.g. mail receiver & an uptime monitor) but nothing out of ordinary.

UNICORN_WORKERS: 4

RGJ · February 15, 2022, 3:54pm

What does free look like after a reboot?

Topic		Replies	Views
Update Problem Installation	1	579	March 10, 2023
Trouble installing discourse Bug arm	3	659	February 6, 2023
Updating failed, unable to rebuild Installation	4	1069	July 9, 2021
Can't build container, no clear errors Installation	3	2035	April 1, 2022
Build failed for specific commit hash Installation	9	620	April 12, 2023

Failed to upgrade discourse instance to Feb 15 2022

Related topics