Upgrading from Commits · discourse/discourse · GitHub to the latest has caused the site to go down, here is notable segments of the logs: I, [2022-02-15T05:27:48.995821 #1] INFO -- : Replacing (?-mix:^host.*all.*all.*127.*$) with host all all 0.0.0.0/0 md5 in /etc/postgresql/13/main/pg_hba.conf I, [2022-02-15T05:27:48.997476 #1] INFO -- : Replacing (?-mix:^host.*all.*all.*::1\/128.*$) with host all all ::/0 md5 in /etc/postgresql/13/main/pg_hba.conf I, [2022-02-15T05:27:48.998553 #1] INFO -- : > HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main I, [2022-02-15T05:27:49.003918 #1] INFO -- : > sleep 5 2022-02-15 05:27:49.151 UTC [41] LOG: starting PostgreSQL 13.5 (Debian 13.5-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit 2022-02-15 05:27:49.152 UTC [41] LOG: listening on IPv4 address "0.0.0.0", port 5432 2022-02-15 05:27:49.152 UTC [41] LOG: listening on IPv6 address "::", port 5432 2022-02-15 05:27:49.162 UTC [41] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" 2022-02-15 05:27:49.184 UTC [44] LOG: database system was shut down at 2022-02-15 00:12:32 UTC 2022-02-15 05:27:49.209 UTC [41] LOG: database system is ready to accept connections I, [2022-02-15T05:27:54.013563 #1] INFO -- : I, [2022-02-15T05:27:54.015179 #1] INFO -- : > su postgres -c 'createdb discourse' || true 2022-02-15 05:27:54.259 UTC [54] postgres@postgres ERROR: database "discourse" already exists 2022-02-15 05:27:54.259 UTC [54] postgres@postgres STATEMENT: CREATE DATABASE discourse; createdb: error: database creation failed: ERROR: database "discourse" already exists I, [2022-02-15T05:27:54.270626 #1] INFO -- : I, [2022-02-15T05:27:54.272627 #1] INFO -- : > su postgres -c 'psql discourse -c "create user discourse;"' || true 2022-02-15 05:27:54.425 UTC [58] postgres@discourse ERROR: role "discourse" already exists 2022-02-15 05:27:54.425 UTC [58] postgres@discourse STATEMENT: create user discourse; ERROR: role "discourse" already exists I, [2022-02-15T05:27:54.431376 #1] INFO -- : I, [2022-02-15T05:27:54.431915 #1] INFO -- : > su postgres -c 'psql discourse -c "grant all privileges on database discourse to discourse;"' || true I, [2022-02-15T05:27:54.533010 #1] INFO -- : GRANT I, [2022-02-15T05:27:55.303411 #1] INFO -- : > echo redis installed I, [2022-02-15T05:27:55.310858 #1] INFO -- : redis installed I, [2022-02-15T05:27:55.312551 #1] INFO -- : > cat /etc/redis/redis.conf | grep logfile I, [2022-02-15T05:27:55.323276 #1] INFO -- : logfile "" I, [2022-02-15T05:27:55.324362 #1] INFO -- : > exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf I, [2022-02-15T05:27:55.329352 #1] INFO -- : > sleep 10 102:C 15 Feb 2022 05:27:55.343 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 102:C 15 Feb 2022 05:27:55.343 # Redis version=6.2.6, bits=64, commit=00000000, modified=0, pid=102, just started 102:C 15 Feb 2022 05:27:55.343 # Configuration loaded 102:M 15 Feb 2022 05:27:55.346 * monotonic clock: POSIX clock_gettime 102:M 15 Feb 2022 05:27:55.349 * Running mode=standalone, port=6379. 102:M 15 Feb 2022 05:27:55.350 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128. 102:M 15 Feb 2022 05:27:55.350 # Server initialized 102:M 15 Feb 2022 05:27:55.350 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. 102:M 15 Feb 2022 05:27:55.352 * Loading RDB produced by version 6.2.6 102:M 15 Feb 2022 05:27:55.352 * RDB age 18923 seconds 102:M 15 Feb 2022 05:27:55.352 * RDB memory usage when created 133.51 Mb 102:M 15 Feb 2022 05:27:55.912 # Done loading RDB, keys loaded: 7294, keys expired: 127. 102:M 15 Feb 2022 05:27:55.912 * DB loaded from disk: 0.560 seconds 102:M 15 Feb 2022 05:27:55.912 * Ready to accept connections I, [2022-02-15T05:28:05.336548 #1] INFO -- : I, [2022-02-15T05:28:05.337696 #1] INFO -- : > thpoff echo "thpoff is installed!" I, [2022-02-15T05:28:05.349534 #1] INFO -- : thpoff is installed! I, [2022-02-15T05:28:05.349959 #1] INFO -- : > /usr/local/bin/ruby -e 'if ENV["DISCOURSE_SMTP_ADDRESS"] == "smtp.example.com"; puts "Aborting! Mail is not configured!"; exit 1; end' I, [2022-02-15T05:28:05.496038 #1] INFO -- : I, [2022-02-15T05:28:05.496604 #1] INFO -- : > /usr/local/bin/ruby -e 'if ENV["DISCOURSE_HOSTNAME"] == "discourse.example.com"; puts "Aborting! Domain is not configured!"; exit 1; end' I, [2022-02-15T05:28:05.617847 #1] INFO -- : I, [2022-02-15T05:28:05.618268 #1] INFO -- : > /usr/local/bin/ruby -e 'if (ENV["DISCOURSE_CDN_URL"] || "")[0..1] == "//"; puts "Aborting! CDN must have a protocol specified. Once fixed you should rebake your posts now to correct all posts."; exit 1; end' I, [2022-02-15T05:28:05.754370 #1] INFO -- : I, [2022-02-15T05:28:05.755864 #1] INFO -- : > rm -f /etc/cron.d/anacron I, [2022-02-15T05:28:05.763371 #1] INFO -- : I, [2022-02-15T05:28:05.773372 #1] INFO -- : File > /etc/cron.d/anacron chmod: chown: I, [2022-02-15T05:28:05.785161 #1] INFO -- : File > /etc/runit/1.d/copy-env chmod: +x chown: I, [2022-02-15T05:28:05.801026 #1] INFO -- : File > /etc/service/unicorn/run chmod: +x chown: I, [2022-02-15T05:28:05.816469 #1] INFO -- : File > /etc/service/nginx/run chmod: +x chown: I, [2022-02-15T05:28:05.828667 #1] INFO -- : File > /etc/runit/3.d/01-nginx chmod: +x chown: I, [2022-02-15T05:28:05.841176 #1] INFO -- : File > /etc/runit/3.d/02-unicorn chmod: +x chown: I, [2022-02-15T05:28:05.841464 #1] INFO -- : Replacing # postgres with sv start postgres || exit 1 in /etc/service/unicorn/run I, [2022-02-15T05:28:05.842241 #1] INFO -- : > exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf I, [2022-02-15T05:28:05.847862 #1] INFO -- : > cd /var/www/discourse && git reset --hard 130:C 15 Feb 2022 05:28:05.853 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 130:C 15 Feb 2022 05:28:05.854 # Redis version=6.2.6, bits=64, commit=00000000, modified=0, pid=130, just started 130:C 15 Feb 2022 05:28:05.855 # Configuration loaded 130:M 15 Feb 2022 05:28:05.857 * monotonic clock: POSIX clock_gettime 130:M 15 Feb 2022 05:28:05.858 # Warning: Could not create server TCP listening socket *:6379: bind: Address already in use 130:M 15 Feb 2022 05:28:05.859 # Failed listening on port 6379 (TCP), aborting. Updating files: 100% (30947/30947), done. I, [2022-02-15T05:28:13.624544 #1] INFO -- : HEAD is now at e1367cde I18N: Improve copy on blocked words error messages (#15741) I, [2022-02-15T05:29:36.205328 #1] INFO -- : > cd /var/www/discourse && [ ! -d 'node_modules' ] || su discourse -c 'yarn install --production && yarn cache clean' warning Resolution field "lodash@4.17.21" is incompatible with requested version "lodash@4.17.15" warning " > @mixer/parallel-prettier@2.0.1" has unmet peer dependency "prettier@^2.0.0". I, [2022-02-15T05:33:01.692668 #1] INFO -- : yarn install v1.22.17 [1/5] Validating package.json... [2/5] Resolving packages... [3/5] Fetching packages... [4/5] Linking dependencies... [5/5] Building fresh packages... $ yarn --cwd app/assets/javascripts/discourse yarn install v1.22.17 [1/4] Resolving packages... [2/4] Fetching packages... [3/4] Linking dependencies... [4/4] Building fresh packages... success Saved lockfile. Done in 133.99s. Done in 195.02s. yarn cache v1.22.17 success Cleared cache. Done in 9.35s. I, [2022-02-15T05:33:01.697905 #1] INFO -- : > cd /var/www/discourse && su discourse -c 'LOAD_PLUGINS=0 bundle exec rake plugin:pull_compatible_all' I, [2022-02-15T05:33:12.324517 #1] INFO -- : discourse-github is already at latest compatible version discourse-solved is already at latest compatible version discourse-chat is already at latest compatible version I, [2022-02-15T05:33:12.325695 #1] INFO -- : > cd /var/www/discourse && su discourse -c 'bundle exec rake db:migrate' 102:M 15 Feb 2022 05:33:37.418 * 100 changes in 300 seconds. Saving... 102:M 15 Feb 2022 05:33:37.423 * Background saving started by pid 1143 1143:C 15 Feb 2022 05:33:38.349 * DB saved on disk 1143:C 15 Feb 2022 05:33:38.352 * RDB: 2 MB of memory used by copy-on-write 102:M 15 Feb 2022 05:33:38.429 * Background saving terminated with success I, [2022-02-15T05:33:38.994811 #1] INFO -- : gem install sawyer -v 0.8.2 -i /var/www/discourse/plugins/discourse-github/gems/2.7.5 --no-document --ignore-dependencies --no-user-install Successfully installed sawyer-0.8.2 1 gem installed gem install octokit -v 4.21.0 -i /var/www/discourse/plugins/discourse-github/gems/2.7.5 --no-document --ignore-dependencies --no-user-install Successfully installed octokit-4.21.0 1 gem installed == 20220214224506 ResetCustomEmojiPostBakesVersionSecureFix: migrating ======== == 20220214224506 ResetCustomEmojiPostBakesVersionSecureFix: migrated (0.0099s) == 20220214233625 AddBounceErrorCodeToEmailLog: migrating ===================== -- add_column(:email_logs, :bounce_error_code, :string, {:null=>true}) -> 0.0103s == 20220214233625 AddBounceErrorCodeToEmailLog: migrated (0.0121s) ============ I, [2022-02-15T05:33:38.996639 #1] INFO -- : > cd /var/www/discourse && su discourse -c 'bundle exec rake themes:update assets:precompile' - Building - Building - building... 102:M 15 Feb 2022 05:38:39.069 * 100 changes in 300 seconds. Saving... 102:M 15 Feb 2022 05:38:39.080 * Background saving started by pid 1213 1213:C 15 Feb 2022 05:38:39.863 * DB saved on disk 1213:C 15 Feb 2022 05:38:39.867 * RDB: 2 MB of memory used by copy-on-write 102:M 15 Feb 2022 05:38:39.883 * Background saving terminated with success [WARN] (broccoli-terser-sourcemap) Minifying "assets/discourse.js" took: 35001ms (more than 20,000ms) Killed error Command failed with exit code 137. I, [2022-02-15T05:52:50.058283 #1] INFO -- : yarn run v1.22.17 $ /var/www/discourse/app/assets/javascripts/node_modules/.bin/ember build -prod WARNING: Node v16.13.2 is not tested against Ember CLI on your platform. We recommend that you use the most-recent "Active LTS" version of Node.js. See https://git.io/v7S5n for details. DEPRECATION: The integration of jQuery into Ember has been deprecated and will be removed with Ember 4.0. You can either opt-out of using jQuery, or install the `@ember/jquery` addon to provide the jQuery integration. Please consult the deprecation guide for further details: https://emberjs.com/deprecations/v3.x#toc_jquery-apis Environment: production info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command. I, [2022-02-15T05:52:50.072549 #1] INFO -- : Terminating async processes I, [2022-02-15T05:52:50.072816 #1] INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main pid: 41 I, [2022-02-15T05:52:50.073275 #1] INFO -- : Sending TERM to exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf pid: 102 102:signal-handler (1644904370) Received SIGTERM scheduling shutdown... 2022-02-15 05:52:50.074 UTC [41] LOG: received fast shutdown request 2022-02-15 05:52:50.090 UTC [41] LOG: aborting any active transactions 2022-02-15 05:52:50.096 UTC [41] LOG: background worker "logical replication launcher" (PID 50) exited with exit code 1 2022-02-15 05:52:50.102 UTC [45] LOG: shutting down 102:M 15 Feb 2022 05:52:50.118 # User requested shutdown... 102:M 15 Feb 2022 05:52:50.120 * Saving the final RDB snapshot before exiting. 2022-02-15 05:52:50.248 UTC [41] LOG: database system is shut down 102:M 15 Feb 2022 05:52:51.075 * DB saved on disk 102:M 15 Feb 2022 05:52:51.076 # Redis is now ready to exit, bye bye... FAILED -------------------- Pups::ExecError: cd /var/www/discourse && su discourse -c 'bundle exec rake themes:update assets:precompile' failed with return # Location of failure: /usr/local/lib/ruby/gems/2.7.0/gems/pups-1.1.1/lib/pups/exec_command.rb:117:in `spawn' exec failed with the params {"cd"=>"$home", "hook"=>"assets_precompile", "cmd"=>["su discourse -c 'bundle exec rake themes:update assets:precompile'"]} 1716fd9590e21cdfdaccea0ac6faabd133a6423e1a24ef3b3b5be0e68ce422c9 ** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one. ./discourse-doctor may help diagnose the problem. Running discourse-doctor now, and gathering more detailed logs. This is the upgrade script I made and have been using successfully for a year or more: https://github.com/balupton/dotfiles/blob/master/commands/discourse-upgrader

[image] balupton: It is the VC1S server from Scaleway You need to add some swap. Looks like you have plenty of disk, so you could add 2G or even 3G for safety. The discourse standard setup should, I think, have already set up the swap for you. Perhaps see Create a swapfile for your Linux server

I can confirm this, I’ve tried this on 3 Different instances and builds are getting failed after reaching the below state. I, [2022-02-15T07:09:36.714521 #1] INFO -- : > cd /var/www/discourse && su discourse -c 'bundle exec rake db:migrate' I, [2022-02-15T07:09:46.598212 #1] INFO -- : I, [2022-02-15T07:09:46.599983 #1] INFO -- : > cd /var/www/discourse && su discourse -c 'bundle exec rake themes:update assets:precompile' - Building - Building - building... Killed error Command failed with exit code 137. I, [2022-02-15T07:30:11.622116 #1] INFO -- : Checking 'Easy Footer' for 'default'... up to date Checking 'Discourse Loading Slider' for 'default'... up to date Checking 'discourse-search-banner' for 'sandbox'... up to date Checking 'discourse-category-sidebars' for 'sandbox'... up to date Checking 'Hide Users Column' for 'sandbox'... up to date Checking 'discourse-sidebar-nav' for 'sandbox'... up to date Checking 'Dracula Theme' for 'sandbox'... up to date Checking 'discourse-buttons' for 'tekduke'... yarn run v1.22.17 $ /var/www/discourse/app/assets/javascripts/node_modules/.bin/ember build -prod WARNING: Node v16.13.2 is not tested against Ember CLI on your platform. We recommend that you use the most-recent "Active LTS" version of Node.js. See https://git.io/v7S5n for details. DEPRECATION: The integration of jQuery into Ember has been deprecated and will be removed with Ember 4.0. You can either opt-out of using jQuery, or install the `@ember/jquery` addon to provide the jQuery integration. Please consult the deprecation guide for further details: https://emberjs.com/deprecations/v3.x#toc_jquery-apis Environment: production info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command. FAILED -------------------- Pups::ExecError: cd /var/www/discourse && su discourse -c 'bundle exec rake themes:update assets:precompile' failed with return # Location of failure: /usr/local/lib/ruby/gems/2.7.0/gems/pups-1.1.1/lib/pups/exec_command.rb:117:in `spawn' exec failed with the params {"cd"=>"$home", "hook"=>"assets_precompile", "cmd"=>["su discourse -c 'bundle exec rake themes:update assets:precompile'"]} 49ed46d025f5fe0a3de022b073b507cc3d94f74f96a60fe2d55042d413def454 ** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one. ./discourse-doctor may help diagnose the problem. it looks like ember CLI related issue.

hey @balupton could you try changing EMBER_CLI_PROD_ASSETS: 0 and try rebuilding again? I have at least 1 successful rebuild after the above change.

These seem to be the only relevant logs that indicate anything is wrong: I, [2022-02-15T07:15:48.531001 #1] INFO -- : > cd /var/www/discourse && [ ! -d 'node_modules' ] || su discourse -c 'yarn install --production && yarn cache clean' warning Resolution field "lodash@4.17.21" is incompatible with requested version "lodash@4.17.15" warning " > @mixer/parallel-prettier@2.0.1" has unmet peer dependency "prettier@^2.0.0". I, [2022-02-15T07:19:45.453691 #1] INFO -- : > cd /var/www/discourse && su discourse -c 'bundle exec rake themes:update assets:precompile' - Building - Building - building... Killed error Command failed with exit code 137. I, [2022-02-15T07:31:39.549100 #1] INFO -- : yarn run v1.22.17 $ /var/www/discourse/app/assets/javascripts/node_modules/.bin/ember build -prod WARNING: Node v16.13.2 is not tested against Ember CLI on your platform. We recommend that you use the most-recent "Active LTS" version of Node.js. See https://git.io/v7S5n for details. DEPRECATION: The integration of jQuery into Ember has been deprecated and will be removed with Ember 4.0. You can either opt-out of using jQuery, or install the `@ember/jquery` addon to provide the jQuery integration. Please consult the deprecation guide for further details: https://emberjs.com/deprecations/v3.x#toc_jquery-apis Environment: production info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command. FAILED -------------------- Pups::ExecError: cd /var/www/discourse && su discourse -c 'bundle exec rake themes:update assets:precompile' failed with return # Location of failure: /usr/local/lib/ruby/gems/2.7.0/gems/pups-1.1.1/lib/pups/exec_command.rb:117:in `spawn' exec failed with the params {"cd"=>"$home", "hook"=>"assets_precompile", "cmd"=>["su discourse -c 'bundle exec rake themes:update assets:precompile'"]} a262c461f2ae24390cb1397df309c44738cdc2688483d5fb7d52a3fde3155409 ** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one. ./discourse-doctor may help diagnose the problem.

Will try that, how do I apply it? Nevermind, found it: [image] New installs will default to Ember CLI builds in Production megaphoneAnnouncements As of today, all new self hosted installs of Discourse will default to using our Ember CLI builds in production. We’ve been running these builds ourselves in production for quite some time now and they should be stable and work with all major plugins. If you encounter any issues and need to turn it off, edit your app.yml and remove EMBER_CLI_PROD_ASSETS: 1 Otherwise, please report any bugs to us and we’ll be quick to fix them. In the near future all installs of Discourse will be using the Emb… Will give it a go now.

Looking at the changelog: [image] Comparing 55007fbf5597e8d6c52464b93f519794bd9fb516...main · discourse/discourse A platform for community discussion. Free, open, simple. - Comparing 55007fbf5597e8d6c52464b93f519794bd9fb516...main · discourse/discourse I wonder if it is this: https://github.com/discourse/discourse/pull/15919 As one of the logs was: 103:M 15 Feb 2022 06:54:18.720 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.

Yep, it worked. vim /var/discourse/containers/app.yml env: # ... ## Workaround for: https://meta.discourse.org/t/failed-to-upgrade-discourse-instance-to-feb-15-2022/218204/7?u=balupton EMBER_CLI_PROD_ASSETS: 0

[image] RGJ: Make sure you have enough (virtual) memory. 1GB of memory and 2GB of swap is probably too low nowadays for a rebuild. with discourse now successfully running, top is currently returning: KiB Mem : **2041324** total, **287808** free, **1033300** used, **720216** buff/cache KiB Swap: **0** total, **0** free, **0** used. **581308** avail Mem It is the VC1S server from Scaleway: https://serverscope.io/providers/scaleway/plans/vc1s Would doing the suggestion in: 103:M 15 Feb 2022 06:54:18.720 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. Also solve it?

The VPS has 4GB RAM and 2GB Swap.

[image] itsbhanusharma: The VPS has 4GB RAM and 2GB Swap. And yet you see an out of memory - that’s worrying. Please paste your output of the free command. [image] balupton: Would doing the suggestion in: 103:M 15 Feb 2022 06:54:18.720 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. Also solve it? I notice that in both my installations (both have just 1G of RAM and 2G of swap) I have at some point followed that advice and set up vm.overcommit_memory=1 in /etc/sysctl.conf. (That’s not to say it would solve this current problem.)

Failed to upgrade discourse instance to Feb 15 2022

Product Support Self-hosting

RGJ (Richard - Communiteq) February 15, 2022, 8:10am 9

Out of memory.

Make sure you have enough (virtual) memory. 1GB of memory and 2GB of swap is probably too low nowadays for a rebuild.

Topic		Replies	Views
Error while updating through launcher rebuild Self-hosting	2	682	April 16, 2024
3.1.x to 3.2.0 upgrade hangs/fails on 1GB instance Self-hosting	34	2220	February 23, 2024
I keep getting errors when updating Product Support	26	762	July 16, 2025
Problem with Upgrade [error 137] Self-hosting server-resources	31	2040	February 11, 2024
Discourse down. Updated certificate, rebuilt app, but still not working Self-hosting two-container	13	790	March 22, 2024

Failed to upgrade discourse instance to Feb 15 2022

Related topics