Discourse offline after failed upgrade "bootstrap failed with exit code 5"

Hi everyone,

My Discourse is currently offline :cry: :pensive:

It failed after trying to upgrade this morning from/to v3.2.0.beta3 +468:

On the command line, the error message is:

And the full console log:

root@greyarro:~# cd /var/discourse/
root@greyarro:/var/discourse# git pull
Already up to date.
root@greyarro:/var/discourse# ./launcher rebuild app
x86_64 arch detected.
WARNING: containers/app.yml file is world-readable. You can secure this file by running: chmod o-rwx containers/app.yml
Ensuring launcher is up to date
Fetching origin
Launcher is up-to-date
2.0.20231023-1945: Pulling from discourse/base
Digest: sha256:2b0eb484d20888cc2daadb690dcfa73522105650c1420212e99345a36a424d77
Status: Image is up to date for discourse/base:2.0.20231023-1945
/usr/local/lib/ruby/gems/3.2.0/gems/pups-1.2.1/lib/pups.rb
/usr/local/bin/pups --stdin
I, [2023-12-11T06:36:42.453494 #1]  INFO -- : Reading from stdin
I, [2023-12-11T06:36:42.461560 #1]  INFO -- : > locale-gen $LANG && update-locale
I, [2023-12-11T06:36:42.523575 #1]  INFO -- : Generating locales (this might take a while)...
Generation complete.

I, [2023-12-11T06:36:42.524082 #1]  INFO -- : > mkdir -p /shared/postgres_run
I, [2023-12-11T06:36:42.529018 #1]  INFO -- : 
I, [2023-12-11T06:36:42.529478 #1]  INFO -- : > chown postgres:postgres /shared/postgres_run
I, [2023-12-11T06:36:42.534085 #1]  INFO -- : 
I, [2023-12-11T06:36:42.534610 #1]  INFO -- : > chmod 775 /shared/postgres_run
I, [2023-12-11T06:36:42.538665 #1]  INFO -- : 
I, [2023-12-11T06:36:42.539119 #1]  INFO -- : > rm -fr /var/run/postgresql
I, [2023-12-11T06:36:42.542429 #1]  INFO -- : 
I, [2023-12-11T06:36:42.542918 #1]  INFO -- : > ln -s /shared/postgres_run /var/run/postgresql
I, [2023-12-11T06:36:42.545917 #1]  INFO -- : 
I, [2023-12-11T06:36:42.546272 #1]  INFO -- : > socat /dev/null UNIX-CONNECT:/shared/postgres_run/.s.PGSQL.5432 || exit 0 && echo postgres already running stop container ; exit 1
2023/12/11 06:36:42 socat[17] E connect(6, AF=1 "/shared/postgres_run/.s.PGSQL.5432", 36): No such file or directory
I, [2023-12-11T06:36:42.557289 #1]  INFO -- : 
I, [2023-12-11T06:36:42.557926 #1]  INFO -- : > rm -fr /shared/postgres_run/.s*
I, [2023-12-11T06:36:42.562906 #1]  INFO -- : 
I, [2023-12-11T06:36:42.563405 #1]  INFO -- : > rm -fr /shared/postgres_run/*.pid
I, [2023-12-11T06:36:42.568125 #1]  INFO -- : 
I, [2023-12-11T06:36:42.568537 #1]  INFO -- : > mkdir -p /shared/postgres_run/13-main.pg_stat_tmp
I, [2023-12-11T06:36:42.571897 #1]  INFO -- : 
I, [2023-12-11T06:36:42.572328 #1]  INFO -- : > chown postgres:postgres /shared/postgres_run/13-main.pg_stat_tmp
I, [2023-12-11T06:36:42.575429 #1]  INFO -- : 
I, [2023-12-11T06:36:42.583069 #1]  INFO -- : File > /etc/service/postgres/run  chmod: +x  chown: 
I, [2023-12-11T06:36:42.591097 #1]  INFO -- : File > /etc/service/postgres/log/run  chmod: +x  chown: 
I, [2023-12-11T06:36:42.599557 #1]  INFO -- : File > /etc/runit/3.d/99-postgres  chmod: +x  chown: 
I, [2023-12-11T06:36:42.607693 #1]  INFO -- : File > /root/upgrade_postgres  chmod: +x  chown: 
I, [2023-12-11T06:36:42.608293 #1]  INFO -- : > chown -R root /var/lib/postgresql/13/main
I, [2023-12-11T06:36:43.061308 #1]  INFO -- : 
I, [2023-12-11T06:36:43.061769 #1]  INFO -- : > [ ! -e /shared/postgres_data ] && install -d -m 0755 -o postgres -g postgres /shared/postgres_data && sudo -E -u postgres /usr/lib/postgresql/13/bin/initdb -D /shared/postgres_data || exit 0
I, [2023-12-11T06:36:43.066134 #1]  INFO -- : 
I, [2023-12-11T06:36:43.066571 #1]  INFO -- : > chown -R postgres:postgres /shared/postgres_data
I, [2023-12-11T06:36:43.105049 #1]  INFO -- : 
I, [2023-12-11T06:36:43.105891 #1]  INFO -- : > chown -R postgres:postgres /var/run/postgresql
I, [2023-12-11T06:36:43.111420 #1]  INFO -- : 
I, [2023-12-11T06:36:43.112072 #1]  INFO -- : > /root/upgrade_postgres
I, [2023-12-11T06:36:43.122177 #1]  INFO -- : 
I, [2023-12-11T06:36:43.122713 #1]  INFO -- : > rm /root/upgrade_postgres
I, [2023-12-11T06:36:43.126702 #1]  INFO -- : 
I, [2023-12-11T06:36:43.127981 #1]  INFO -- : Replacing data_directory = '/var/lib/postgresql/13/main' with data_directory = '/shared/postgres_data' in /etc/postgresql/13/main/postgresql.conf
I, [2023-12-11T06:36:43.128927 #1]  INFO -- : Replacing (?-mix:#?listen_addresses *=.*) with listen_addresses = '*' in /etc/postgresql/13/main/postgresql.conf
I, [2023-12-11T06:36:43.129583 #1]  INFO -- : Replacing (?-mix:#?synchronous_commit *=.*) with synchronous_commit = $db_synchronous_commit in /etc/postgresql/13/main/postgresql.conf
I, [2023-12-11T06:36:43.130214 #1]  INFO -- : Replacing (?-mix:#?shared_buffers *=.*) with shared_buffers = $db_shared_buffers in /etc/postgresql/13/main/postgresql.conf
I, [2023-12-11T06:36:43.130899 #1]  INFO -- : Replacing (?-mix:#?work_mem *=.*) with work_mem = $db_work_mem in /etc/postgresql/13/main/postgresql.conf
I, [2023-12-11T06:36:43.131457 #1]  INFO -- : Replacing (?-mix:#?default_text_search_config *=.*) with default_text_search_config = '$db_default_text_search_config' in /etc/postgresql/13/main/postgresql.conf
I, [2023-12-11T06:36:43.132096 #1]  INFO -- : > install -d -m 0755 -o postgres -g postgres /shared/postgres_backup
I, [2023-12-11T06:36:43.137173 #1]  INFO -- : 
I, [2023-12-11T06:36:43.137747 #1]  INFO -- : Replacing (?-mix:#?checkpoint_segments *=.*) with checkpoint_segments = $db_checkpoint_segments in /etc/postgresql/13/main/postgresql.conf
I, [2023-12-11T06:36:43.138339 #1]  INFO -- : Replacing (?-mix:#?logging_collector *=.*) with logging_collector = $db_logging_collector in /etc/postgresql/13/main/postgresql.conf
I, [2023-12-11T06:36:43.139065 #1]  INFO -- : Replacing (?-mix:#?log_min_duration_statement *=.*) with log_min_duration_statement = $db_log_min_duration_statement in /etc/postgresql/13/main/postgresql.conf
I, [2023-12-11T06:36:43.140177 #1]  INFO -- : Replacing (?-mix:^#local +replication +postgres +peer$) with local replication postgres  peer in /etc/postgresql/13/main/pg_hba.conf
I, [2023-12-11T06:36:43.140923 #1]  INFO -- : Replacing (?-mix:^host.*all.*all.*127.*$) with host all all 0.0.0.0/0 md5 in /etc/postgresql/13/main/pg_hba.conf
I, [2023-12-11T06:36:43.141403 #1]  INFO -- : Replacing (?-mix:^host.*all.*all.*::1\/128.*$) with host all all ::/0 md5 in /etc/postgresql/13/main/pg_hba.conf
I, [2023-12-11T06:36:43.141961 #1]  INFO -- : > HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main
I, [2023-12-11T06:36:43.144661 #1]  INFO -- : > sleep 5
2023-12-11 06:36:43.213 UTC [40] LOG:  starting PostgreSQL 13.12 (Debian 13.12-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
2023-12-11 06:36:43.213 UTC [40] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2023-12-11 06:36:43.213 UTC [40] LOG:  listening on IPv6 address "::", port 5432
2023-12-11 06:36:43.217 UTC [40] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2023-12-11 06:36:43.223 UTC [43] LOG:  database system was shut down at 2023-12-11 06:33:31 UTC
2023-12-11 06:36:43.236 UTC [40] LOG:  database system is ready to accept connections
I, [2023-12-11T06:36:48.151212 #1]  INFO -- : 
I, [2023-12-11T06:36:48.151469 #1]  INFO -- : > su postgres -c 'createdb discourse' || true
2023-12-11 06:36:48.241 UTC [53] postgres@postgres ERROR:  database "discourse" already exists
2023-12-11 06:36:48.241 UTC [53] postgres@postgres STATEMENT:  CREATE DATABASE discourse;
createdb: error: database creation failed: ERROR:  database "discourse" already exists
I, [2023-12-11T06:36:48.244084 #1]  INFO -- : 
I, [2023-12-11T06:36:48.244295 #1]  INFO -- : > su postgres -c 'psql discourse -c "create user discourse;"' || true
2023-12-11 06:36:48.310 UTC [57] postgres@discourse ERROR:  role "discourse" already exists
2023-12-11 06:36:48.310 UTC [57] postgres@discourse STATEMENT:  create user discourse;
ERROR:  role "discourse" already exists
I, [2023-12-11T06:36:48.313540 #1]  INFO -- : 
I, [2023-12-11T06:36:48.313932 #1]  INFO -- : > su postgres -c 'psql discourse -c "grant all privileges on database discourse to discourse;"' || true
I, [2023-12-11T06:36:48.372120 #1]  INFO -- : GRANT

I, [2023-12-11T06:36:48.372374 #1]  INFO -- : > su postgres -c 'psql discourse -c "alter schema public owner to discourse;"'
I, [2023-12-11T06:36:48.428450 #1]  INFO -- : ALTER SCHEMA

I, [2023-12-11T06:36:48.428697 #1]  INFO -- : > su postgres -c 'psql template1 -c "create extension if not exists hstore;"'
NOTICE:  extension "hstore" already exists, skipping
I, [2023-12-11T06:36:48.509937 #1]  INFO -- : CREATE EXTENSION

I, [2023-12-11T06:36:48.510198 #1]  INFO -- : > su postgres -c 'psql template1 -c "create extension if not exists pg_trgm;"'
NOTICE:  extension "pg_trgm" already exists, skipping
I, [2023-12-11T06:36:48.575901 #1]  INFO -- : CREATE EXTENSION

I, [2023-12-11T06:36:48.576560 #1]  INFO -- : > su postgres -c 'psql template1 -c "create extension if not exists vector;"'
NOTICE:  extension "vector" already exists, skipping
I, [2023-12-11T06:36:48.634184 #1]  INFO -- : CREATE EXTENSION

I, [2023-12-11T06:36:48.634407 #1]  INFO -- : > su postgres -c 'psql discourse -c "create extension if not exists hstore;"'
NOTICE:  extension "hstore" already exists, skipping
I, [2023-12-11T06:36:48.691197 #1]  INFO -- : CREATE EXTENSION

I, [2023-12-11T06:36:48.691402 #1]  INFO -- : > su postgres -c 'psql discourse -c "create extension if not exists pg_trgm;"'
NOTICE:  extension "pg_trgm" already exists, skipping
I, [2023-12-11T06:36:48.747583 #1]  INFO -- : CREATE EXTENSION

I, [2023-12-11T06:36:48.747801 #1]  INFO -- : > su postgres -c 'psql discourse -c "create extension if not exists vector;"'
NOTICE:  extension "vector" already exists, skipping
I, [2023-12-11T06:36:48.805422 #1]  INFO -- : CREATE EXTENSION

I, [2023-12-11T06:36:48.805817 #1]  INFO -- : > sudo -u postgres psql discourse
I, [2023-12-11T06:36:48.808688 #1]  INFO -- : update pg_database set encoding = pg_char_to_encoding('UTF8') where datname = 'discourse' AND encoding = pg_char_to_encoding('SQL_ASCII');

I, [2023-12-11T06:36:48.891679 #1]  INFO -- : File > /var/lib/postgresql/take-database-backup  chmod: +x  chown: postgres:postgres
I, [2023-12-11T06:36:48.896830 #1]  INFO -- : File > /var/spool/cron/crontabs/postgres  chmod:   chown: 
I, [2023-12-11T06:36:48.896998 #1]  INFO -- : > echo postgres installed!
I, [2023-12-11T06:36:48.899962 #1]  INFO -- : postgres installed!

I, [2023-12-11T06:36:48.907627 #1]  INFO -- : File > /etc/service/redis/run  chmod: +x  chown: 
I, [2023-12-11T06:36:48.914411 #1]  INFO -- : File > /etc/service/redis/log/run  chmod: +x  chown: 
I, [2023-12-11T06:36:48.920982 #1]  INFO -- : File > /etc/runit/3.d/10-redis  chmod: +x  chown: 
I, [2023-12-11T06:36:48.922023 #1]  INFO -- : Replacing daemonize yes with  in /etc/redis/redis.conf
I, [2023-12-11T06:36:48.922829 #1]  INFO -- : Replacing (?-mix:^pidfile.*$) with  in /etc/redis/redis.conf
I, [2023-12-11T06:36:48.923923 #1]  INFO -- : > install -d -m 0755 -o redis -g redis /shared/redis_data
I, [2023-12-11T06:36:48.928742 #1]  INFO -- : 
I, [2023-12-11T06:36:48.929799 #1]  INFO -- : Replacing (?-mix:^logfile.*$) with logfile "" in /etc/redis/redis.conf
I, [2023-12-11T06:36:48.930627 #1]  INFO -- : Replacing (?-mix:^bind .*$) with  in /etc/redis/redis.conf
I, [2023-12-11T06:36:48.931845 #1]  INFO -- : Replacing (?-mix:^dir .*$) with dir /shared/redis_data in /etc/redis/redis.conf
I, [2023-12-11T06:36:48.933205 #1]  INFO -- : Replacing (?-mix:^protected-mode yes) with protected-mode no in /etc/redis/redis.conf
I, [2023-12-11T06:36:48.934373 #1]  INFO -- : Replacing # io-threads 4 with io-threads $redis_io_threads in /etc/redis/redis.conf
I, [2023-12-11T06:36:48.935362 #1]  INFO -- : > echo redis installed
I, [2023-12-11T06:36:48.939050 #1]  INFO -- : redis installed

I, [2023-12-11T06:36:48.939481 #1]  INFO -- : > cat /etc/redis/redis.conf | grep logfile
I, [2023-12-11T06:36:48.947715 #1]  INFO -- : logfile ""

I, [2023-12-11T06:36:48.948224 #1]  INFO -- : > exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf
I, [2023-12-11T06:36:48.950934 #1]  INFO -- : > sleep 10
109:C 11 Dec 2023 06:36:48.962 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
109:C 11 Dec 2023 06:36:48.962 # Redis version=7.0.7, bits=64, commit=00000000, modified=0, pid=109, just started
109:C 11 Dec 2023 06:36:48.962 # Configuration loaded
109:M 11 Dec 2023 06:36:48.963 * monotonic clock: POSIX clock_gettime
109:M 11 Dec 2023 06:36:48.964 * Running mode=standalone, port=6379.
109:M 11 Dec 2023 06:36:48.964 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
109:M 11 Dec 2023 06:36:48.964 # Server initialized
109:M 11 Dec 2023 06:36:48.964 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
109:M 11 Dec 2023 06:36:48.965 * Loading RDB produced by version 7.0.7
109:M 11 Dec 2023 06:36:48.965 * RDB age 196 seconds
109:M 11 Dec 2023 06:36:48.965 * RDB memory usage when created 238.89 Mb
109:M 11 Dec 2023 06:36:49.836 * Done loading RDB, keys loaded: 78979, keys expired: 55.
109:M 11 Dec 2023 06:36:49.836 * DB loaded from disk: 0.871 seconds
109:M 11 Dec 2023 06:36:49.836 * Ready to accept connections
I, [2023-12-11T06:36:58.954574 #1]  INFO -- : 
I, [2023-12-11T06:36:58.955135 #1]  INFO -- : > thpoff echo "thpoff is installed!"
I, [2023-12-11T06:36:58.960865 #1]  INFO -- : thpoff is installed!

I, [2023-12-11T06:36:58.961207 #1]  INFO -- : > /usr/local/bin/ruby -e 'if ENV["DISCOURSE_SMTP_ADDRESS"] == "smtp.example.com"; puts "Aborting! Mail is not configured!"; exit 1; end'
I, [2023-12-11T06:36:59.059159 #1]  INFO -- : 
I, [2023-12-11T06:36:59.059736 #1]  INFO -- : > /usr/local/bin/ruby -e 'if ENV["DISCOURSE_HOSTNAME"] == "discourse.example.com"; puts "Aborting! Domain is not configured!"; exit 1; end'
I, [2023-12-11T06:36:59.148371 #1]  INFO -- : 
I, [2023-12-11T06:36:59.149030 #1]  INFO -- : > /usr/local/bin/ruby -e 'if (ENV["DISCOURSE_CDN_URL"] || "")[0..1] == "//"; puts "Aborting! CDN must have a protocol specified. Once fixed you should rebake your posts now to correct all posts."; exit 1; end'
I, [2023-12-11T06:36:59.236730 #1]  INFO -- : 
I, [2023-12-11T06:36:59.237387 #1]  INFO -- : > rm -f /etc/cron.d/anacron
I, [2023-12-11T06:36:59.242323 #1]  INFO -- : 
I, [2023-12-11T06:36:59.247436 #1]  INFO -- : File > /etc/cron.d/anacron  chmod:   chown: 
I, [2023-12-11T06:36:59.254458 #1]  INFO -- : File > /etc/runit/1.d/copy-env  chmod: +x  chown: 
I, [2023-12-11T06:36:59.261375 #1]  INFO -- : File > /etc/service/unicorn/run  chmod: +x  chown: 
I, [2023-12-11T06:36:59.268779 #1]  INFO -- : File > /etc/service/nginx/run  chmod: +x  chown: 
I, [2023-12-11T06:36:59.276427 #1]  INFO -- : File > /etc/runit/3.d/01-nginx  chmod: +x  chown: 
I, [2023-12-11T06:36:59.283178 #1]  INFO -- : File > /etc/runit/3.d/02-unicorn  chmod: +x  chown: 
I, [2023-12-11T06:36:59.283492 #1]  INFO -- : Replacing # postgres with sv start postgres || exit 1 in /etc/service/unicorn/run
I, [2023-12-11T06:36:59.284057 #1]  INFO -- : > exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf
I, [2023-12-11T06:36:59.287039 #1]  INFO -- : > cd /var/www/discourse && sudo -H -E -u discourse git reset --hard
137:C 11 Dec 2023 06:36:59.295 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
137:C 11 Dec 2023 06:36:59.295 # Redis version=7.0.7, bits=64, commit=00000000, modified=0, pid=137, just started
137:C 11 Dec 2023 06:36:59.295 # Configuration loaded
137:M 11 Dec 2023 06:36:59.295 * monotonic clock: POSIX clock_gettime
137:M 11 Dec 2023 06:36:59.295 # Warning: Could not create server TCP listening socket *:6379: bind: Address already in use
137:M 11 Dec 2023 06:36:59.295 # Failed listening on port 6379 (TCP), aborting.
Updating files: 100% (33858/33858), done.
I, [2023-12-11T06:37:03.209327 #1]  INFO -- : HEAD is now at 043b4a418 FIX: replaces data-tooltip usage by <DTooltip /> (#24062)

I, [2023-12-11T06:37:03.209496 #1]  INFO -- : > cd /var/www/discourse && sudo -H -E -u discourse git clean -f
I, [2023-12-11T06:37:03.416955 #1]  INFO -- : 
I, [2023-12-11T06:37:03.417150 #1]  INFO -- : > cd /var/www/discourse && sudo -H -E -u discourse bash -c '
  set -o errexit
  if [ $(git rev-parse --is-shallow-repository) == "true" ]; then
      git remote set-branches --add origin main
      git remote set-branches origin tests-passed
      git fetch --depth 1 origin tests-passed
  else
      git fetch --tags --prune-tags --prune --force origin
  fi
'
From https://github.com/discourse/discourse
 - [deleted]             (none)     -> origin/0-flake-key
 - [deleted]             (none)     -> origin/0-grant-badge-fix
 - [deleted]             (none)     -> origin/0-obscuring
 - [deleted]             (none)     -> origin/0-router-lint
 - [deleted]             (none)     -> origin/0-router-service
 - [deleted]             (none)     -> origin/a-smokes-test
 - [deleted]             (none)     -> origin/autoimport-cleanup
 - [deleted]             (none)     -> origin/category-bullet-only
 - [deleted]             (none)     -> origin/dependabot/bundler/syntax_tree-6.2.0
 - [deleted]             (none)     -> origin/dependabot/npm_and_yarn/app/assets/javascripts/uppy/aws-s3-3.4.0
 - [deleted]             (none)     -> origin/dependabot/npm_and_yarn/app/assets/javascripts/uppy/aws-s3-multipart-3.8.0
 - [deleted]             (none)     -> origin/dependabot/npm_and_yarn/app/assets/javascripts/uppy/core-3.6.0
 - [deleted]             (none)     -> origin/dependabot/npm_and_yarn/app/assets/javascripts/uppy/utils-5.5.2
 - [deleted]             (none)     -> origin/dependabot/npm_and_yarn/app/assets/javascripts/uppy/xhr-upload-3.4.2
 - [deleted]             (none)     -> origin/dev-upgrade-reject-reason-reviewable-modal
 - [deleted]             (none)     -> origin/dev/ability-to-collect-stats-without-public-exposing
 - [deleted]             (none)     -> origin/dev/minio-s3-system-specs
 - [deleted]             (none)     -> origin/dev/re-enable-minio-specs
 - [deleted]             (none)     -> origin/dev/switch-over-category-settings
 - [deleted]             (none)     -> origin/dev/switch-over-more-category-settings
 - [deleted]             (none)     -> origin/discovery-named-outlets
 - [deleted]             (none)     -> origin/feature/theme-settings-migration
 - [deleted]             (none)     -> origin/fix/account-for-empty-string-site-settings
 - [deleted]             (none)     -> origin/footnote-plugin
 - [deleted]             (none)     -> origin/metadata-update
 - [deleted]             (none)     -> origin/modal-ux
 - [deleted]             (none)     -> origin/neil-2
 - [deleted]             (none)     -> origin/perf_cache_async_search
 - [deleted]             (none)     -> origin/redo-post-focus-setting
 - [deleted]             (none)     -> origin/spoiler-plugin
 - [deleted]             (none)     -> origin/support-search-customizationz
   043b4a418..4949d85c1  main                  -> origin/main
 * [new branch]          0-class-attribute     -> origin/0-class-attribute
 * [new branch]          0-for-attributes      -> origin/0-for-attributes
 * [new branch]          0-i18n-case           -> origin/0-i18n-case
 * [new branch]          add-custom-emoji-sanitization-clean -> origin/add-custom-emoji-sanitization-clean
 * [new branch]          add-ds-store-to-gitignore -> origin/add-ds-store-to-gitignore
 + 64520a075...ff17131c9 add-new-chat-indicator-preference-only-mentions -> origin/add-new-chat-indicator-preference-only-mentions  (forced update)
 * [new branch]          api-topic-title-decorator -> origin/api-topic-title-decorator
 * [new branch]          array-custom-field-order -> origin/array-custom-field-order
 * [new branch]          boosted-searches      -> origin/boosted-searches
 * [new branch]          bulk-button-class     -> origin/bulk-button-class
 * [new branch]          category_definition   -> origin/category_definition
 * [new branch]          chat-github-onebox    -> origin/chat-github-onebox
 * [new branch]          chat-reaction-emoji-alignment -> origin/chat-reaction-emoji-alignment
 * [new branch]          chat/direct-messages  -> origin/chat/direct-messages
 * [new branch]          confirm-session-only-after-5-mins -> origin/confirm-session-only-after-5-mins
 * [new branch]          custom-field-store-cleanup -> origin/custom-field-store-cleanup
 * [new branch]          delete-backups-based-on-time-window -> origin/delete-backups-based-on-time-window
 * [new branch]          dependabot/github_actions/actions/labeler-5 -> origin/dependabot/github_actions/actions/labeler-5
 * [new branch]          dependabot/github_actions/actions/stale-9 -> origin/dependabot/github_actions/actions/stale-9
 * [new branch]          dependabot/npm_and_yarn/app/assets/javascripts/uppy/aws-s3-3.5.0 -> origin/dependabot/npm_and_yarn/app/assets/javascripts/uppy/aws-s3-3.5.0
 * [new branch]          dependabot/npm_and_yarn/app/assets/javascripts/uppy/aws-s3-multipart-3.9.0 -> origin/dependabot/npm_and_yarn/app/assets/javascripts/uppy/aws-s3-multipart-3.9.0
 * [new branch]          dependabot/npm_and_yarn/app/assets/javascripts/uppy/core-3.7.1 -> origin/dependabot/npm_and_yarn/app/assets/javascripts/uppy/core-3.7.1
 * [new branch]          dependabot/npm_and_yarn/app/assets/javascripts/uppy/utils-5.6.0 -> origin/dependabot/npm_and_yarn/app/assets/javascripts/uppy/utils-5.6.0
 * [new branch]          dependabot/npm_and_yarn/app/assets/javascripts/uppy/xhr-upload-3.5.0 -> origin/dependabot/npm_and_yarn/app/assets/javascripts/uppy/xhr-upload-3.5.0
 * [new branch]          desktop-notification-handlers -> origin/desktop-notification-handlers
 * [new branch]          dev/add-a-reference-column-to-notifications -> origin/dev/add-a-reference-column-to-notifications
 + ee95e6359...a2b413786 dev/add-new-chat-metrics -> origin/dev/add-new-chat-metrics  (forced update)
 * [new branch]          dev/drop-notification_id-column-on-chat-mentions -> origin/dev/drop-notification_id-column-on-chat-mentions
 * [new branch]          dev/dry-up-user-status-serializers -> origin/dev/dry-up-user-status-serializers
 * [new branch]          dev/flaky-chat-details-collapser-spec -> origin/dev/flaky-chat-details-collapser-spec
 * [new branch]          dev/no-need-to-include-user-status-twice -> origin/dev/no-need-to-include-user-status-twice
 * [new branch]          dev/remove-legacy-hamburger-menu -> origin/dev/remove-legacy-hamburger-menu
 * [new branch]          dev/rethink-mass-mentions -> origin/dev/rethink-mass-mentions
 * [new branch]          dev/rethink-mass-mentions-polymorphic -> origin/dev/rethink-mass-mentions-polymorphic
 * [new branch]          dev/revert-guardian-devex -> origin/dev/revert-guardian-devex
 + 54940ca5c...3eacfe150 dev_paginate_categories_json -> origin/dev_paginate_categories_json  (forced update)
 * [new branch]          dev_watched_words     -> origin/dev_watched_words
 + 46fe4511f...a9d0e7069 ember-4               -> origin/ember-4  (forced update)
 * [new branch]          ember-5-plugins       -> origin/ember-5-plugins
 * [new branch]          extract_stripper      -> origin/extract_stripper
 * [new branch]          feature/admin-sidebar-v0 -> origin/feature/admin-sidebar-v0
   9a6c6fd5b..8780daeee  feature/bookmark-menu-and-modal-redesign -> origin/feature/bookmark-menu-and-modal-redesign
 * [new branch]          feature/migrations-schema-generator -> origin/feature/migrations-schema-generator
 * [new branch]          feature_watched_words_validation -> origin/feature_watched_words_validation
 * [new branch]          file-size-restriction -> origin/file-size-restriction
   43df37774..4393dd37e  filtered-events       -> origin/filtered-events
 * [new branch]          fix/count-mass-mentions-correctly -> origin/fix/count-mass-mentions-correctly
 * [new branch]          full-page             -> origin/full-page
   ce7fb6906..31e32f8a1  generic-import-latest -> origin/generic-import-latest
 * [new branch]          generic-import-latest-snowflake -> origin/generic-import-latest-snowflake
 * [new branch]          generic-import-update -> origin/generic-import-update
 * [new branch]          image-assets          -> origin/image-assets
 * [new branch]          independent-theme-compiler -> origin/independent-theme-compiler
 * [new branch]          input-text-field-component -> origin/input-text-field-component
 * [new branch]          issue/fix-guardian-init-basic-user -> origin/issue/fix-guardian-init-basic-user
 * [new branch]          keyword-site-setting  -> origin/keyword-site-setting
 + bdcc24f96...92e545292 list-wrapper          -> origin/list-wrapper  (forced update)
   e3636078c..5def32699  mock-chat-index-redesign -> origin/mock-chat-index-redesign
 * [new branch]          move_inflector        -> origin/move_inflector
 * [new branch]          multi-invite          -> origin/multi-invite
 * [new branch]          noti-glimmer          -> origin/noti-glimmer
 * [new branch]          oklab                 -> origin/oklab
 * [new branch]          poc/auto-after-before-plugin-outlets -> origin/poc/auto-after-before-plugin-outlets
 * [new branch]          poc/insert-before-after-wrapping-outlet -> origin/poc/insert-before-after-wrapping-outlet
 * [new branch]          pretty-text-please    -> origin/pretty-text-please
 * [new branch]          reload-page-user-prefs-2fa -> origin/reload-page-user-prefs-2fa
 * [new branch]          revert-maxmind        -> origin/revert-maxmind
   ef3f41543..223626d7a  stable                -> origin/stable
 * [new branch]          support-load-more-btn -> origin/support-load-more-btn
 * [new branch]          test-issue-temporary  -> origin/test-issue-temporary
   043b4a418..4949d85c1  tests-passed          -> origin/tests-passed
 * [new branch]          tmp/reimplement-all-and-here-mentions -> origin/tmp/reimplement-all-and-here-mentions
 * [new branch]          tobiaseigen-patch-3   -> origin/tobiaseigen-patch-3
 * [new branch]          transform-hooks       -> origin/transform-hooks
 * [new branch]          transform-post-fn-api -> origin/transform-post-fn-api
 * [new branch]          tweaks-2-fr           -> origin/tweaks-2-fr
 * [new branch]          username_cache        -> origin/username_cache
 * [new branch]          ux-add-sorting-params-to-outlet -> origin/ux-add-sorting-params-to-outlet
 t [tag update]          beta                  -> beta
 t [tag update]          latest-release        -> latest-release
 * [new tag]             v3.1.3                -> v3.1.3
 * [new tag]             v3.2.0.beta3          -> v3.2.0.beta3
I, [2023-12-11T06:37:05.107220 #1]  INFO -- : 
I, [2023-12-11T06:37:05.107337 #1]  INFO -- : > cd /var/www/discourse && sudo -H -E -u discourse bash -c '
  set -o errexit
  if [[ $(git symbolic-ref --short HEAD) == tests-passed ]] ; then
      git pull
  else
      git -c advice.detachedHead=false checkout tests-passed
  fi
'
Switched to a new branch 'tests-passed'
I, [2023-12-11T06:37:10.039336 #1]  INFO -- : Branch 'tests-passed' set up to track remote branch 'tests-passed' from 'origin'.

I, [2023-12-11T06:37:10.039485 #1]  INFO -- : > cd /var/www/discourse && sudo -H -E -u discourse git config user.discourse-version tests-passed
I, [2023-12-11T06:37:10.055796 #1]  INFO -- : 
I, [2023-12-11T06:37:10.055932 #1]  INFO -- : > cd /var/www/discourse && mkdir -p tmp
I, [2023-12-11T06:37:10.062383 #1]  INFO -- : 
I, [2023-12-11T06:37:10.062911 #1]  INFO -- : > cd /var/www/discourse && chown discourse:www-data tmp
I, [2023-12-11T06:37:10.068906 #1]  INFO -- : 
I, [2023-12-11T06:37:10.069051 #1]  INFO -- : > cd /var/www/discourse && mkdir -p tmp/pids
I, [2023-12-11T06:37:10.075337 #1]  INFO -- : 
I, [2023-12-11T06:37:10.075478 #1]  INFO -- : > cd /var/www/discourse && mkdir -p tmp/sockets
I, [2023-12-11T06:37:10.081087 #1]  INFO -- : 
I, [2023-12-11T06:37:10.081222 #1]  INFO -- : > cd /var/www/discourse && touch tmp/.gitkeep
I, [2023-12-11T06:37:10.087004 #1]  INFO -- : 
I, [2023-12-11T06:37:10.087139 #1]  INFO -- : > cd /var/www/discourse && mkdir -p                    /shared/log/rails
I, [2023-12-11T06:37:10.093140 #1]  INFO -- : 
I, [2023-12-11T06:37:10.093305 #1]  INFO -- : > cd /var/www/discourse && bash -c "touch -a           /shared/log/rails/{production,production_errors,unicorn.stdout,unicorn.stderr,sidekiq}.log"
I, [2023-12-11T06:37:10.100413 #1]  INFO -- : 
I, [2023-12-11T06:37:10.100570 #1]  INFO -- : > cd /var/www/discourse && bash -c "ln    -s           /shared/log/rails/{production,production_errors,unicorn.stdout,unicorn.stderr,sidekiq}.log /var/www/discourse/log"
I, [2023-12-11T06:37:10.108011 #1]  INFO -- : 
I, [2023-12-11T06:37:10.108130 #1]  INFO -- : > cd /var/www/discourse && bash -c "mkdir -p           /shared/{uploads,backups}"
I, [2023-12-11T06:37:10.116596 #1]  INFO -- : 
I, [2023-12-11T06:37:10.116726 #1]  INFO -- : > cd /var/www/discourse && bash -c "ln    -s           /shared/{uploads,backups} /var/www/discourse/public"
I, [2023-12-11T06:37:10.124273 #1]  INFO -- : 
I, [2023-12-11T06:37:10.124421 #1]  INFO -- : > cd /var/www/discourse && bash -c "mkdir -p           /shared/tmp/{backups,restores}"
I, [2023-12-11T06:37:10.134061 #1]  INFO -- : 
I, [2023-12-11T06:37:10.134228 #1]  INFO -- : > cd /var/www/discourse && bash -c "ln    -s           /shared/tmp/{backups,restores} /var/www/discourse/tmp"
I, [2023-12-11T06:37:10.141458 #1]  INFO -- : 
I, [2023-12-11T06:37:10.141602 #1]  INFO -- : > cd /var/www/discourse && chown -R discourse:www-data /shared/log/rails /shared/uploads /shared/backups /shared/tmp
I, [2023-12-11T06:37:10.163707 #1]  INFO -- : 
I, [2023-12-11T06:37:10.163854 #1]  INFO -- : > cd /var/www/discourse && [ ! -d public/plugins ] || find public/plugins/ -maxdepth 1 -xtype l -delete
I, [2023-12-11T06:37:10.169471 #1]  INFO -- : 
I, [2023-12-11T06:37:10.170368 #1]  INFO -- : Replacing # redis with sv start redis || exit 1 in /etc/service/unicorn/run
I, [2023-12-11T06:37:10.171519 #1]  INFO -- : > cd /var/www/discourse/plugins && git clone https://github.com/discourse/docker_manager.git
Cloning into 'docker_manager'...
I, [2023-12-11T06:37:11.142485 #1]  INFO -- : 
I, [2023-12-11T06:37:11.142680 #1]  INFO -- : > cd /var/www/discourse/plugins && git clone https://github.com/discourse/discourse-solved.git
Cloning into 'discourse-solved'...
I, [2023-12-11T06:37:11.877496 #1]  INFO -- : 
I, [2023-12-11T06:37:11.878167 #1]  INFO -- : > cd /var/www/discourse/plugins && git clone https://github.com/paviliondev/discourse-locations.git
Cloning into 'discourse-locations'...
I, [2023-12-11T06:37:12.713978 #1]  INFO -- : 
I, [2023-12-11T06:37:12.714105 #1]  INFO -- : > cd /var/www/discourse/plugins && git clone https://github.com/discourse/discourse-data-explorer.git
Cloning into 'discourse-data-explorer'...
I, [2023-12-11T06:37:13.532864 #1]  INFO -- : 
I, [2023-12-11T06:37:13.533001 #1]  INFO -- : > cd /var/www/discourse/plugins && git clone https://github.com/discourse/discourse-adplugin.git
Cloning into 'discourse-adplugin'...
I, [2023-12-11T06:37:14.305555 #1]  INFO -- : 
I, [2023-12-11T06:37:14.305817 #1]  INFO -- : > cd /var/www/discourse/plugins && git clone https://github.com/discourse/discourse-reactions.git
Cloning into 'discourse-reactions'...
I, [2023-12-11T06:37:15.045125 #1]  INFO -- : 
I, [2023-12-11T06:37:15.045247 #1]  INFO -- : > cd /var/www/discourse/plugins && git clone https://github.com/discourse/discourse-templates.git
Cloning into 'discourse-templates'...
I, [2023-12-11T06:37:15.809915 #1]  INFO -- : 
I, [2023-12-11T06:37:15.810046 #1]  INFO -- : > cd /var/www/discourse/plugins && git clone https://github.com/discourse/discourse-affiliate.git
Cloning into 'discourse-affiliate'...
I, [2023-12-11T06:37:16.433061 #1]  INFO -- : 
I, [2023-12-11T06:37:16.434123 #1]  INFO -- : > cp /var/www/discourse/config/nginx.sample.conf /etc/nginx/conf.d/discourse.conf
I, [2023-12-11T06:37:16.442090 #1]  INFO -- : 
I, [2023-12-11T06:37:16.442586 #1]  INFO -- : > rm /etc/nginx/sites-enabled/default
I, [2023-12-11T06:37:16.446759 #1]  INFO -- : 
I, [2023-12-11T06:37:16.447263 #1]  INFO -- : > mkdir -p /var/nginx/cache
I, [2023-12-11T06:37:16.451474 #1]  INFO -- : 
I, [2023-12-11T06:37:16.452405 #1]  INFO -- : Replacing pid /run/nginx.pid; with daemon off; in /etc/nginx/nginx.conf
I, [2023-12-11T06:37:16.453144 #1]  INFO -- : Replacing (?m-ix:upstream[^\}]+\}) with upstream discourse { server 127.0.0.1:3000; } in /etc/nginx/conf.d/discourse.conf
I, [2023-12-11T06:37:16.453873 #1]  INFO -- : Replacing (?-mix:server_name.+$) with server_name _ ; in /etc/nginx/conf.d/discourse.conf
I, [2023-12-11T06:37:16.454418 #1]  INFO -- : Replacing (?-mix:client_max_body_size.+$) with client_max_body_size $upload_size ; in /etc/nginx/conf.d/discourse.conf
I, [2023-12-11T06:37:16.454991 #1]  INFO -- : Replacing (?-mix:worker_connections.+$) with worker_connections $nginx_worker_connections ; in /etc/nginx/nginx.conf
I, [2023-12-11T06:37:16.455481 #1]  INFO -- : > echo "done configuring web"
I, [2023-12-11T06:37:16.459601 #1]  INFO -- : done configuring web

I, [2023-12-11T06:37:16.460326 #1]  INFO -- : > cd /var/www/discourse && gem install bundler --conservative -v $(awk '/BUNDLED WITH/ { getline; gsub(/ /,""); print $0 }' Gemfile.lock)
I, [2023-12-11T06:37:16.836806 #1]  INFO -- : 
I, [2023-12-11T06:37:16.836974 #1]  INFO -- : > cd /var/www/discourse && find /var/www/discourse ! -user discourse -exec chown discourse {} \+
I, [2023-12-11T06:37:19.438303 #1]  INFO -- : 
I, [2023-12-11T06:37:19.438984 #1]  INFO -- : > cd /var/www/discourse && if [ "tests-passed" != "tests-passed" ]; then
  rm -rf app/assets/javascripts/node_modules
fi
I, [2023-12-11T06:37:19.442692 #1]  INFO -- : 
I, [2023-12-11T06:37:19.442789 #1]  INFO -- : > cd /var/www/discourse && su discourse -c 'yarn install --frozen-lockfile && yarn cache clean'
warning " > @glint/environment-ember-loose@1.1.0" has unmet peer dependency "@glimmer/component@^1.1.2".
warning " > @glint/environment-ember-template-imports@1.1.0" has unmet peer dependency "ember-template-imports@^3.0.0".
warning Resolution field "unset-value@2.0.1" is incompatible with requested version "unset-value@^1.0.0"
warning Pattern ["wrap-ansi@^7.0.0"] is trying to unpack in the same destination "/home/discourse/.cache/yarn/v6/npm-wrap-ansi-cjs-7.0.0-67e145cff510a6a6984bdf1152911d69d2eb9e43-integrity/node_modules/wrap-ansi-cjs" as pattern ["wrap-ansi-cjs@npm:wrap-ansi@^7.0.0"]. This could result in non-deterministic behavior, skipping.
warning " > discourse-markdown-it@1.0.0" has unmet peer dependency "xss@*".
warning "workspace-aggregator-4a0ff1cd-82ba-483f-8207-6b2c0f0b31fb > discourse > @ember/legacy-built-in-components@0.5.0" has incorrect peer dependency "ember-source@>= 4.8".
warning "workspace-aggregator-4a0ff1cd-82ba-483f-8207-6b2c0f0b31fb > discourse > @uppy/aws-s3@3.0.6" has incorrect peer dependency "@uppy/core@^3.1.2".
warning "workspace-aggregator-4a0ff1cd-82ba-483f-8207-6b2c0f0b31fb > discourse > @uppy/aws-s3-multipart@3.1.3" has incorrect peer dependency "@uppy/core@^3.1.2".
warning "workspace-aggregator-4a0ff1cd-82ba-483f-8207-6b2c0f0b31fb > discourse > @uppy/xhr-upload@3.1.1" has incorrect peer dependency "@uppy/core@^3.1.2".
warning "workspace-aggregator-4a0ff1cd-82ba-483f-8207-6b2c0f0b31fb > discourse-plugins > ember-this-fallback@0.4.0" has unmet peer dependency "ember-source@^3.28.11 || ^4.0.0".
warning "workspace-aggregator-4a0ff1cd-82ba-483f-8207-6b2c0f0b31fb > discourse > @uppy/aws-s3 > @uppy/xhr-upload@3.3.0" has incorrect peer dependency "@uppy/core@^3.2.1".
I, [2023-12-11T06:39:22.730680 #1]  INFO -- : yarn install v1.22.19
[1/5] Validating package.json...
[2/5] Resolving packages...
[3/5] Fetching packages...
[4/5] Linking dependencies...
[5/5] Building fresh packages...
$ yarn --cwd app/assets/javascripts $(node -e 'const argv = JSON.parse(process.env.npm_config_argv).original; const passthrough = [`--frozen-lockfile`, `-s`].filter(arg => argv.includes(arg)); console.log(passthrough.join(` `));')
yarn install v1.22.19
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
$ ./run-patch-package
patch-package 8.0.0
Applying patches...
babel-plugin-debug-macros@0.3.4 āœ”
content-tag@1.1.2 āœ”
ember-cli@5.0.0 āœ”
ember-this-fallback@0.4.0 (1 deprecation-name) āœ”
ember-this-fallback@0.4.0 (2 themes) āœ”
virtual-dom@2.1.1 āœ”
Done in 64.07s.
Done in 115.18s.
yarn cache v1.22.19
success Cleared cache.
Done in 7.53s.

I, [2023-12-11T06:39:22.731495 #1]  INFO -- : > cd /var/www/discourse && su discourse -c 'bundle config --local deployment true'
I, [2023-12-11T06:39:23.103216 #1]  INFO -- : 
I, [2023-12-11T06:39:23.103337 #1]  INFO -- : > cd /var/www/discourse && su discourse -c 'bundle config --local without "development test"'
I, [2023-12-11T06:39:23.331037 #1]  INFO -- : You are replacing the current local value of without, which is currently "test:development"

I, [2023-12-11T06:39:23.331186 #1]  INFO -- : > cd /var/www/discourse && su discourse -c 'bundle install --retry 3 --jobs 4'
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

current directory:
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rbtrace-0.5.0/ext
/usr/local/bin/ruby extconf.rb
  -- tar zxvfo msgpack-1.1.0.tar.gz
  -- env CFLAGS=nil LDFLAGS=nil CC=nil
-- ./configure --disable-dependency-tracking --disable-shared --with-pic
--prefix=/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rbtrace-0.5.0/ext/dst/
--libdir=/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rbtrace-0.5.0/ext/dst/lib
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers.  Check the mkmf.log file for more details.  You may
need configuration options.

Provided configuration options:
	--with-opt-dir
	--without-opt-dir
	--with-opt-include
	--without-opt-include=${opt-dir}/include
	--with-opt-lib
	--without-opt-lib=${opt-dir}/lib
	--with-make-prog
	--without-make-prog
	--srcdir=.
	--curdir
	--ruby=/usr/local/bin/$(RUBY_BASE_NAME)
extconf.rb:6:in `sys': ./configure --disable-dependency-tracking
--disable-shared --with-pic
--prefix=/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rbtrace-0.5.0/ext/dst/
--libdir=/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rbtrace-0.5.0/ext/dst/lib
failed, please report to https://github.com/tmm1/rbtrace/issues (RuntimeError)
	from extconf.rb:42:in `block (2 levels) in <main>'
	from extconf.rb:34:in `chdir'
	from extconf.rb:34:in `block in <main>'
	from extconf.rb:30:in `chdir'
	from extconf.rb:30:in `<main>'

To see why this extension failed to compile, please check the mkmf.log which can
be found here:

/var/www/discourse/vendor/bundle/ruby/3.2.0/extensions/x86_64-linux/3.2.0/rbtrace-0.5.0/mkmf.log

extconf failed, exit code 1

Gem files will remain installed in
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rbtrace-0.5.0 for inspection.
Results logged to
/var/www/discourse/vendor/bundle/ruby/3.2.0/extensions/x86_64-linux/3.2.0/rbtrace-0.5.0/gem_make.out

  /usr/local/lib/ruby/site_ruby/3.2.0/rubygems/ext/builder.rb:120:in `run'
/usr/local/lib/ruby/site_ruby/3.2.0/rubygems/ext/ext_conf_builder.rb:28:in
`build'
/usr/local/lib/ruby/site_ruby/3.2.0/rubygems/ext/builder.rb:188:in
`build_extension'
/usr/local/lib/ruby/site_ruby/3.2.0/rubygems/ext/builder.rb:222:in `block in
build_extensions'
  /usr/local/lib/ruby/site_ruby/3.2.0/rubygems/ext/builder.rb:219:in `each'
/usr/local/lib/ruby/site_ruby/3.2.0/rubygems/ext/builder.rb:219:in
`build_extensions'
/usr/local/lib/ruby/site_ruby/3.2.0/rubygems/installer.rb:844:in
`build_extensions'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/rubygems_gem_installer.rb:72:in
`build_extensions'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/rubygems_gem_installer.rb:28:in
`install'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/source/rubygems.rb:198:in
`install'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/installer/gem_installer.rb:54:in
`install'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/installer/gem_installer.rb:16:in
`install_from_spec'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/installer/parallel_installer.rb:156:in
`do_install'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/installer/parallel_installer.rb:147:in
`block in worker_pool'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/worker.rb:62:in
`apply_func'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/worker.rb:57:in
`block in process_queue'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/worker.rb:54:in
`loop'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/worker.rb:54:in
`process_queue'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/worker.rb:90:in
`block (2 levels) in create_threads'

An error occurred while installing rbtrace (0.5.0), and Bundler cannot continue.

In Gemfile:
  rbtrace
I, [2023-12-11T06:40:09.131584 #1]  INFO -- : Fetching gem metadata from https://rubygems.org/.........
Fetching rake 13.1.0
Installing rake 13.1.0
Using concurrent-ruby 1.2.2
Using minitest 5.20.0
Using builder 3.2.4
Using erubi 1.12.0
Fetching racc 1.7.3
Using crass 1.0.6
Using rack 2.2.8
Using mini_mime 1.1.5
Fetching date 3.3.4
Fetching public_suffix 5.0.4
Fetching timeout 0.4.1
Installing timeout 0.4.1
Installing public_suffix 5.0.4
Installing date 3.3.4 with native extensions
Installing racc 1.7.3 with native extensions
Fetching aws-eventstream 1.3.0
Installing aws-eventstream 1.3.0
Using aws-partitions 1.583.0
Using jmespath 1.6.2
Fetching base64 0.2.0
Installing base64 0.2.0
Fetching bigdecimal 3.1.4
Using msgpack 1.7.2
Using bundler 2.4.13
Using byebug 11.1.3
Using cbor 0.5.9.6
Fetching cgi 0.4.1
Installing bigdecimal 3.1.4 with native extensions
Installing cgi 0.4.1 with native extensions
Using chunky_png 1.4.0
Using coderay 1.1.3
Using colored2 4.0.0
Using connection_pool 2.4.1
Using openssl 3.2.0
Using cppjieba_rb 0.4.2
Using diffy 3.4.2
Using digest 3.1.1
Using discourse-fonts 0.0.9
Using email_reply_trimmer 0.1.13
Fetching excon 0.105.0
Installing excon 0.105.0
Using execjs 2.9.1
Using exifr 1.4.0
Using faraday-net_http 3.0.2
Using ruby2_keywords 0.0.5
Using fast_blank 1.0.1
Using fast_xs 0.8.0
Using fastimage 2.2.7
Using ffi 1.16.3
Using fspath 3.1.2
Using gc_tracer 1.5.1
Fetching google-protobuf 3.25.1 (x86_64-linux)
Installing google-protobuf 3.25.1 (x86_64-linux)
Using guess_html_encoding 0.0.11
Using hana 1.3.7
Using hashie 5.0.0
Using highline 2.1.0
Using htmlentities 4.3.4
Using http_accept_language 2.1.1
Using image_size 3.3.0
Using in_threads 1.6.0
Using progress 3.6.0
Fetching json 2.7.1
Installing json 2.7.1 with native extensions
Fetching regexp_parser 2.8.3
Installing regexp_parser 2.8.3
Fetching unf_ext 0.0.9.1
Installing unf_ext 0.0.9.1 with native extensions
Using jwt 2.7.1
Using kgio 2.11.4
Using libv8-node 18.16.0.0 (x86_64-linux)
Using method_source 1.0.0
Using thor 1.3.0
Using zeitwerk 2.6.12
Using logstash-event 1.2.02
Fetching logster 2.13.1
Installing logster 2.13.1
Using lru_redux 1.1.0
Using lz4-ruby 0.3.3
Using maxminddb 0.1.22
Using memory_profiler 1.0.1
Using redis 4.8.1
Using mini_sql 1.5.0
Using multi_json 1.15.0
Using multi_xml 0.6.0
Using mustache 1.1.1
Fetching uri 0.13.0
Installing uri 0.13.0
Fetching nio4r 2.7.0
Installing nio4r 2.7.0 with native extensions
Using version_gem 1.1.3
Using optimist 3.1.0
Using pg 1.5.4
Using raindrops 0.20.1
Using rchardet 1.8.0
Using rexml 3.2.6
Using rinku 2.0.6
Using rotp 6.3.0
Using rqrcode_core 1.2.0
Using rubyzip 2.3.2
Using sshkey 3.0.0
Using stackprof 0.2.25
Using i18n 1.14.1
Using tzinfo 2.0.6
Using rack-test 2.1.0
Using request_store 1.5.1
Using message_bus 4.3.8
Fetching rack-mini-profiler 3.3.0
Installing rack-mini-profiler 3.3.0
Using rack-protection 3.1.0
Using sprockets 3.7.2 from https://github.com/rails/sprockets (at 3.x@f4d3dae)
Fetching net-protocol 0.2.2
Installing net-protocol 0.2.2
Fetching addressable 2.8.6
Installing addressable 2.8.6
Using aws-sigv4 1.5.0
Fetching bootsnap 1.17.0
Installing bootsnap 1.17.0 with native extensions
Using openssl-signature_algorithm 1.3.0
Using uglifier 4.2.0
Fetching faraday 2.7.12
Installing faraday 2.7.12
Using mini_suffix 0.3.3
Fetching nokogiri 1.15.5 (x86_64-linux)
Installing nokogiri 1.15.5 (x86_64-linux)
Using image_optim 0.31.3
Using omniauth 1.9.2
Fetching sass-embedded 1.69.5 (x86_64-linux-gnu)
Installing sass-embedded 1.69.5 (x86_64-linux-gnu)
Using unf 0.1.4
Fetching web-push 3.0.1
Installing web-push 3.0.1
Using logstash-logger 0.26.1
Using mini_racer 0.8.0
Using pry 0.14.2
Using sidekiq 6.5.12
Using redis-namespace 1.11.0
Fetching net-http 0.4.0
Installing net-http 0.4.0
Using oauth-tty 1.0.5
Using snaky_hash 2.0.1
Fetching rbtrace 0.5.0
Installing rbtrace 0.5.0 with native extensions
Using activesupport 7.0.7
Using rqrcode 2.2.0
Using rss 0.3.0
Using tzinfo-data 1.2023.3
Using unicorn 6.1.0
Using net-pop 0.1.2
Using net-smtp 0.4.0
Using css_parser 1.16.0
Using aws-sdk-core 3.130.2
Using cose 1.3.0
Using faraday-retry 2.2.0
Using oauth2 1.4.11
Fetching loofah 2.22.0
Installing loofah 2.22.0
Using ruby-readability 0.7.0
Using sanitize 6.1.0
Fetching sassc-embedded 1.68.6
Installing sassc-embedded 1.68.6
Using simpleidn 0.2.1
Using rtlcss 0.2.1
Using pry-byebug 3.10.1
Using pry-rails 0.3.9
Using mini_scheduler 0.16.0
Using oauth 1.1.0
Using rails-dom-testing 2.2.0
Using globalid 1.2.1
Using activemodel 7.0.7
Using aws-sdk-kms 1.56.0
Using aws-sdk-sns 1.53.0
Using omniauth-oauth2 1.7.3
Using rails-html-sanitizer 1.6.0
Fetching json_schemer 2.1.1
Installing json_schemer 2.1.1
Using omniauth-oauth 1.2.0
Using actionview 7.0.7
Using activejob 7.0.7
Using active_model_serializers 0.8.4
Using activerecord 7.0.7
Using aws-sdk-s3 1.114.0
Using omniauth-facebook 9.0.0
Using omniauth-github 1.4.0
Using omniauth-google-oauth2 0.8.2
Using omniauth-twitter 1.4.0
Using actionpack 7.0.7
Using actionview_precompiler 0.3.0
Using discourse-seed-fu 2.3.12
Using railties 7.0.7
Using sprockets-rails 3.4.2
Using lograge 0.14.0
Using rails_failover 2.0.1
Using rails_multisite 5.0.0
Using puma 6.4.0
Fetching net-imap 0.4.7
Installing net-imap 0.4.7
Using mail 2.8.1
Using actionmailer 7.0.7
Fetching oj 3.16.2
Installing oj 3.16.2 with native extensions

I, [2023-12-11T06:40:09.132711 #1]  INFO -- : Terminating async processes
I, [2023-12-11T06:40:09.133069 #1]  INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main pid: 40
2023-12-11 06:40:09.133 UTC [40] LOG:  received fast shutdown request
I, [2023-12-11T06:40:09.133831 #1]  INFO -- : Sending TERM to exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf pid: 109
109:signal-handler (1702276809) Received SIGTERM scheduling shutdown...
2023-12-11 06:40:09.138 UTC [40] LOG:  aborting any active transactions
2023-12-11 06:40:09.144 UTC [40] LOG:  background worker "logical replication launcher" (PID 49) exited with exit code 1
2023-12-11 06:40:09.146 UTC [44] LOG:  shutting down
2023-12-11 06:40:09.168 UTC [40] LOG:  database system is shut down
109:M 11 Dec 2023 06:40:09.173 # User requested shutdown...
109:M 11 Dec 2023 06:40:09.173 * Saving the final RDB snapshot before exiting.
109:M 11 Dec 2023 06:40:10.723 * DB saved on disk
109:M 11 Dec 2023 06:40:10.723 # Redis is now ready to exit, bye bye...


FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && su discourse -c 'bundle install --retry 3 --jobs 4' failed with return #<Process::Status: pid 585 exit 5>
Location of failure: /usr/local/lib/ruby/gems/3.2.0/gems/pups-1.2.1/lib/pups/exec_command.rb:132:in `spawn'
exec failed with the params {"cd"=>"$home", "hook"=>"bundle_exec", "cmd"=>["su discourse -c 'bundle config --local deployment true'", "su discourse -c 'bundle config --local without \"development test\"'", "su discourse -c 'bundle install --retry 3 --jobs 4'"]}
bootstrap failed with exit code 5
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.
9d6071df716185b1c31a8703028a03d6fb0198216704c41506a69b09c67d4519
root@greyarro:/var/discourse# 

I ran ./discourse-doctor which looked like it did lots of scary things that I didnā€™t understand but unfortunately it made no difference and it ended with what appears to be the same message:

FAILED

--------------------

Pups::ExecError: cd /var/www/discourse && su discourse -c 'bundle install --retry 3 --jobs 4' failed with return #<Process::Status: pid 585 exit 5>

Location of failure: /usr/local/lib/ruby/gems/3.2.0/gems/pups-1.2.1/lib/pups/exec_command.rb:132:in `spawn'

exec failed with the params {"cd"=>"$home", "hook"=>"bundle_exec", "cmd"=>["su discourse -c 'bundle config --local deployment true'", "su discourse -c 'bundle config --local without \"development test\"'", "su discourse -c 'bundle install --retry 3 --jobs 4'"]}

bootstrap failed with exit code 5

** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.

./discourse-doctor may help diagnose the problem.

9f72d4c67a46bda65d95233149ac8f661bdae02e70ae9a265666920d71de14e2

==================== END REBUILD LOG ====================

Failed to rebuild app.

Can anybody shed any light on the issue, or help me get us back online??

Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

current directory:
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rbtrace-0.5.0/ext

Is this a standard install?

1 Like

As far as I can recall @RGJ - but it was six years or so ago that I first created it, but I donā€™t recall doing anything out of the ordinary back then.

I always update to latest around once per week too, using the command line, this is the first time itā€™s failed to update in all those years.

Iā€™m not at all familiar with ruby, what is that message suggesting to you? :thinking:

The message suggests that something is outdated or non-standard, but it is pulling the latest image, so that doesnā€™t make sense.

1 Like

Could this be an OS out of date issue? :thinking:

Currently running on:

NAME="Ubuntu"

VERSION="18.04.6 LTS (Bionic Beaver)"

ID=ubuntu

ID_LIKE=debian

PRETTY_NAME="Ubuntu 18.04.6 LTS"

VERSION_ID="18.04"

HOME_URL="https://www.ubuntu.com/"

SUPPORT_URL="https://help.ubuntu.com/"

BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"

PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"

VERSION_CODENAME=bionic

UBUNTU_CODENAME=bionic

Iā€™m going to have to assume the worst now, that my Discourse is dead.

Iā€™ve got a copy of my latest app.yml and Iā€™ve just downloaded last nights database backup from S3.

I think Iā€™ll do-release-upgrade my Ubuntu from 18.04.6 LTS to latest, the I guess Iā€™ll do a fresh installing following the guide at discourse/docs/INSTALL-cloud.md at main Ā· discourse/discourse Ā· GitHub

Once installed, can I then upload my 1.5GB database backup via the admin panel? :thinking:

Iā€™ve been using S3 for uploads for years so Iā€™m hoping the uploaded user content is all still accessible from there.

Are there any gotchas? Is there anything thatā€™s not going to be in the database? Any settings? Any files or other content?

:flushed:

Iā€™ve taken a (Digital Ocean) snapshot of the server in itā€™s current state, release upgrade now underway.

Out of interest, I was upgrading from https://github.com/discourse/discourse/compare/aadc104817bb0306856b4718f5c50fb481e77c16ā€¦4949d85c15be99abb3515e659ef674d58cc2c4c4 and youā€™ll see I was only 128x commits / 11x days behind latest.

Is there a way I can git pull that previous version back down, and rebuild using that? Just to get our community up and running again? :thinking:

Did you try ./launcher start app to try and get the pre-rebuild one back up?

2 Likes

Alas, in my blind panic at it all failing this morning, I removed all the containers which werenā€™t running :pensive:

I think as long as you have your app.yml info and your recent backup you should be back up in no time on a new droplet. :+1:

3 Likes

Whoa!!

I think I may have gifted myself a lifeline :scream:

While waiting for the distro to upgrade, it would appear I have rather cleverly enabled Digital Ocean backups :scream:

Time to see if theyā€™re any good :grimacing:

Ok, Plan B thenā€¦

Restore last weeks Digital Ocean snapshot backup from 2023-12-04

Upload my database backup from last night (2023-12-11 @ 2am)

I think that might get us roughly back to were we were at 6am this morning when it all fell apart.

Iā€™ve not done a database restore before.

Assuming the D.O. backup/restore works and gets me back to Monday last week, can I just upload the 1.5GB backup.tar.gz file directly via admin backup menus? do I need to extract it first or anything?

Should I put the site in read-only mode or anything else, while doing so?

Server up! (from last Mondayā€™s snapshot)

Turns out you very clever people at Discourse built a ā€œRestoreā€ button directly in to the admin menu so I donā€™t even need to upload the backup :bowing_man:

:crossed_fingers:

:sob:

Possibly because I put the site in read-only mode before trying the restore.

Turning off read only mode and trying to restore againā€¦

No, that didnā€™t work either.

Error log:

Iā€™ve just stopped the app again.

If I look at my sidekiq it has 12,000+ jobs in it.

It looks like itā€™s trying to re-send out scheduled email announcements from last week again :man_facepalming:

Is there a command line action I can take to kill all those off? I guess these queued items are stopping the restore.

What I would do is spin up a new droplet and follow Move a Discourse site to another VPS with rsync.
But I guess you donā€™t have your /var/discourse anymore unless you made another backup before you restored the one you found?

So spin up a new droplet, copy your app.yml and restore yesterdayā€™s backup from S3. This is what to do.

Also, whatā€™s your docker version? I had a rebuild fail with an error that looks like this (Iā€™ll track down that topic when I get to my laptop).

1 Like

I have a full server snapshot from 7 days ago, which Iā€™ve restored, and my Discourse is now running again - but with a 7x days out of date database.

Iā€™m now trying to restore the database from last nights backup which is on S3.

1 Like

While that falls, Spin up a new droplet. (which is what you want even if it will restore)

1 Like

Thanks Jay :slight_smile:

I manually killed off all the sidekiq jobs and it looks like the database restore is now running:

:crossed_fingers:

If this also fails, Iā€™ll pull the plug and go your rsync route, thank you :blush:

1 Like

You want a new os. I donā€™t trust upgrades. You want to use the newer hardware youā€™ll get off your create a new droplet.

Canā€™t say Iā€™m a fan either :slight_smile:

The weekly Digital Ocean backup from 7x days ago, which I restored, has also put the Ubuntu 18.04 operating system back, so Iā€™m now pre-upgrade once again.

Database restore looks like it was successful.

Iā€™m just taking another fresh snapshot now - just in case.

If all this works and gets us back online, I can then look at doing a rsync move to the latest OS - but at my leisure rather than in a rushed panic :see_no_evil:

1 Like