Discourse nach fehlgeschlagenem Upgrade offline "bootstrap fehlgeschlagen mit Exit-Code 5"

Installation
1

Hi everyone,

My Discourse is currently offline :cry: :pensive:

It failed after trying to upgrade this morning from/to v3.2.0.beta3 +468:

On the command line, the error message is:

And the full console log:

root@greyarro:~# cd /var/discourse/
root@greyarro:/var/discourse# git pull
Already up to date.
root@greyarro:/var/discourse# ./launcher rebuild app
x86_64 arch detected.
WARNING: containers/app.yml file is world-readable. You can secure this file by running: chmod o-rwx containers/app.yml
Ensuring launcher is up to date
Fetching origin
Launcher is up-to-date
2.0.20231023-1945: Pulling from discourse/base
Digest: sha256:2b0eb484d20888cc2daadb690dcfa73522105650c1420212e99345a36a424d77
Status: Image is up to date for discourse/base:2.0.20231023-1945
/usr/local/lib/ruby/gems/3.2.0/gems/pups-1.2.1/lib/pups.rb
/usr/local/bin/pups --stdin
I, [2023-12-11T06:36:42.453494 #1]  INFO -- : Reading from stdin
I, [2023-12-11T06:36:42.461560 #1]  INFO -- : > locale-gen $LANG && update-locale
I, [2023-12-11T06:36:42.523575 #1]  INFO -- : Generating locales (this might take a while)...
Generation complete.

I, [2023-12-11T06:36:42.524082 #1]  INFO -- : > mkdir -p /shared/postgres_run
I, [2023-12-11T06:36:42.529018 #1]  INFO -- : 
I, [2023-12-11T06:36:42.529478 #1]  INFO -- : > chown postgres:postgres /shared/postgres_run
I, [2023-12-11T06:36:42.534085 #1]  INFO -- : 
I, [2023-12-11T06:36:42.534610 #1]  INFO -- : > chmod 775 /shared/postgres_run
I, [2023-12-11T06:36:42.538665 #1]  INFO -- : 
I, [2023-12-11T06:36:42.539119 #1]  INFO -- : > rm -fr /var/run/postgresql
I, [2023-12-11T06:36:42.542429 #1]  INFO -- : 
I, [2023-12-11T06:36:42.542918 #1]  INFO -- : > ln -s /shared/postgres_run /var/run/postgresql
I, [2023-12-11T06:36:42.545917 #1]  INFO -- : 
I, [2023-12-11T06:36:42.546272 #1]  INFO -- : > socat /dev/null UNIX-CONNECT:/shared/postgres_run/.s.PGSQL.5432 || exit 0 && echo postgres already running stop container ; exit 1
2023/12/11 06:36:42 socat[17] E connect(6, AF=1 "/shared/postgres_run/.s.PGSQL.5432", 36): No such file or directory
I, [2023-12-11T06:36:42.557289 #1]  INFO -- : 
I, [2023-12-11T06:36:42.557926 #1]  INFO -- : > rm -fr /shared/postgres_run/.s*
I, [2023-12-11T06:36:42.562906 #1]  INFO -- : 
I, [2023-12-11T06:36:42.563405 #1]  INFO -- : > rm -fr /shared/postgres_run/*.pid
I, [2023-12-11T06:36:42.568125 #1]  INFO -- : 
I, [2023-12-11T06:36:42.568537 #1]  INFO -- : > mkdir -p /shared/postgres_run/13-main.pg_stat_tmp
I, [2023-12-11T06:36:42.571897 #1]  INFO -- : 
I, [2023-12-11T06:36:42.572328 #1]  INFO -- : > chown postgres:postgres /shared/postgres_run/13-main.pg_stat_tmp
I, [2023-12-11T06:36:42.575429 #1]  INFO -- : 
I, [2023-12-11T06:36:42.583069 #1]  INFO -- : File > /etc/service/postgres/run  chmod: +x  chown: 
I, [2023-12-11T06:36:42.591097 #1]  INFO -- : File > /etc/service/postgres/log/run  chmod: +x  chown: 
I, [2023-12-11T06:36:42.599557 #1]  INFO -- : File > /etc/runit/3.d/99-postgres  chmod: +x  chown: 
I, [2023-12-11T06:36:42.607693 #1]  INFO -- : File > /root/upgrade_postgres  chmod: +x  chown: 
I, [2023-12-11T06:36:42.608293 #1]  INFO -- : > chown -R root /var/lib/postgresql/13/main
I, [2023-12-11T06:36:43.061308 #1]  INFO -- : 
I, [2023-12-11T06:36:43.061769 #1]  INFO -- : > [ ! -e /shared/postgres_data ] && install -d -m 0755 -o postgres -g postgres /shared/postgres_data && sudo -E -u postgres /usr/lib/postgresql/13/bin/initdb -D /shared/postgres_data || exit 0
I, [2023-12-11T06:36:43.066134 #1]  INFO -- : 
I, [2023-12-11T06:36:43.066571 #1]  INFO -- : > chown -R postgres:postgres /shared/postgres_data
I, [2023-12-11T06:36:43.105049 #1]  INFO -- : 
I, [2023-12-11T06:36:43.105891 #1]  INFO -- : > chown -R postgres:postgres /var/run/postgresql
I, [2023-12-11T06:36:43.111420 #1]  INFO -- : 
I, [2023-12-11T06:36:43.112072 #1]  INFO -- : > /root/upgrade_postgres
I, [2023-12-11T06:36:43.122177 #1]  INFO -- : 
I, [2023-12-11T06:36:43.122713 #1]  INFO -- : > rm /root/upgrade_postgres
I, [2023-12-11T06:36:43.126702 #1]  INFO -- : 
I, [2023-12-11T06:36:43.127981 #1]  INFO -- : Replacing data_directory = '/var/lib/postgresql/13/main' with data_directory = '/shared/postgres_data' in /etc/postgresql/13/main/postgresql.conf
I, [2023-12-11T06:36:43.128927 #1]  INFO -- : Replacing (?-mix:#?listen_addresses *=.*) with listen_addresses = '*' in /etc/postgresql/13/main/postgresql.conf
I, [2023-12-11T06:36:43.129583 #1]  INFO -- : Replacing (?-mix:#?synchronous_commit *=.*) with synchronous_commit = $db_synchronous_commit in /etc/postgresql/13/main/postgresql.conf
I, [2023-12-11T06:36:43.130214 #1]  INFO -- : Replacing (?-mix:#?shared_buffers *=.*) with shared_buffers = $db_shared_buffers in /etc/postgresql/13/main/postgresql.conf
I, [2023-12-11T06:36:43.130899 #1]  INFO -- : Replacing (?-mix:#?work_mem *=.*) with work_mem = $db_work_mem in /etc/postgresql/13/main/postgresql.conf
I, [2023-12-11T06:36:43.131457 #1]  INFO -- : Replacing (?-mix:#?default_text_search_config *=.*) with default_text_search_config = '$db_default_text_search_config' in /etc/postgresql/13/main/postgresql.conf
I, [2023-12-11T06:36:43.132096 #1]  INFO -- : > install -d -m 0755 -o postgres -g postgres /shared/postgres_backup
I, [2023-12-11T06:36:43.137173 #1]  INFO -- : 
I, [2023-12-11T06:36:43.137747 #1]  INFO -- : Replacing (?-mix:#?checkpoint_segments *=.*) with checkpoint_segments = $db_checkpoint_segments in /etc/postgresql/13/main/postgresql.conf
I, [2023-12-11T06:36:43.138339 #1]  INFO -- : Replacing (?-mix:#?logging_collector *=.*) with logging_collector = $db_logging_collector in /etc/postgresql/13/main/postgresql.conf
I, [2023-12-11T06:36:43.139065 #1]  INFO -- : Replacing (?-mix:#?log_min_duration_statement *=.*) with log_min_duration_statement = $db_log_min_duration_statement in /etc/postgresql/13/main/postgresql.conf
I, [2023-12-11T06:36:43.140177 #1]  INFO -- : Replacing (?-mix:^#local +replication +postgres +peer$) with local replication postgres  peer in /etc/postgresql/13/main/pg_hba.conf
I, [2023-12-11T06:36:43.140923 #1]  INFO -- : Replacing (?-mix:^host.*all.*all.*127.*$) with host all all 0.0.0.0/0 md5 in /etc/postgresql/13/main/pg_hba.conf
I, [2023-12-11T06:36:43.141403 #1]  INFO -- : Replacing (?-mix:^host.*all.*all.*::1\/128.*$) with host all all ::/0 md5 in /etc/postgresql/13/main/pg_hba.conf
I, [2023-12-11T06:36:43.141961 #1]  INFO -- : > HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main
I, [2023-12-11T06:36:43.144661 #1]  INFO -- : > sleep 5
2023-12-11 06:36:43.213 UTC [40] LOG:  starting PostgreSQL 13.12 (Debian 13.12-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
2023-12-11 06:36:43.213 UTC [40] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2023-12-11 06:36:43.213 UTC [40] LOG:  listening on IPv6 address "::", port 5432
2023-12-11 06:36:43.217 UTC [40] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2023-12-11 06:36:43.223 UTC [43] LOG:  database system was shut down at 2023-12-11 06:33:31 UTC
2023-12-11 06:36:43.236 UTC [40] LOG:  database system is ready to accept connections
I, [2023-12-11T06:36:48.151212 #1]  INFO -- : 
I, [2023-12-11T06:36:48.151469 #1]  INFO -- : > su postgres -c 'createdb discourse' || true
2023-12-11 06:36:48.241 UTC [53] postgres@postgres ERROR:  database "discourse" already exists
2023-12-11 06:36:48.241 UTC [53] postgres@postgres STATEMENT:  CREATE DATABASE discourse;
createdb: error: database creation failed: ERROR:  database "discourse" already exists
I, [2023-12-11T06:36:48.244084 #1]  INFO -- : 
I, [2023-12-11T06:36:48.244295 #1]  INFO -- : > su postgres -c 'psql discourse -c "create user discourse;"' || true
2023-12-11 06:36:48.310 UTC [57] postgres@discourse ERROR:  role "discourse" already exists
2023-12-11 06:36:48.310 UTC [57] postgres@discourse STATEMENT:  create user discourse;
ERROR:  role "discourse" already exists
I, [2023-12-11T06:36:48.313540 #1]  INFO -- : 
I, [2023-12-11T06:36:48.313932 #1]  INFO -- : > su postgres -c 'psql discourse -c "grant all privileges on database discourse to discourse;"' || true
I, [2023-12-11T06:36:48.372120 #1]  INFO -- : GRANT

I, [2023-12-11T06:36:48.372374 #1]  INFO -- : > su postgres -c 'psql discourse -c "alter schema public owner to discourse;"'
I, [2023-12-11T06:36:48.428450 #1]  INFO -- : ALTER SCHEMA

I, [2023-12-11T06:36:48.428697 #1]  INFO -- : > su postgres -c 'psql template1 -c "create extension if not exists hstore;"'
NOTICE:  extension "hstore" already exists, skipping
I, [2023-12-11T06:36:48.509937 #1]  INFO -- : CREATE EXTENSION

I, [2023-12-11T06:36:48.510198 #1]  INFO -- : > su postgres -c 'psql template1 -c "create extension if not exists pg_trgm;"'
NOTICE:  extension "pg_trgm" already exists, skipping
I, [2023-12-11T06:36:48.575901 #1]  INFO -- : CREATE EXTENSION

I, [2023-12-11T06:36:48.576560 #1]  INFO -- : > su postgres -c 'psql template1 -c "create extension if not exists vector;"'
NOTICE:  extension "vector" already exists, skipping
I, [2023-12-11T06:36:48.634184 #1]  INFO -- : CREATE EXTENSION

I, [2023-12-11T06:36:48.634407 #1]  INFO -- : > su postgres -c 'psql discourse -c "create extension if not exists hstore;"'
NOTICE:  extension "hstore" already exists, skipping
I, [2023-12-11T06:36:48.691197 #1]  INFO -- : CREATE EXTENSION

I, [2023-12-11T06:36:48.691402 #1]  INFO -- : > su postgres -c 'psql discourse -c "create extension if not exists pg_trgm;"'
NOTICE:  extension "pg_trgm" already exists, skipping
I, [2023-12-11T06:36:48.747583 #1]  INFO -- : CREATE EXTENSION

I, [2023-12-11T06:36:48.747801 #1]  INFO -- : > su postgres -c 'psql discourse -c "create extension if not exists vector;"'
NOTICE:  extension "vector" already exists, skipping
I, [2023-12-11T06:36:48.805422 #1]  INFO -- : CREATE EXTENSION

I, [2023-12-11T06:36:48.805817 #1]  INFO -- : > sudo -u postgres psql discourse
I, [2023-12-11T06:36:48.808688 #1]  INFO -- : update pg_database set encoding = pg_char_to_encoding('UTF8') where datname = 'discourse' AND encoding = pg_char_to_encoding('SQL_ASCII');

I, [2023-12-11T06:36:48.891679 #1]  INFO -- : File > /var/lib/postgresql/take-database-backup  chmod: +x  chown: postgres:postgres
I, [2023-12-11T06:36:48.896830 #1]  INFO -- : File > /var/spool/cron/crontabs/postgres  chmod:   chown: 
I, [2023-12-11T06:36:48.896998 #1]  INFO -- : > echo postgres installed!
I, [2023-12-11T06:36:48.899962 #1]  INFO -- : postgres installed!

I, [2023-12-11T06:36:48.907627 #1]  INFO -- : File > /etc/service/redis/run  chmod: +x  chown: 
I, [2023-12-11T06:36:48.914411 #1]  INFO -- : File > /etc/service/redis/log/run  chmod: +x  chown: 
I, [2023-12-11T06:36:48.920982 #1]  INFO -- : File > /etc/runit/3.d/10-redis  chmod: +x  chown: 
I, [2023-12-11T06:36:48.922023 #1]  INFO -- : Replacing daemonize yes with  in /etc/redis/redis.conf
I, [2023-12-11T06:36:48.922829 #1]  INFO -- : Replacing (?-mix:^pidfile.*$) with  in /etc/redis/redis.conf
I, [2023-12-11T06:36:48.923923 #1]  INFO -- : > install -d -m 0755 -o redis -g redis /shared/redis_data
I, [2023-12-11T06:36:48.928742 #1]  INFO -- : 
I, [2023-12-11T06:36:48.929799 #1]  INFO -- : Replacing (?-mix:^logfile.*$) with logfile "" in /etc/redis/redis.conf
I, [2023-12-11T06:36:48.930627 #1]  INFO -- : Replacing (?-mix:^bind .*$) with  in /etc/redis/redis.conf
I, [2023-12-11T06:36:48.931845 #1]  INFO -- : Replacing (?-mix:^dir .*$) with dir /shared/redis_data in /etc/redis/redis.conf
I, [2023-12-11T06:36:48.933205 #1]  INFO -- : Replacing (?-mix:^protected-mode yes) with protected-mode no in /etc/redis/redis.conf
I, [2023-12-11T06:36:48.934373 #1]  INFO -- : Replacing # io-threads 4 with io-threads $redis_io_threads in /etc/redis/redis.conf
I, [2023-12-11T06:36:48.935362 #1]  INFO -- : > echo redis installed
I, [2023-12-11T06:36:48.939050 #1]  INFO -- : redis installed

I, [2023-12-11T06:36:48.939481 #1]  INFO -- : > cat /etc/redis/redis.conf | grep logfile
I, [2023-12-11T06:36:48.947715 #1]  INFO -- : logfile ""

I, [2023-12-11T06:36:48.948224 #1]  INFO -- : > exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf
I, [2023-12-11T06:36:48.950934 #1]  INFO -- : > sleep 10
109:C 11 Dec 2023 06:36:48.962 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
109:C 11 Dec 2023 06:36:48.962 # Redis version=7.0.7, bits=64, commit=00000000, modified=0, pid=109, just started
109:C 11 Dec 2023 06:36:48.962 # Configuration loaded
109:M 11 Dec 2023 06:36:48.963 * monotonic clock: POSIX clock_gettime
109:M 11 Dec 2023 06:36:48.964 * Running mode=standalone, port=6379.
109:M 11 Dec 2023 06:36:48.964 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
109:M 11 Dec 2023 06:36:48.964 # Server initialized
109:M 11 Dec 2023 06:36:48.964 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
109:M 11 Dec 2023 06:36:48.965 * Loading RDB produced by version 7.0.7
109:M 11 Dec 2023 06:36:48.965 * RDB age 196 seconds
109:M 11 Dec 2023 06:36:48.965 * RDB memory usage when created 238.89 Mb
109:M 11 Dec 2023 06:36:49.836 * Done loading RDB, keys loaded: 78979, keys expired: 55.
109:M 11 Dec 2023 06:36:49.836 * DB loaded from disk: 0.871 seconds
109:M 11 Dec 2023 06:36:49.836 * Ready to accept connections
I, [2023-12-11T06:36:58.954574 #1]  INFO -- : 
I, [2023-12-11T06:36:58.955135 #1]  INFO -- : > thpoff echo "thpoff is installed!"
I, [2023-12-11T06:36:58.960865 #1]  INFO -- : thpoff is installed!

I, [2023-12-11T06:36:58.961207 #1]  INFO -- : > /usr/local/bin/ruby -e 'if ENV["DISCOURSE_SMTP_ADDRESS"] == "smtp.example.com"; puts "Aborting! Mail is not configured!"; exit 1; end'
I, [2023-12-11T06:36:59.059159 #1]  INFO -- : 
I, [2023-12-11T06:36:59.059736 #1]  INFO -- : > /usr/local/bin/ruby -e 'if ENV["DISCOURSE_HOSTNAME"] == "discourse.example.com"; puts "Aborting! Domain is not configured!"; exit 1; end'
I, [2023-12-11T06:36:59.148371 #1]  INFO -- : 
I, [2023-12-11T06:36:59.149030 #1]  INFO -- : > /usr/local/bin/ruby -e 'if (ENV["DISCOURSE_CDN_URL"] || "")[0..1] == "//"; puts "Aborting! CDN must have a protocol specified. Once fixed you should rebake your posts now to correct all posts."; exit 1; end'
I, [2023-12-11T06:36:59.236730 #1]  INFO -- : 
I, [2023-12-11T06:36:59.237387 #1]  INFO -- : > rm -f /etc/cron.d/anacron
I, [2023-12-11T06:36:59.242323 #1]  INFO -- : 
I, [2023-12-11T06:36:59.247436 #1]  INFO -- : File > /etc/cron.d/anacron  chmod:   chown: 
I, [2023-12-11T06:36:59.254458 #1]  INFO -- : File > /etc/runit/1.d/copy-env  chmod: +x  chown: 
I, [2023-12-11T06:36:59.261375 #1]  INFO -- : File > /etc/service/unicorn/run  chmod: +x  chown: 
I, [2023-12-11T06:36:59.268779 #1]  INFO -- : File > /etc/service/nginx/run  chmod: +x  chown: 
I, [2023-12-11T06:36:59.276427 #1]  INFO -- : File > /etc/runit/3.d/01-nginx  chmod: +x  chown: 
I, [2023-12-11T06:36:59.283178 #1]  INFO -- : File > /etc/runit/3.d/02-unicorn  chmod: +x  chown: 
I, [2023-12-11T06:36:59.283492 #1]  INFO -- : Replacing # postgres with sv start postgres || exit 1 in /etc/service/unicorn/run
I, [2023-12-11T06:36:59.284057 #1]  INFO -- : > exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf
I, [2023-12-11T06:36:59.287039 #1]  INFO -- : > cd /var/www/discourse && sudo -H -E -u discourse git reset --hard
137:C 11 Dec 2023 06:36:59.295 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
137:C 11 Dec 2023 06:36:59.295 # Redis version=7.0.7, bits=64, commit=00000000, modified=0, pid=137, just started
137:C 11 Dec 2023 06:36:59.295 # Configuration loaded
137:M 11 Dec 2023 06:36:59.295 * monotonic clock: POSIX clock_gettime
137:M 11 Dec 2023 06:36:59.295 # Warning: Could not create server TCP listening socket *:6379: bind: Address already in use
137:M 11 Dec 2023 06:36:59.295 # Failed listening on port 6379 (TCP), aborting.
Updating files: 100% (33858/33858), done.
I, [2023-12-11T06:37:03.209327 #1]  INFO -- : HEAD is now at 043b4a418 FIX: replaces data-tooltip usage by <DTooltip /> (#24062)

I, [2023-12-11T06:37:03.209496 #1]  INFO -- : > cd /var/www/discourse && sudo -H -E -u discourse git clean -f
I, [2023-12-11T06:37:03.416955 #1]  INFO -- : 
I, [2023-12-11T06:37:03.417150 #1]  INFO -- : > cd /var/www/discourse && sudo -H -E -u discourse bash -c '
  set -o errexit
  if [ $(git rev-parse --is-shallow-repository) == "true" ]; then
      git remote set-branches --add origin main
      git remote set-branches origin tests-passed
      git fetch --depth 1 origin tests-passed
  else
      git fetch --tags --prune-tags --prune --force origin
  fi
'
From https://github.com/discourse/discourse
 - [deleted]             (none)     -> origin/0-flake-key
 - [deleted]             (none)     -> origin/0-grant-badge-fix
 - [deleted]             (none)     -> origin/0-obscuring
 - [deleted]             (none)     -> origin/0-router-lint
 - [deleted]             (none)     -> origin/0-router-service
 - [deleted]             (none)     -> origin/a-smokes-test
 - [deleted]             (none)     -> origin/autoimport-cleanup
 - [deleted]             (none)     -> origin/category-bullet-only
 - [deleted]             (none)     -> origin/dependabot/bundler/syntax_tree-6.2.0
 - [deleted]             (none)     -> origin/dependabot/npm_and_yarn/app/assets/javascripts/uppy/aws-s3-3.4.0
 - [deleted]             (none)     -> origin/dependabot/npm_and_yarn/app/assets/javascripts/uppy/aws-s3-multipart-3.8.0
 - [deleted]             (none)     -> origin/dependabot/npm_and_yarn/app/assets/javascripts/uppy/core-3.6.0
 - [deleted]             (none)     -> origin/dependabot/npm_and_yarn/app/assets/javascripts/uppy/utils-5.5.2
 - [deleted]             (none)     -> origin/dependabot/npm_and_yarn/app/assets/javascripts/uppy/xhr-upload-3.4.2
 - [deleted]             (none)     -> origin/dev-upgrade-reject-reason-reviewable-modal
 - [deleted]             (none)     -> origin/dev/ability-to-collect-stats-without-public-exposing
 - [deleted]             (none)     -> origin/dev/minio-s3-system-specs
 - [deleted]             (none)     -> origin/dev/re-enable-minio-specs
 - [deleted]             (none)     -> origin/dev/switch-over-category-settings
 - [deleted]             (none)     -> origin/dev/switch-over-more-category-settings
 - [deleted]             (none)     -> origin/discovery-named-outlets
 - [deleted]             (none)     -> origin/feature/theme-settings-migration
 - [deleted]             (none)     -> origin/fix/account-for-empty-string-site-settings
 - [deleted]             (none)     -> origin/footnote-plugin
 - [deleted]             (none)     -> origin/metadata-update
 - [deleted]             (none)     -> origin/modal-ux
 - [deleted]             (none)     -> origin/neil-2
 - [deleted]             (none)     -> origin/perf_cache_async_search
 - [deleted]             (none)     -> origin/redo-post-focus-setting
 - [deleted]             (none)     -> origin/spoiler-plugin
 - [deleted]             (none)     -> origin/support-search-customizationz
   043b4a418..4949d85c1  main                  -> origin/main
 * [new branch]          0-class-attribute     -> origin/0-class-attribute
 * [new branch]          0-for-attributes      -> origin/0-for-attributes
 * [new branch]          0-i18n-case           -> origin/0-i18n-case
 * [new branch]          add-custom-emoji-sanitization-clean -> origin/add-custom-emoji-sanitization-clean
 * [new branch]          add-ds-store-to-gitignore -> origin/add-ds-store-to-gitignore
 + 64520a075...ff17131c9 add-new-chat-indicator-preference-only-mentions -> origin/add-new-chat-indicator-preference-only-mentions  (forced update)
 * [new branch]          api-topic-title-decorator -> origin/api-topic-title-decorator
 * [new branch]          array-custom-field-order -> origin/array-custom-field-order
 * [new branch]          boosted-searches      -> origin/boosted-searches
 * [new branch]          bulk-button-class     -> origin/bulk-button-class
 * [new branch]          category_definition   -> origin/category_definition
 * [new branch]          chat-github-onebox    -> origin/chat-github-onebox
 * [new branch]          chat-reaction-emoji-alignment -> origin/chat-reaction-emoji-alignment
 * [new branch]          chat/direct-messages  -> origin/chat/direct-messages
 * [new branch]          confirm-session-only-after-5-mins -> origin/confirm-session-only-after-5-mins
 * [new branch]          custom-field-store-cleanup -> origin/custom-field-store-cleanup
 * [new branch]          delete-backups-based-on-time-window -> origin/delete-backups-based-on-time-window
 * [new branch]          dependabot/github_actions/actions/labeler-5 -> origin/dependabot/github_actions/actions/labeler-5
 * [new branch]          dependabot/github_actions/actions/stale-9 -> origin/dependabot/github_actions/actions/stale-9
 * [new branch]          dependabot/npm_and_yarn/app/assets/javascripts/uppy/aws-s3-3.5.0 -> origin/dependabot/npm_and_yarn/app/assets/javascripts/uppy/aws-s3-3.5.0
 * [new branch]          dependabot/npm_and_yarn/app/assets/javascripts/uppy/aws-s3-multipart-3.9.0 -> origin/dependabot/npm_and_yarn/app/assets/javascripts/uppy/aws-s3-multipart-3.9.0
 * [new branch]          dependabot/npm_and_yarn/app/assets/javascripts/uppy/core-3.7.1 -> origin/dependabot/npm_and_yarn/app/assets/javascripts/uppy/core-3.7.1
 * [new branch]          dependabot/npm_and_yarn/app/assets/javascripts/uppy/utils-5.6.0 -> origin/dependabot/npm_and_yarn/app/assets/javascripts/uppy/utils-5.6.0
 * [new branch]          dependabot/npm_and_yarn/app/assets/javascripts/uppy/xhr-upload-3.5.0 -> origin/dependabot/npm_and_yarn/app/assets/javascripts/uppy/xhr-upload-3.5.0
 * [new branch]          desktop-notification-handlers -> origin/desktop-notification-handlers
 * [new branch]          dev/add-a-reference-column-to-notifications -> origin/dev/add-a-reference-column-to-notifications
 + ee95e6359...a2b413786 dev/add-new-chat-metrics -> origin/dev/add-new-chat-metrics  (forced update)
 * [new branch]          dev/drop-notification_id-column-on-chat-mentions -> origin/dev/drop-notification_id-column-on-chat-mentions
 * [new branch]          dev/dry-up-user-status-serializers -> origin/dev/dry-up-user-status-serializers
 * [new branch]          dev/flaky-chat-details-collapser-spec -> origin/dev/flaky-chat-details-collapser-spec
 * [new branch]          dev/no-need-to-include-user-status-twice -> origin/dev/no-need-to-include-user-status-twice
 * [new branch]          dev/remove-legacy-hamburger-menu -> origin/dev/remove-legacy-hamburger-menu
 * [new branch]          dev/rethink-mass-mentions -> origin/dev/rethink-mass-mentions
 * [new branch]          dev/rethink-mass-mentions-polymorphic -> origin/dev/rethink-mass-mentions-polymorphic
 * [new branch]          dev/revert-guardian-devex -> origin/dev/revert-guardian-devex
 + 54940ca5c...3eacfe150 dev_paginate_categories_json -> origin/dev_paginate_categories_json  (forced update)
 * [new branch]          dev_watched_words     -> origin/dev_watched_words
 + 46fe4511f...a9d0e7069 ember-4               -> origin/ember-4  (forced update)
 * [new branch]          ember-5-plugins       -> origin/ember-5-plugins
 * [new branch]          extract_stripper      -> origin/extract_stripper
 * [new branch]          feature/admin-sidebar-v0 -> origin/feature/admin-sidebar-v0
   9a6c6fd5b..8780daeee  feature/bookmark-menu-and-modal-redesign -> origin/feature/bookmark-menu-and-modal-redesign
 * [new branch]          feature/migrations-schema-generator -> origin/feature/migrations-schema-generator
 * [new branch]          feature_watched_words_validation -> origin/feature_watched_words_validation
 * [new branch]          file-size-restriction -> origin/file-size-restriction
   43df37774..4393dd37e  filtered-events       -> origin/filtered-events
 * [new branch]          fix/count-mass-mentions-correctly -> origin/fix/count-mass-mentions-correctly
 * [new branch]          full-page             -> origin/full-page
   ce7fb6906..31e32f8a1  generic-import-latest -> origin/generic-import-latest
 * [new branch]          generic-import-latest-snowflake -> origin/generic-import-latest-snowflake
 * [new branch]          generic-import-update -> origin/generic-import-update
 * [new branch]          image-assets          -> origin/image-assets
 * [new branch]          independent-theme-compiler -> origin/independent-theme-compiler
 * [new branch]          input-text-field-component -> origin/input-text-field-component
 * [new branch]          issue/fix-guardian-init-basic-user -> origin/issue/fix-guardian-init-basic-user
 * [new branch]          keyword-site-setting  -> origin/keyword-site-setting
 + bdcc24f96...92e545292 list-wrapper          -> origin/list-wrapper  (forced update)
   e3636078c..5def32699  mock-chat-index-redesign -> origin/mock-chat-index-redesign
 * [new branch]          move_inflector        -> origin/move_inflector
 * [new branch]          multi-invite          -> origin/multi-invite
 * [new branch]          noti-glimmer          -> origin/noti-glimmer
 * [new branch]          oklab                 -> origin/oklab
 * [new branch]          poc/auto-after-before-plugin-outlets -> origin/poc/auto-after-before-plugin-outlets
 * [new branch]          poc/insert-before-after-wrapping-outlet -> origin/poc/insert-before-after-wrapping-outlet
 * [new branch]          pretty-text-please    -> origin/pretty-text-please
 * [new branch]          reload-page-user-prefs-2fa -> origin/reload-page-user-prefs-2fa
 * [new branch]          revert-maxmind        -> origin/revert-maxmind
   ef3f41543..223626d7a  stable                -> origin/stable
 * [new branch]          support-load-more-btn -> origin/support-load-more-btn
 * [new branch]          test-issue-temporary  -> origin/test-issue-temporary
   043b4a418..4949d85c1  tests-passed          -> origin/tests-passed
 * [new branch]          tmp/reimplement-all-and-here-mentions -> origin/tmp/reimplement-all-and-here-mentions
 * [new branch]          tobiaseigen-patch-3   -> origin/tobiaseigen-patch-3
 * [new branch]          transform-hooks       -> origin/transform-hooks
 * [new branch]          transform-post-fn-api -> origin/transform-post-fn-api
 * [new branch]          tweaks-2-fr           -> origin/tweaks-2-fr
 * [new branch]          username_cache        -> origin/username_cache
 * [new branch]          ux-add-sorting-params-to-outlet -> origin/ux-add-sorting-params-to-outlet
 t [tag update]          beta                  -> beta
 t [tag update]          latest-release        -> latest-release
 * [new tag]             v3.1.3                -> v3.1.3
 * [new tag]             v3.2.0.beta3          -> v3.2.0.beta3
I, [2023-12-11T06:37:05.107220 #1]  INFO -- : 
I, [2023-12-11T06:37:05.107337 #1]  INFO -- : > cd /var/www/discourse && sudo -H -E -u discourse bash -c '
  set -o errexit
  if [[ $(git symbolic-ref --short HEAD) == tests-passed ]] ; then
      git pull
  else
      git -c advice.detachedHead=false checkout tests-passed
  fi
'
Switched to a new branch 'tests-passed'
I, [2023-12-11T06:37:10.039336 #1]  INFO -- : Branch 'tests-passed' set up to track remote branch 'tests-passed' from 'origin'.

I, [2023-12-11T06:37:10.039485 #1]  INFO -- : > cd /var/www/discourse && sudo -H -E -u discourse git config user.discourse-version tests-passed
I, [2023-12-11T06:37:10.055796 #1]  INFO -- : 
I, [2023-12-11T06:37:10.055932 #1]  INFO -- : > cd /var/www/discourse && mkdir -p tmp
I, [2023-12-11T06:37:10.062383 #1]  INFO -- : 
I, [2023-12-11T06:37:10.062911 #1]  INFO -- : > cd /var/www/discourse && chown discourse:www-data tmp
I, [2023-12-11T06:37:10.068906 #1]  INFO -- : 
I, [2023-12-11T06:37:10.069051 #1]  INFO -- : > cd /var/www/discourse && mkdir -p tmp/pids
I, [2023-12-11T06:37:10.075337 #1]  INFO -- : 
I, [2023-12-11T06:37:10.075478 #1]  INFO -- : > cd /var/www/discourse && mkdir -p tmp/sockets
I, [2023-12-11T06:37:10.081087 #1]  INFO -- : 
I, [2023-12-11T06:37:10.081222 #1]  INFO -- : > cd /var/www/discourse && touch tmp/.gitkeep
I, [2023-12-11T06:37:10.087004 #1]  INFO -- : 
I, [2023-12-11T06:37:10.087139 #1]  INFO -- : > cd /var/www/discourse && mkdir -p                    /shared/log/rails
I, [2023-12-11T06:37:10.093140 #1]  INFO -- : 
I, [2023-12-11T06:37:10.093305 #1]  INFO -- : > cd /var/www/discourse && bash -c "touch -a           /shared/log/rails/{production,production_errors,unicorn.stdout,unicorn.stderr,sidekiq}.log"
I, [2023-12-11T06:37:10.100413 #1]  INFO -- : 
I, [2023-12-11T06:37:10.100570 #1]  INFO -- : > cd /var/www/discourse && bash -c "ln    -s           /shared/log/rails/{production,production_errors,unicorn.stdout,unicorn.stderr,sidekiq}.log /var/www/discourse/log"
I, [2023-12-11T06:37:10.108011 #1]  INFO -- : 
I, [2023-12-11T06:37:10.108130 #1]  INFO -- : > cd /var/www/discourse && bash -c "mkdir -p           /shared/{uploads,backups}"
I, [2023-12-11T06:37:10.116596 #1]  INFO -- : 
I, [2023-12-11T06:37:10.116726 #1]  INFO -- : > cd /var/www/discourse && bash -c "ln    -s           /shared/{uploads,backups} /var/www/discourse/public"
I, [2023-12-11T06:37:10.124273 #1]  INFO -- : 
I, [2023-12-11T06:37:10.124421 #1]  INFO -- : > cd /var/www/discourse && bash -c "mkdir -p           /shared/tmp/{backups,restores}"
I, [2023-12-11T06:37:10.134061 #1]  INFO -- : 
I, [2023-12-11T06:37:10.134228 #1]  INFO -- : > cd /var/www/discourse && bash -c "ln    -s           /shared/tmp/{backups,restores} /var/www/discourse/tmp"
I, [2023-12-11T06:37:10.141458 #1]  INFO -- : 
I, [2023-12-11T06:37:10.141602 #1]  INFO -- : > cd /var/www/discourse && chown -R discourse:www-data /shared/log/rails /shared/uploads /shared/backups /shared/tmp
I, [2023-12-11T06:37:10.163707 #1]  INFO -- : 
I, [2023-12-11T06:37:10.163854 #1]  INFO -- : > cd /var/www/discourse && [ ! -d public/plugins ] || find public/plugins/ -maxdepth 1 -xtype l -delete
I, [2023-12-11T06:37:10.169471 #1]  INFO -- : 
I, [2023-12-11T06:37:10.170368 #1]  INFO -- : Replacing # redis with sv start redis || exit 1 in /etc/service/unicorn/run
I, [2023-12-11T06:37:10.171519 #1]  INFO -- : > cd /var/www/discourse/plugins && git clone https://github.com/discourse/docker_manager.git
Cloning into 'docker_manager'...
I, [2023-12-11T06:37:11.142485 #1]  INFO -- : 
I, [2023-12-11T06:37:11.142680 #1]  INFO -- : > cd /var/www/discourse/plugins && git clone https://github.com/discourse/discourse-solved.git
Cloning into 'discourse-solved'...
I, [2023-12-11T06:37:11.877496 #1]  INFO -- : 
I, [2023-12-11T06:37:11.878167 #1]  INFO -- : > cd /var/www/discourse/plugins && git clone https://github.com/paviliondev/discourse-locations.git
Cloning into 'discourse-locations'...
I, [2023-12-11T06:37:12.713978 #1]  INFO -- : 
I, [2023-12-11T06:37:12.714105 #1]  INFO -- : > cd /var/www/discourse/plugins && git clone https://github.com/discourse/discourse-data-explorer.git
Cloning into 'discourse-data-explorer'...
I, [2023-12-11T06:37:13.532864 #1]  INFO -- : 
I, [2023-12-11T06:37:13.533001 #1]  INFO -- : > cd /var/www/discourse/plugins && git clone https://github.com/discourse/discourse-adplugin.git
Cloning into 'discourse-adplugin'...
I, [2023-12-11T06:37:14.305555 #1]  INFO -- : 
I, [2023-12-11T06:37:14.305817 #1]  INFO -- : > cd /var/www/discourse/plugins && git clone https://github.com/discourse/discourse-reactions.git
Cloning into 'discourse-reactions'...
I, [2023-12-11T06:37:15.045125 #1]  INFO -- : 
I, [2023-12-11T06:37:15.045247 #1]  INFO -- : > cd /var/www/discourse/plugins && git clone https://github.com/discourse/discourse-templates.git
Cloning into 'discourse-templates'...
I, [2023-12-11T06:37:15.809915 #1]  INFO -- : 
I, [2023-12-11T06:37:15.810046 #1]  INFO -- : > cd /var/www/discourse/plugins && git clone https://github.com/discourse/discourse-affiliate.git
Cloning into 'discourse-affiliate'...
I, [2023-12-11T06:37:16.433061 #1]  INFO -- : 
I, [2023-12-11T06:37:16.434123 #1]  INFO -- : > cp /var/www/discourse/config/nginx.sample.conf /etc/nginx/conf.d/discourse.conf
I, [2023-12-11T06:37:16.442090 #1]  INFO -- : 
I, [2023-12-11T06:37:16.442586 #1]  INFO -- : > rm /etc/nginx/sites-enabled/default
I, [2023-12-11T06:37:16.446759 #1]  INFO -- : 
I, [2023-12-11T06:37:16.447263 #1]  INFO -- : > mkdir -p /var/nginx/cache
I, [2023-12-11T06:37:16.451474 #1]  INFO -- : 
I, [2023-12-11T06:37:16.452405 #1]  INFO -- : Replacing pid /run/nginx.pid; with daemon off; in /etc/nginx/nginx.conf
I, [2023-12-11T06:37:16.453144 #1]  INFO -- : Replacing (?m-ix:upstream[^\}]+\}) with upstream discourse { server 127.0.0.1:3000; } in /etc/nginx/conf.d/discourse.conf
I, [2023-12-11T06:37:16.453873 #1]  INFO -- : Replacing (?-mix:server_name.+$) with server_name _ ; in /etc/nginx/conf.d/discourse.conf
I, [2023-12-11T06:37:16.454418 #1]  INFO -- : Replacing (?-mix:client_max_body_size.+$) with client_max_body_size $upload_size ; in /etc/nginx/conf.d/discourse.conf
I, [2023-12-11T06:37:16.454991 #1]  INFO -- : Replacing (?-mix:worker_connections.+$) with worker_connections $nginx_worker_connections ; in /etc/nginx/nginx.conf
I, [2023-12-11T06:37:16.455481 #1]  INFO -- : > echo "done configuring web"
I, [2023-12-11T06:37:16.459601 #1]  INFO -- : done configuring web

I, [2023-12-11T06:37:16.460326 #1]  INFO -- : > cd /var/www/discourse && gem install bundler --conservative -v $(awk '/BUNDLED WITH/ { getline; gsub(/ /,""); print $0 }' Gemfile.lock)
I, [2023-12-11T06:37:16.836806 #1]  INFO -- : 
I, [2023-12-11T06:37:16.836974 #1]  INFO -- : > cd /var/www/discourse && find /var/www/discourse ! -user discourse -exec chown discourse {} \+
I, [2023-12-11T06:37:19.438303 #1]  INFO -- : 
I, [2023-12-11T06:37:19.438984 #1]  INFO -- : > cd /var/www/discourse && if [ "tests-passed" != "tests-passed" ]; then
  rm -rf app/assets/javascripts/node_modules
fi
I, [2023-12-11T06:37:19.442692 #1]  INFO -- : 
I, [2023-12-11T06:37:19.442789 #1]  INFO -- : > cd /var/www/discourse && su discourse -c 'yarn install --frozen-lockfile && yarn cache clean'
warning " > @glint/environment-ember-loose@1.1.0" has unmet peer dependency "@glimmer/component@^1.1.2".
warning " > @glint/environment-ember-template-imports@1.1.0" has unmet peer dependency "ember-template-imports@^3.0.0".
warning Resolution field "unset-value@2.0.1" is incompatible with requested version "unset-value@^1.0.0"
warning Pattern ["wrap-ansi@^7.0.0"] is trying to unpack in the same destination "/home/discourse/.cache/yarn/v6/npm-wrap-ansi-cjs-7.0.0-67e145cff510a6a6984bdf1152911d69d2eb9e43-integrity/node_modules/wrap-ansi-cjs" as pattern ["wrap-ansi-cjs@npm:wrap-ansi@^7.0.0"]. This could result in non-deterministic behavior, skipping.
warning " > discourse-markdown-it@1.0.0" has unmet peer dependency "xss@*".
warning "workspace-aggregator-4a0ff1cd-82ba-483f-8207-6b2c0f0b31fb > discourse > @ember/legacy-built-in-components@0.5.0" has incorrect peer dependency "ember-source@>= 4.8".
warning "workspace-aggregator-4a0ff1cd-82ba-483f-8207-6b2c0f0b31fb > discourse > @uppy/aws-s3@3.0.6" has incorrect peer dependency "@uppy/core@^3.1.2".
warning "workspace-aggregator-4a0ff1cd-82ba-483f-8207-6b2c0f0b31fb > discourse > @uppy/aws-s3-multipart@3.1.3" has incorrect peer dependency "@uppy/core@^3.1.2".
warning "workspace-aggregator-4a0ff1cd-82ba-483f-8207-6b2c0f0b31fb > discourse > @uppy/xhr-upload@3.1.1" has incorrect peer dependency "@uppy/core@^3.1.2".
warning "workspace-aggregator-4a0ff1cd-82ba-483f-8207-6b2c0f0b31fb > discourse-plugins > ember-this-fallback@0.4.0" has unmet peer dependency "ember-source@^3.28.11 || ^4.0.0".
warning "workspace-aggregator-4a0ff1cd-82ba-483f-8207-6b2c0f0b31fb > discourse > @uppy/aws-s3 > @uppy/xhr-upload@3.3.0" has incorrect peer dependency "@uppy/core@^3.2.1".
I, [2023-12-11T06:39:22.730680 #1]  INFO -- : yarn install v1.22.19
[1/5] Validating package.json...
[2/5] Resolving packages...
[3/5] Fetching packages...
[4/5] Linking dependencies...
[5/5] Building fresh packages...
$ yarn --cwd app/assets/javascripts $(node -e 'const argv = JSON.parse(process.env.npm_config_argv).original; const passthrough = [`--frozen-lockfile`, `-s`].filter(arg => argv.includes(arg)); console.log(passthrough.join(` `));')
yarn install v1.22.19
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
$ ./run-patch-package
patch-package 8.0.0
Applying patches...
babel-plugin-debug-macros@0.3.4 ✔
content-tag@1.1.2 ✔
ember-cli@5.0.0 ✔
ember-this-fallback@0.4.0 (1 deprecation-name) ✔
ember-this-fallback@0.4.0 (2 themes) ✔
virtual-dom@2.1.1 ✔
Done in 64.07s.
Done in 115.18s.
yarn cache v1.22.19
success Cleared cache.
Done in 7.53s.

I, [2023-12-11T06:39:22.731495 #1]  INFO -- : > cd /var/www/discourse && su discourse -c 'bundle config --local deployment true'
I, [2023-12-11T06:39:23.103216 #1]  INFO -- : 
I, [2023-12-11T06:39:23.103337 #1]  INFO -- : > cd /var/www/discourse && su discourse -c 'bundle config --local without "development test"'
I, [2023-12-11T06:39:23.331037 #1]  INFO -- : You are replacing the current local value of without, which is currently "test:development"

I, [2023-12-11T06:39:23.331186 #1]  INFO -- : > cd /var/www/discourse && su discourse -c 'bundle install --retry 3 --jobs 4'
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

current directory:
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rbtrace-0.5.0/ext
/usr/local/bin/ruby extconf.rb
  -- tar zxvfo msgpack-1.1.0.tar.gz
  -- env CFLAGS=nil LDFLAGS=nil CC=nil
-- ./configure --disable-dependency-tracking --disable-shared --with-pic
--prefix=/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rbtrace-0.5.0/ext/dst/
--libdir=/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rbtrace-0.5.0/ext/dst/lib
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers.  Check the mkmf.log file for more details.  You may
need configuration options.

Provided configuration options:
	--with-opt-dir
	--without-opt-dir
	--with-opt-include
	--without-opt-include=${opt-dir}/include
	--with-opt-lib
	--without-opt-lib=${opt-dir}/lib
	--with-make-prog
	--without-make-prog
	--srcdir=.
	--curdir
	--ruby=/usr/local/bin/$(RUBY_BASE_NAME)
extconf.rb:6:in `sys': ./configure --disable-dependency-tracking
--disable-shared --with-pic
--prefix=/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rbtrace-0.5.0/ext/dst/
--libdir=/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rbtrace-0.5.0/ext/dst/lib
failed, please report to https://github.com/tmm1/rbtrace/issues (RuntimeError)
	from extconf.rb:42:in `block (2 levels) in <main>'
	from extconf.rb:34:in `chdir'
	from extconf.rb:34:in `block in <main>'
	from extconf.rb:30:in `chdir'
	from extconf.rb:30:in `<main>'

To see why this extension failed to compile, please check the mkmf.log which can
be found here:

/var/www/discourse/vendor/bundle/ruby/3.2.0/extensions/x86_64-linux/3.2.0/rbtrace-0.5.0/mkmf.log

extconf failed, exit code 1

Gem files will remain installed in
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rbtrace-0.5.0 for inspection.
Results logged to
/var/www/discourse/vendor/bundle/ruby/3.2.0/extensions/x86_64-linux/3.2.0/rbtrace-0.5.0/gem_make.out

  /usr/local/lib/ruby/site_ruby/3.2.0/rubygems/ext/builder.rb:120:in `run'
/usr/local/lib/ruby/site_ruby/3.2.0/rubygems/ext/ext_conf_builder.rb:28:in
`build'
/usr/local/lib/ruby/site_ruby/3.2.0/rubygems/ext/builder.rb:188:in
`build_extension'
/usr/local/lib/ruby/site_ruby/3.2.0/rubygems/ext/builder.rb:222:in `block in
build_extensions'
  /usr/local/lib/ruby/site_ruby/3.2.0/rubygems/ext/builder.rb:219:in `each'
/usr/local/lib/ruby/site_ruby/3.2.0/rubygems/ext/builder.rb:219:in
`build_extensions'
/usr/local/lib/ruby/site_ruby/3.2.0/rubygems/installer.rb:844:in
`build_extensions'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/rubygems_gem_installer.rb:72:in
`build_extensions'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/rubygems_gem_installer.rb:28:in
`install'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/source/rubygems.rb:198:in
`install'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/installer/gem_installer.rb:54:in
`install'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/installer/gem_installer.rb:16:in
`install_from_spec'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/installer/parallel_installer.rb:156:in
`do_install'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/installer/parallel_installer.rb:147:in
`block in worker_pool'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/worker.rb:62:in
`apply_func'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/worker.rb:57:in
`block in process_queue'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/worker.rb:54:in
`loop'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/worker.rb:54:in
`process_queue'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/worker.rb:90:in
`block (2 levels) in create_threads'

An error occurred while installing rbtrace (0.5.0), and Bundler cannot continue.

In Gemfile:
  rbtrace
I, [2023-12-11T06:40:09.131584 #1]  INFO -- : Fetching gem metadata from https://rubygems.org/.........
Fetching rake 13.1.0
Installing rake 13.1.0
Using concurrent-ruby 1.2.2
Using minitest 5.20.0
Using builder 3.2.4
Using erubi 1.12.0
Fetching racc 1.7.3
Using crass 1.0.6
Using rack 2.2.8
Using mini_mime 1.1.5
Fetching date 3.3.4
Fetching public_suffix 5.0.4
Fetching timeout 0.4.1
Installing timeout 0.4.1
Installing public_suffix 5.0.4
Installing date 3.3.4 with native extensions
Installing racc 1.7.3 with native extensions
Fetching aws-eventstream 1.3.0
Installing aws-eventstream 1.3.0
Using aws-partitions 1.583.0
Using jmespath 1.6.2
Fetching base64 0.2.0
Installing base64 0.2.0
Fetching bigdecimal 3.1.4
Using msgpack 1.7.2
Using bundler 2.4.13
Using byebug 11.1.3
Using cbor 0.5.9.6
Fetching cgi 0.4.1
Installing bigdecimal 3.1.4 with native extensions
Installing cgi 0.4.1 with native extensions
Using chunky_png 1.4.0
Using coderay 1.1.3
Using colored2 4.0.0
Using connection_pool 2.4.1
Using openssl 3.2.0
Using cppjieba_rb 0.4.2
Using diffy 3.4.2
Using digest 3.1.1
Using discourse-fonts 0.0.9
Using email_reply_trimmer 0.1.13
Fetching excon 0.105.0
Installing excon 0.105.0
Using execjs 2.9.1
Using exifr 1.4.0
Using faraday-net_http 3.0.2
Using ruby2_keywords 0.0.5
Using fast_blank 1.0.1
Using fast_xs 0.8.0
Using fastimage 2.2.7
Using ffi 1.16.3
Using fspath 3.1.2
Using gc_tracer 1.5.1
Fetching google-protobuf 3.25.1 (x86_64-linux)
Installing google-protobuf 3.25.1 (x86_64-linux)
Using guess_html_encoding 0.0.11
Using hana 1.3.7
Using hashie 5.0.0
Using highline 2.1.0
Using htmlentities 4.3.4
Using http_accept_language 2.1.1
Using image_size 3.3.0
Using in_threads 1.6.0
Using progress 3.6.0
Fetching json 2.7.1
Installing json 2.7.1 with native extensions
Fetching regexp_parser 2.8.3
Installing regexp_parser 2.8.3
Fetching unf_ext 0.0.9.1
Installing unf_ext 0.0.9.1 with native extensions
Using jwt 2.7.1
Using kgio 2.11.4
Using libv8-node 18.16.0.0 (x86_64-linux)
Using method_source 1.0.0
Using thor 1.3.0
Using zeitwerk 2.6.12
Using logstash-event 1.2.02
Fetching logster 2.13.1
Installing logster 2.13.1
Using lru_redux 1.1.0
Using lz4-ruby 0.3.3
Using maxminddb 0.1.22
Using memory_profiler 1.0.1
Using redis 4.8.1
Using mini_sql 1.5.0
Using multi_json 1.15.0
Using multi_xml 0.6.0
Using mustache 1.1.1
Fetching uri 0.13.0
Installing uri 0.13.0
Fetching nio4r 2.7.0
Installing nio4r 2.7.0 with native extensions
Using version_gem 1.1.3
Using optimist 3.1.0
Using pg 1.5.4
Using raindrops 0.20.1
Using rchardet 1.8.0
Using rexml 3.2.6
Using rinku 2.0.6
Using rotp 6.3.0
Using rqrcode_core 1.2.0
Using rubyzip 2.3.2
Using sshkey 3.0.0
Using stackprof 0.2.25
Using i18n 1.14.1
Using tzinfo 2.0.6
Using rack-test 2.1.0
Using request_store 1.5.1
Using message_bus 4.3.8
Fetching rack-mini-profiler 3.3.0
Installing rack-mini-profiler 3.3.0
Using rack-protection 3.1.0
Using sprockets 3.7.2 from https://github.com/rails/sprockets (at 3.x@f4d3dae)
Fetching net-protocol 0.2.2
Installing net-protocol 0.2.2
Fetching addressable 2.8.6
Installing addressable 2.8.6
Using aws-sigv4 1.5.0
Fetching bootsnap 1.17.0
Installing bootsnap 1.17.0 with native extensions
Using openssl-signature_algorithm 1.3.0
Using uglifier 4.2.0
Fetching faraday 2.7.12
Installing faraday 2.7.12
Using mini_suffix 0.3.3
Fetching nokogiri 1.15.5 (x86_64-linux)
Installing nokogiri 1.15.5 (x86_64-linux)
Using image_optim 0.31.3
Using omniauth 1.9.2
Fetching sass-embedded 1.69.5 (x86_64-linux-gnu)
Installing sass-embedded 1.69.5 (x86_64-linux-gnu)
Using unf 0.1.4
Fetching web-push 3.0.1
Installing web-push 3.0.1
Using logstash-logger 0.26.1
Using mini_racer 0.8.0
Using pry 0.14.2
Using sidekiq 6.5.12
Using redis-namespace 1.11.0
Fetching net-http 0.4.0
Installing net-http 0.4.0
Using oauth-tty 1.0.5
Using snaky_hash 2.0.1
Fetching rbtrace 0.5.0
Installing rbtrace 0.5.0 with native extensions
Using activesupport 7.0.7
Using rqrcode 2.2.0
Using rss 0.3.0
Using tzinfo-data 1.2023.3
Using unicorn 6.1.0
Using net-pop 0.1.2
Using net-smtp 0.4.0
Using css_parser 1.16.0
Using aws-sdk-core 3.130.2
Using cose 1.3.0
Using faraday-retry 2.2.0
Using oauth2 1.4.11
Fetching loofah 2.22.0
Installing loofah 2.22.0
Using ruby-readability 0.7.0
Using sanitize 6.1.0
Fetching sassc-embedded 1.68.6
Installing sassc-embedded 1.68.6
Using simpleidn 0.2.1
Using rtlcss 0.2.1
Using pry-byebug 3.10.1
Using pry-rails 0.3.9
Using mini_scheduler 0.16.0
Using oauth 1.1.0
Using rails-dom-testing 2.2.0
Using globalid 1.2.1
Using activemodel 7.0.7
Using aws-sdk-kms 1.56.0
Using aws-sdk-sns 1.53.0
Using omniauth-oauth2 1.7.3
Using rails-html-sanitizer 1.6.0
Fetching json_schemer 2.1.1
Installing json_schemer 2.1.1
Using omniauth-oauth 1.2.0
Using actionview 7.0.7
Using activejob 7.0.7
Using active_model_serializers 0.8.4
Using activerecord 7.0.7
Using aws-sdk-s3 1.114.0
Using omniauth-facebook 9.0.0
Using omniauth-github 1.4.0
Using omniauth-google-oauth2 0.8.2
Using omniauth-twitter 1.4.0
Using actionpack 7.0.7
Using actionview_precompiler 0.3.0
Using discourse-seed-fu 2.3.12
Using railties 7.0.7
Using sprockets-rails 3.4.2
Using lograge 0.14.0
Using rails_failover 2.0.1
Using rails_multisite 5.0.0
Using puma 6.4.0
Fetching net-imap 0.4.7
Installing net-imap 0.4.7
Using mail 2.8.1
Using actionmailer 7.0.7
Fetching oj 3.16.2
Installing oj 3.16.2 with native extensions

I, [2023-12-11T06:40:09.132711 #1]  INFO -- : Terminating async processes
I, [2023-12-11T06:40:09.133069 #1]  INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main pid: 40
2023-12-11 06:40:09.133 UTC [40] LOG:  received fast shutdown request
I, [2023-12-11T06:40:09.133831 #1]  INFO -- : Sending TERM to exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf pid: 109
109:signal-handler (1702276809) Received SIGTERM scheduling shutdown...
2023-12-11 06:40:09.138 UTC [40] LOG:  aborting any active transactions
2023-12-11 06:40:09.144 UTC [40] LOG:  background worker "logical replication launcher" (PID 49) exited with exit code 1
2023-12-11 06:40:09.146 UTC [44] LOG:  shutting down
2023-12-11 06:40:09.168 UTC [40] LOG:  database system is shut down
109:M 11 Dec 2023 06:40:09.173 # User requested shutdown...
109:M 11 Dec 2023 06:40:09.173 * Saving the final RDB snapshot before exiting.
109:M 11 Dec 2023 06:40:10.723 * DB saved on disk
109:M 11 Dec 2023 06:40:10.723 # Redis is now ready to exit, bye bye...


FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && su discourse -c 'bundle install --retry 3 --jobs 4' failed with return #<Process::Status: pid 585 exit 5>
Location of failure: /usr/local/lib/ruby/gems/3.2.0/gems/pups-1.2.1/lib/pups/exec_command.rb:132:in `spawn'
exec failed with the params {"cd"=>"$home", "hook"=>"bundle_exec", "cmd"=>["su discourse -c 'bundle config --local deployment true'", "su discourse -c 'bundle config --local without \"development test\"'", "su discourse -c 'bundle install --retry 3 --jobs 4'"]}
bootstrap failed with exit code 5
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.
9d6071df716185b1c31a8703028a03d6fb0198216704c41506a69b09c67d4519
root@greyarro:/var/discourse#

I ran ./discourse-doctor which looked like it did lots of scary things that I didn’t understand but unfortunately it made no difference and it ended with what appears to be the same message:

FAILED

--------------------

Pups::ExecError: cd /var/www/discourse && su discourse -c 'bundle install --retry 3 --jobs 4' failed with return #<Process::Status: pid 585 exit 5>

Location of failure: /usr/local/lib/ruby/gems/3.2.0/gems/pups-1.2.1/lib/pups/exec_command.rb:132:in `spawn'

exec failed with the params {"cd"=>"$home", "hook"=>"bundle_exec", "cmd"=>["su discourse -c 'bundle config --local deployment true'", "su discourse -c 'bundle config --local without \"development test\"'", "su discourse -c 'bundle install --retry 3 --jobs 4'"]}

bootstrap failed with exit code 5

** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.

./discourse-doctor may help diagnose the problem.

9f72d4c67a46bda65d95233149ac8f661bdae02e70ae9a265666920d71de14e2

==================== END REBUILD LOG ====================

Failed to rebuild app.

Can anybody shed any light on the issue, or help me get us back online??

2 
Gem::Ext::BuildError: FEHLER: Native Erweiterung konnte nicht erstellt werden.

aktuelles Verzeichnis:
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rbtrace-0.5.0/ext

Ist dies eine Standardinstallation?

1 „Gefällt mir“
3

Soweit ich mich erinnern kann, @RGJ – aber es ist ungefähr sechs Jahre her, seit ich es zum ersten Mal erstellt habe, aber ich erinnere mich nicht, damals etwas Ungewöhnliches getan zu haben.

Ich aktualisiere auch immer etwa einmal pro Woche mit der Befehlszeile, dies ist das erste Mal, dass das Update in all diesen Jahren fehlgeschlagen ist.

Ich bin mit Ruby überhaupt nicht vertraut, was schlägt diese Nachricht Ihnen vor? :thinking:

4

Die Nachricht legt nahe, dass etwas veraltet oder nicht standardmäßig ist, aber sie zieht das neueste Image, daher ergibt das keinen Sinn.

1 „Gefällt mir“
5

Könnte dies ein Problem mit einem veralteten Betriebssystem sein? :thinking:

Läuft derzeit auf:

NAME="Ubuntu"

VERSION="18.04.6 LTS (Bionic Beaver)"

ID=ubuntu

ID_LIKE=debian

PRETTY_NAME="Ubuntu 18.04.6 LTS"

VERSION_ID="18.04"

HOME_URL="https://www.ubuntu.com/"

SUPPORT_URL="https://help.ubuntu.com/"

BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"

PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"

VERSION_CODENAME=bionic

UBUNTU_CODENAME=bionic
6

Ich muss jetzt vom Schlimmsten ausgehen, dass mein Discourse tot ist.

Ich habe eine Kopie meiner neuesten app.yml und gerade das gestrige Datenbank-Backup von S3 heruntergeladen.

Ich glaube, ich werde mein Ubuntu von 18.04.6 LTS auf die neueste Version mit do-release-upgrade aktualisieren. Dann werde ich wohl eine Neuinstallation gemäß der Anleitung unter discourse/docs/INSTALL-cloud.md at main · discourse/discourse · GitHub durchführen.

Nach der Installation kann ich dann mein 1,5 GB großes Datenbank-Backup über das Admin-Panel hochladen? :thinking:

Ich benutze seit Jahren S3 für Uploads, daher hoffe ich, dass die hochgeladenen Benutzerinhalte von dort noch zugänglich sind.

Gibt es irgendwelche Fallstricke? Gibt es etwas, das nicht in der Datenbank enthalten sein wird? Irgendwelche Einstellungen? Irgendwelche Dateien oder andere Inhalte?

:flushed:

7

Ich habe einen (Digital Ocean) Snapshot des Servers in seinem aktuellen Zustand erstellt, das Release-Upgrade läuft jetzt.\n\nAus Interesse: Ich habe ein Upgrade von https://github.com/discourse/discourse/compare/aadc104817bb0306856b4718f5c50fb481e77c16…4949d85c15be99abb3515e659ef674d58cc2c4c4 durchgeführt und Sie werden sehen, dass ich nur 128x Commits / 11x Tage hinter dem neuesten Stand war.\n\nGibt es eine Möglichkeit, diese vorherige Version mit git pull wieder herunterzuladen und damit neu zu erstellen? Nur um unsere Community wieder zum Laufen zu bringen? :thinking:

8

Haben Sie versucht, ./launcher start app auszuführen, um die Vorab-Rebuild-Version wiederherzustellen?

2 „Gefällt mir“
9

Ach, in meiner blinden Panik, dass heute Morgen alles fehlschlägt, habe ich alle Container entfernt, die nicht liefen :pensive:

10

Ich denke, solange Sie Ihre app.yml-Informationen und Ihr aktuelles Backup haben, sollten Sie im Handumdrehen auf einem neuen Droplet wieder online sein. :+1:

3 „Gefällt mir“
11

Whoa!!

Ich glaube, ich habe mir selbst einen Rettungsanker zugeworfen :scream:

Während ich auf das Upgrade der Distribution warte, habe ich anscheinend sehr clever die Digital Ocean-Backups aktiviert :scream:

Screenshot 2023-12-11 at 09.42.14
Screenshot 2023-12-11 at 09.42.142192×1040 155 KB

Zeit zu sehen, ob sie gut sind :grimacing:

Ok, Plan B also…

Stelle das Digital Ocean-Snapshot-Backup von letzter Woche vom 04.12.2023 wieder her

Lade mein Datenbank-Backup von gestern Abend (11.12.2023, 2 Uhr morgens) hoch

Ich denke, das bringt uns ungefähr dorthin zurück, wo wir heute Morgen um 6 Uhr waren, als alles zusammenbrach.

Ich habe noch nie eine Datenbank wiederhergestellt.

Angenommen, das D.O.-Backup/-Wiederherstellung funktioniert und bringt mich bis zum Montag letzter Woche zurück, kann ich die 1,5 GB große backup.tar.gz-Datei einfach über die Admin-Backup-Menüs hochladen? Muss ich sie vorher extrahieren oder etwas anderes tun?

Sollte ich die Website währenddessen in den schreibgeschützten Modus versetzen oder etwas anderes tun?

12

Server ist hochgefahren! (vom Snapshot letzten Montag)

Es stellt sich heraus, dass ihr, die sehr klugen Leute von Discourse, einen “Wiederherstellen”-Button direkt in das Admin-Menü eingebaut habt, sodass ich das Backup nicht einmal hochladen muss :bowing_man:

Screenshot 2023-12-11 at 09.59.56
Screenshot 2023-12-11 at 09.59.562130×424 29.4 KB

:crossed_fingers:

13

Screenshot 2023-12-11 at 10.01.29
Screenshot 2023-12-11 at 10.01.291790×590 66.8 KB

:sob:

Möglicherweise, weil ich die Website vor dem Wiederherstellungsversuch in den schreibgeschützten Modus versetzt habe.

Ich schalte den schreibgeschützten Modus aus und versuche die Wiederherstellung erneut…

14

Nein, das hat auch nicht funktioniert.

Fehlerprotokoll:

Ich habe die App gerade wieder gestoppt.

Wenn ich mir mein Sidekiq ansehe, hat es über 12.000 Jobs.

Screenshot 2023-12-11 at 10.09.06
Screenshot 2023-12-11 at 10.09.062382×444 86.2 KB

Es sieht so aus, als würde versucht, die geplanten E-Mail-Ankündigungen von letzter Woche erneut zu versenden :man_facepalming:

Gibt es eine Befehlszeilenaktion, mit der ich all diese löschen kann? Ich vermute, dass diese anstehenden Elemente die Wiederherstellung blockieren.

15

Ich würde einen neuen Droplet starten und Move a Discourse site to another VPS with rsync folgen.
Aber ich schätze, Sie haben Ihr /var/discourse nicht mehr, es sei denn, Sie haben ein weiteres Backup erstellt, bevor Sie das wiederhergestellt haben, das Sie gefunden haben?

Starten Sie also einen neuen Droplet, kopieren Sie Ihre app.yml und stellen Sie das gestrige Backup von S3 wieder her. Das ist, was zu tun ist.

Außerdem, was ist Ihre Docker-Version? Ich hatte einen fehlgeschlagenen Rebuild mit einem Fehler, der so aussieht (ich werde dieses Thema finden, wenn ich an meinem Laptop bin).

1 „Gefällt mir“
16

Ich habe einen vollständigen Server-Snapshot von vor 7 Tagen wiederhergestellt, und mein Discourse läuft jetzt wieder – allerdings mit einer 7 Tage alten Datenbank.

Ich versuche nun, die Datenbank aus dem gestrigen Backup wiederherzustellen, das sich auf S3 befindet.

1 „Gefällt mir“
17

Während dieser ausfällt, starten Sie einen neuen Droplet. (was Sie auch wollen, selbst wenn er wiederhergestellt wird)

1 „Gefällt mir“
18

Danke Jay :slight_smile:

Ich habe alle Sidekiq-Jobs manuell beendet und es sieht so aus, als ob die Datenbankwiederherstellung jetzt läuft:

:crossed_fingers:

Wenn das auch fehlschlägt, ziehe ich den Stecker und gehe Ihren rsync-Weg, danke :blush:

1 „Gefällt mir“
19

Sie möchten ein neues Betriebssystem. Ich traue Upgrades nicht. Wenn Sie die neuere Hardware verwenden möchten, erstellen Sie einen neuen Droplet.

20

Ich bin auch kein Fan :slight_smile:

Das wöchentliche Digital Ocean Backup von vor 7 Tagen, das ich wiederhergestellt habe, hat auch das Ubuntu 18.04 Betriebssystem zurückgesetzt, sodass ich mich jetzt wieder vor dem Upgrade befinde.

Die Datenbankwiederherstellung scheint erfolgreich gewesen zu sein.

Ich mache jetzt noch einen frischen Snapshot – nur für den Fall.

Wenn das alles funktioniert und uns wieder online bringt, kann ich mir ansehen, wie ich einen rsync-Umzug zum neuesten Betriebssystem durchführe – aber in aller Ruhe und nicht in einer überstürzten Panik :see_no_evil:

1 „Gefällt mir“