Rebuild failed - FAILED TO BOOTSTRAP

installation
1

Been a while since this forum was updated, getting this message after a git pull and rebuild:

FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && su discourse -c 'bundle install --retry 3 --jobs 4' failed with return #<Process::Status: pid 502 exit 5>
Location of failure: /usr/local/lib/ruby/gems/3.2.0/gems/pups-1.2.1/lib/pups/exec_command.rb:132:in `spawn'
exec failed with the params {"cd"=>"$home", "hook"=>"bundle_exec", "cmd"=>["su discourse -c 'bundle config --local deployment true'", "su discourse -c 'bundle config --local without \"development test\"'", "su discourse -c 'bundle install --retry 3 --jobs 4'"]}
bootstrap failed with exit code 5
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.

Full output:

2.0.20231218-0429: Pulling from discourse/base
Digest: sha256:468f70b9bb4c6d0c6c2bbb3efc1a5e12d145eae57bdb6946b7fe5558beb52dc1
Status: Image is up to date for discourse/base:2.0.20231218-0429
docker.io/discourse/base:2.0.20231218-0429
/usr/local/lib/ruby/gems/3.2.0/gems/pups-1.2.1/lib/pups.rb
/usr/local/bin/pups --stdin
I, [2023-12-22T05:08:37.278012 #1]  INFO -- : Reading from stdin
I, [2023-12-22T05:08:37.281481 #1]  INFO -- : > locale-gen $LANG && update-locale
I, [2023-12-22T05:08:37.307162 #1]  INFO -- : Generating locales (this might take a while)...
Generation complete.

I, [2023-12-22T05:08:37.307385 #1]  INFO -- : > mkdir -p /shared/postgres_run
I, [2023-12-22T05:08:37.309463 #1]  INFO -- : 
I, [2023-12-22T05:08:37.309725 #1]  INFO -- : > chown postgres:postgres /shared/postgres_run
I, [2023-12-22T05:08:37.311271 #1]  INFO -- : 
I, [2023-12-22T05:08:37.311378 #1]  INFO -- : > chmod 775 /shared/postgres_run
I, [2023-12-22T05:08:37.312650 #1]  INFO -- : 
I, [2023-12-22T05:08:37.312753 #1]  INFO -- : > rm -fr /var/run/postgresql
I, [2023-12-22T05:08:37.314298 #1]  INFO -- : 
I, [2023-12-22T05:08:37.314429 #1]  INFO -- : > ln -s /shared/postgres_run /var/run/postgresql
I, [2023-12-22T05:08:37.315752 #1]  INFO -- : 
I, [2023-12-22T05:08:37.315876 #1]  INFO -- : > socat /dev/null UNIX-CONNECT:/shared/postgres_run/.s.PGSQL.5432 || exit 0 && echo postgres already running stop container ; exit 1
2023/12/22 05:08:37 socat[19] E connect(6, AF=1 "/shared/postgres_run/.s.PGSQL.5432", 36): No such file or directory
I, [2023-12-22T05:08:37.319328 #1]  INFO -- : 
I, [2023-12-22T05:08:37.319428 #1]  INFO -- : > rm -fr /shared/postgres_run/.s*
I, [2023-12-22T05:08:37.321411 #1]  INFO -- : 
I, [2023-12-22T05:08:37.321499 #1]  INFO -- : > rm -fr /shared/postgres_run/*.pid
I, [2023-12-22T05:08:37.323413 #1]  INFO -- : 
I, [2023-12-22T05:08:37.323501 #1]  INFO -- : > mkdir -p /shared/postgres_run/13-main.pg_stat_tmp
I, [2023-12-22T05:08:37.324999 #1]  INFO -- : 
I, [2023-12-22T05:08:37.325134 #1]  INFO -- : > chown postgres:postgres /shared/postgres_run/13-main.pg_stat_tmp
I, [2023-12-22T05:08:37.326603 #1]  INFO -- : 
I, [2023-12-22T05:08:37.329796 #1]  INFO -- : File > /etc/service/postgres/run  chmod: +x  chown: 
I, [2023-12-22T05:08:37.332807 #1]  INFO -- : File > /etc/service/postgres/log/run  chmod: +x  chown: 
I, [2023-12-22T05:08:37.335852 #1]  INFO -- : File > /etc/runit/3.d/99-postgres  chmod: +x  chown: 
I, [2023-12-22T05:08:37.338874 #1]  INFO -- : File > /root/upgrade_postgres  chmod: +x  chown: 
I, [2023-12-22T05:08:37.339061 #1]  INFO -- : > chown -R root /var/lib/postgresql/13/main
I, [2023-12-22T05:08:42.054512 #1]  INFO -- : 
I, [2023-12-22T05:08:42.054728 #1]  INFO -- : > [ ! -e /shared/postgres_data ] && install -d -m 0755 -o postgres -g postgres /shared/postgres_data && sudo -E -u postgres /usr/lib/postgresql/13/bin/initdb -D /shared/postgres_data || exit 0
I, [2023-12-22T05:08:42.057145 #1]  INFO -- : 
I, [2023-12-22T05:08:42.057222 #1]  INFO -- : > chown -R postgres:postgres /shared/postgres_data
I, [2023-12-22T05:08:42.083069 #1]  INFO -- : 
I, [2023-12-22T05:08:42.083263 #1]  INFO -- : > chown -R postgres:postgres /var/run/postgresql
I, [2023-12-22T05:08:42.085856 #1]  INFO -- : 
I, [2023-12-22T05:08:42.086020 #1]  INFO -- : > /root/upgrade_postgres
I, [2023-12-22T05:08:42.090800 #1]  INFO -- : 
I, [2023-12-22T05:08:42.090927 #1]  INFO -- : > rm /root/upgrade_postgres
I, [2023-12-22T05:08:42.093016 #1]  INFO -- : 
I, [2023-12-22T05:08:42.093274 #1]  INFO -- : Replacing data_directory = '/var/lib/postgresql/13/main' with data_directory = '/shared/postgres_data' in /etc/postgresql/13/main/postgresql.conf
I, [2023-12-22T05:08:42.093955 #1]  INFO -- : Replacing (?-mix:#?listen_addresses *=.*) with listen_addresses = '*' in /etc/postgresql/13/main/postgresql.conf
I, [2023-12-22T05:08:42.094372 #1]  INFO -- : Replacing (?-mix:#?synchronous_commit *=.*) with synchronous_commit = $db_synchronous_commit in /etc/postgresql/13/main/postgresql.conf
I, [2023-12-22T05:08:42.094778 #1]  INFO -- : Replacing (?-mix:#?shared_buffers *=.*) with shared_buffers = $db_shared_buffers in /etc/postgresql/13/main/postgresql.conf
I, [2023-12-22T05:08:42.095239 #1]  INFO -- : Replacing (?-mix:#?work_mem *=.*) with work_mem = $db_work_mem in /etc/postgresql/13/main/postgresql.conf
I, [2023-12-22T05:08:42.095631 #1]  INFO -- : Replacing (?-mix:#?default_text_search_config *=.*) with default_text_search_config = '$db_default_text_search_config' in /etc/postgresql/13/main/postgresql.conf
I, [2023-12-22T05:08:42.096023 #1]  INFO -- : > install -d -m 0755 -o postgres -g postgres /shared/postgres_backup
I, [2023-12-22T05:08:42.099445 #1]  INFO -- : 
I, [2023-12-22T05:08:42.099790 #1]  INFO -- : Replacing (?-mix:#?checkpoint_segments *=.*) with checkpoint_segments = $db_checkpoint_segments in /etc/postgresql/13/main/postgresql.conf
I, [2023-12-22T05:08:42.100149 #1]  INFO -- : Replacing (?-mix:#?logging_collector *=.*) with logging_collector = $db_logging_collector in /etc/postgresql/13/main/postgresql.conf
I, [2023-12-22T05:08:42.100622 #1]  INFO -- : Replacing (?-mix:#?log_min_duration_statement *=.*) with log_min_duration_statement = $db_log_min_duration_statement in /etc/postgresql/13/main/postgresql.conf
I, [2023-12-22T05:08:42.101045 #1]  INFO -- : Replacing (?-mix:^#local +replication +postgres +peer$) with local replication postgres  peer in /etc/postgresql/13/main/pg_hba.conf
I, [2023-12-22T05:08:42.101428 #1]  INFO -- : Replacing (?-mix:^host.*all.*all.*127.*$) with host all all 0.0.0.0/0 md5 in /etc/postgresql/13/main/pg_hba.conf
I, [2023-12-22T05:08:42.101741 #1]  INFO -- : Replacing (?-mix:^host.*all.*all.*::1\/128.*$) with host all all ::/0 md5 in /etc/postgresql/13/main/pg_hba.conf
I, [2023-12-22T05:08:42.102004 #1]  INFO -- : > HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main
I, [2023-12-22T05:08:42.104006 #1]  INFO -- : > sleep 5
2023-12-22 05:08:42.198 UTC [42] LOG:  starting PostgreSQL 13.13 (Debian 13.13-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
2023-12-22 05:08:42.198 UTC [42] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2023-12-22 05:08:42.198 UTC [42] LOG:  listening on IPv6 address "::", port 5432
2023-12-22 05:08:42.207 UTC [42] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2023-12-22 05:08:42.218 UTC [45] LOG:  database system was shut down at 2023-12-22 05:06:01 UTC
2023-12-22 05:08:42.269 UTC [42] LOG:  database system is ready to accept connections
I, [2023-12-22T05:08:47.106408 #1]  INFO -- : 
I, [2023-12-22T05:08:47.106576 #1]  INFO -- : > su postgres -c 'createdb discourse' || true
2023-12-22 05:08:47.147 UTC [55] postgres@postgres ERROR:  database "discourse" already exists
2023-12-22 05:08:47.147 UTC [55] postgres@postgres STATEMENT:  CREATE DATABASE discourse;
createdb: error: database creation failed: ERROR:  database "discourse" already exists
I, [2023-12-22T05:08:47.148162 #1]  INFO -- : 
I, [2023-12-22T05:08:47.148325 #1]  INFO -- : > su postgres -c 'psql discourse -c "create user discourse;"' || true
2023-12-22 05:08:47.298 UTC [59] postgres@discourse ERROR:  role "discourse" already exists
2023-12-22 05:08:47.298 UTC [59] postgres@discourse STATEMENT:  create user discourse;
ERROR:  role "discourse" already exists
I, [2023-12-22T05:08:47.303244 #1]  INFO -- : 
I, [2023-12-22T05:08:47.303895 #1]  INFO -- : > su postgres -c 'psql discourse -c "grant all privileges on database discourse to discourse;"' || true
I, [2023-12-22T05:08:47.369029 #1]  INFO -- : GRANT

I, [2023-12-22T05:08:47.369202 #1]  INFO -- : > su postgres -c 'psql discourse -c "alter schema public owner to discourse;"'
I, [2023-12-22T05:08:47.405677 #1]  INFO -- : ALTER SCHEMA

I, [2023-12-22T05:08:47.405889 #1]  INFO -- : > su postgres -c 'psql template1 -c "create extension if not exists hstore;"'
NOTICE:  extension "hstore" already exists, skipping
I, [2023-12-22T05:08:47.446288 #1]  INFO -- : CREATE EXTENSION

I, [2023-12-22T05:08:47.446467 #1]  INFO -- : > su postgres -c 'psql template1 -c "create extension if not exists pg_trgm;"'
NOTICE:  extension "pg_trgm" already exists, skipping
I, [2023-12-22T05:08:47.483049 #1]  INFO -- : CREATE EXTENSION

I, [2023-12-22T05:08:47.483227 #1]  INFO -- : > su postgres -c 'psql template1 -c "create extension if not exists vector;"'
NOTICE:  extension "vector" already exists, skipping
I, [2023-12-22T05:08:47.520869 #1]  INFO -- : CREATE EXTENSION

I, [2023-12-22T05:08:47.521070 #1]  INFO -- : > su postgres -c 'psql discourse -c "create extension if not exists hstore;"'
NOTICE:  extension "hstore" already exists, skipping
I, [2023-12-22T05:08:47.558070 #1]  INFO -- : CREATE EXTENSION

I, [2023-12-22T05:08:47.558241 #1]  INFO -- : > su postgres -c 'psql discourse -c "create extension if not exists pg_trgm;"'
NOTICE:  extension "pg_trgm" already exists, skipping
I, [2023-12-22T05:08:47.595347 #1]  INFO -- : CREATE EXTENSION

I, [2023-12-22T05:08:47.595558 #1]  INFO -- : > su postgres -c 'psql discourse -c "create extension if not exists vector;"'
NOTICE:  extension "vector" already exists, skipping
I, [2023-12-22T05:08:47.631831 #1]  INFO -- : CREATE EXTENSION

I, [2023-12-22T05:08:47.632018 #1]  INFO -- : > sudo -u postgres psql discourse
I, [2023-12-22T05:08:47.633193 #1]  INFO -- : update pg_database set encoding = pg_char_to_encoding('UTF8') where datname = 'discourse' AND encoding = pg_char_to_encoding('SQL_ASCII');

I, [2023-12-22T05:08:47.751019 #1]  INFO -- : File > /var/lib/postgresql/take-database-backup  chmod: +x  chown: postgres:postgres
I, [2023-12-22T05:08:47.758567 #1]  INFO -- : File > /var/spool/cron/crontabs/postgres  chmod:   chown: 
I, [2023-12-22T05:08:47.758737 #1]  INFO -- : > echo postgres installed!
I, [2023-12-22T05:08:47.762674 #1]  INFO -- : postgres installed!

I, [2023-12-22T05:08:47.769427 #1]  INFO -- : File > /etc/service/redis/run  chmod: +x  chown: 
I, [2023-12-22T05:08:47.775270 #1]  INFO -- : File > /etc/service/redis/log/run  chmod: +x  chown: 
I, [2023-12-22T05:08:47.781182 #1]  INFO -- : File > /etc/runit/3.d/10-redis  chmod: +x  chown: 
I, [2023-12-22T05:08:47.781399 #1]  INFO -- : Replacing daemonize yes with  in /etc/redis/redis.conf
I, [2023-12-22T05:08:47.782104 #1]  INFO -- : Replacing (?-mix:^pidfile.*$) with  in /etc/redis/redis.conf
I, [2023-12-22T05:08:47.782611 #1]  INFO -- : > install -d -m 0755 -o redis -g redis /shared/redis_data
I, [2023-12-22T05:08:47.785928 #1]  INFO -- : 
I, [2023-12-22T05:08:47.786315 #1]  INFO -- : Replacing (?-mix:^logfile.*$) with logfile "" in /etc/redis/redis.conf
I, [2023-12-22T05:08:47.786883 #1]  INFO -- : Replacing (?-mix:^bind .*$) with  in /etc/redis/redis.conf
I, [2023-12-22T05:08:47.787440 #1]  INFO -- : Replacing (?-mix:^dir .*$) with dir /shared/redis_data in /etc/redis/redis.conf
I, [2023-12-22T05:08:47.788017 #1]  INFO -- : Replacing (?-mix:^protected-mode yes) with protected-mode no in /etc/redis/redis.conf
I, [2023-12-22T05:08:47.788528 #1]  INFO -- : Replacing # io-threads 4 with io-threads $redis_io_threads in /etc/redis/redis.conf
I, [2023-12-22T05:08:47.789008 #1]  INFO -- : > echo redis installed
I, [2023-12-22T05:08:47.791574 #1]  INFO -- : redis installed

I, [2023-12-22T05:08:47.791751 #1]  INFO -- : > cat /etc/redis/redis.conf | grep logfile
I, [2023-12-22T05:08:47.796186 #1]  INFO -- : logfile ""

I, [2023-12-22T05:08:47.796457 #1]  INFO -- : > exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf
I, [2023-12-22T05:08:47.798054 #1]  INFO -- : > sleep 10
111:C 22 Dec 2023 05:08:47.806 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
111:C 22 Dec 2023 05:08:47.806 # Redis version=7.0.7, bits=64, commit=00000000, modified=0, pid=111, just started
111:C 22 Dec 2023 05:08:47.806 # Configuration loaded
111:M 22 Dec 2023 05:08:47.807 * monotonic clock: POSIX clock_gettime
111:M 22 Dec 2023 05:08:47.808 * Running mode=standalone, port=6379.
111:M 22 Dec 2023 05:08:47.808 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
111:M 22 Dec 2023 05:08:47.808 # Server initialized
111:M 22 Dec 2023 05:08:47.808 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
111:M 22 Dec 2023 05:08:47.808 * Loading RDB produced by version 7.0.7
111:M 22 Dec 2023 05:08:47.808 * RDB age 167 seconds
111:M 22 Dec 2023 05:08:47.808 * RDB memory usage when created 14.97 Mb
111:M 22 Dec 2023 05:08:47.844 * Done loading RDB, keys loaded: 5831, keys expired: 20.
111:M 22 Dec 2023 05:08:47.844 * DB loaded from disk: 0.035 seconds
111:M 22 Dec 2023 05:08:47.844 * Ready to accept connections
I, [2023-12-22T05:08:57.800809 #1]  INFO -- : 
I, [2023-12-22T05:08:57.801346 #1]  INFO -- : > thpoff echo "thpoff is installed!"
I, [2023-12-22T05:08:57.810768 #1]  INFO -- : thpoff is installed!

I, [2023-12-22T05:08:57.811280 #1]  INFO -- : > /usr/local/bin/ruby -e 'if ENV["DISCOURSE_SMTP_ADDRESS"] == "smtp.example.com"; puts "Aborting! Mail is not configured!"; exit 1; end'
I, [2023-12-22T05:08:57.891116 #1]  INFO -- : 
I, [2023-12-22T05:08:57.891326 #1]  INFO -- : > /usr/local/bin/ruby -e 'if ENV["DISCOURSE_HOSTNAME"] == "discourse.example.com"; puts "Aborting! Domain is not configured!"; exit 1; end'
I, [2023-12-22T05:08:57.940738 #1]  INFO -- : 
I, [2023-12-22T05:08:57.940927 #1]  INFO -- : > /usr/local/bin/ruby -e 'if (ENV["DISCOURSE_CDN_URL"] || "")[0..1] == "//"; puts "Aborting! CDN must have a protocol specified. Once fixed you should rebake your posts now to correct all posts."; exit 1; end'
I, [2023-12-22T05:08:57.991332 #1]  INFO -- : 
I, [2023-12-22T05:08:57.991527 #1]  INFO -- : > rm -f /etc/cron.d/anacron
I, [2023-12-22T05:08:57.993535 #1]  INFO -- : 
I, [2023-12-22T05:08:57.995477 #1]  INFO -- : File > /etc/cron.d/anacron  chmod:   chown: 
I, [2023-12-22T05:08:57.998662 #1]  INFO -- : File > /etc/runit/1.d/copy-env  chmod: +x  chown: 
I, [2023-12-22T05:08:58.001978 #1]  INFO -- : File > /etc/service/unicorn/run  chmod: +x  chown: 
I, [2023-12-22T05:08:58.005133 #1]  INFO -- : File > /etc/service/nginx/run  chmod: +x  chown: 
I, [2023-12-22T05:08:58.008321 #1]  INFO -- : File > /etc/runit/3.d/01-nginx  chmod: +x  chown: 
I, [2023-12-22T05:08:58.011430 #1]  INFO -- : File > /etc/runit/3.d/02-unicorn  chmod: +x  chown: 
I, [2023-12-22T05:08:58.011548 #1]  INFO -- : Replacing # postgres with sv start postgres || exit 1 in /etc/service/unicorn/run
I, [2023-12-22T05:08:58.011719 #1]  INFO -- : > exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf
I, [2023-12-22T05:08:58.013090 #1]  INFO -- : > cd /var/www/discourse && sudo -H -E -u discourse git reset --hard
138:C 22 Dec 2023 05:08:58.017 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
138:C 22 Dec 2023 05:08:58.017 # Redis version=7.0.7, bits=64, commit=00000000, modified=0, pid=138, just started
138:C 22 Dec 2023 05:08:58.017 # Configuration loaded
138:M 22 Dec 2023 05:08:58.017 * monotonic clock: POSIX clock_gettime
138:M 22 Dec 2023 05:08:58.017 # Warning: Could not create server TCP listening socket *:6379: bind: Address already in use
138:M 22 Dec 2023 05:08:58.017 # Failed listening on port 6379 (TCP), aborting.
Updating files: 100% (33715/33715), done.
I, [2023-12-22T05:09:00.540189 #1]  INFO -- : HEAD is now at 6ab1a19e9 DEV: Convert min_trust_level_to_allow_invite to groups (#24893)

I, [2023-12-22T05:09:00.540340 #1]  INFO -- : > cd /var/www/discourse && sudo -H -E -u discourse git clean -f
I, [2023-12-22T05:09:00.642416 #1]  INFO -- : 
I, [2023-12-22T05:09:00.642594 #1]  INFO -- : > cd /var/www/discourse && sudo -H -E -u discourse bash -c '
  set -o errexit
  if [ $(git rev-parse --is-shallow-repository) == "true" ]; then
      git remote set-branches --add origin main
      git remote set-branches origin tests-passed
      git fetch --depth 1 origin tests-passed
  else
      git fetch --tags --prune-tags --prune --force origin
  fi
'
From https://github.com/discourse/discourse
 - [deleted]             (none)     -> origin/0-drop-code
 - [deleted]             (none)     -> origin/dependabot/bundler/excon-0.108.0
 - [deleted]             (none)     -> origin/dependabot/bundler/hashdiff-1.1.0
 - [deleted]             (none)     -> origin/dependabot/bundler/parallel-1.24.0
 - [deleted]             (none)     -> origin/dependabot/bundler/rubocop-discourse-3.6.0
 - [deleted]             (none)     -> origin/dev/allow-chat-mentions-to-have-several-notifications
 - [deleted]             (none)     -> origin/dev/convert-min-trust-level-to-ignore-to-groups
 - [deleted]             (none)     -> origin/fix-modal-flash-minor
 - [deleted]             (none)     -> origin/global-notice-reactive
 - [deleted]             (none)     -> origin/table-builder-modal-modifier
 - [deleted]             (none)     -> origin/wip_categories
 - [deleted]             (none)     -> origin/wizard-split
   6ab1a19e9..d3625f228  main                 -> origin/main
 * [new branch]          0-extend-native      -> origin/0-extend-native
 * [new branch]          dependabot/bundler/faraday-2.8.1 -> origin/dependabot/bundler/faraday-2.8.1
 * [new branch]          dependabot/npm_and_yarn/app/assets/javascripts/glimmer/syntax-0.86.0 -> origin/dependabot/npm_and_yarn/app/assets/javascripts/glimmer/syntax-0.86.0
 * [new branch]          dev/add-group-tl-auto-map-in-deprecations -> origin/dev/add-group-tl-auto-map-in-deprecations
 + 236458e7c...b0fb985bf dev/add-new-chat-metrics -> origin/dev/add-new-chat-metrics  (forced update)
 + 332564610...c89fecfc8 dev/convert-min-trust-level-to-create-tag-to-group -> origin/dev/convert-min-trust-level-to-create-tag-to-group  (forced update)
 * [new branch]          dev/convert-min-trust-to-allow-self-wiki-to-groups -> origin/dev/convert-min-trust-to-allow-self-wiki-to-groups
 * [new branch]          dev/convert-min-trust-to-send-email-messages-to-groups -> origin/dev/convert-min-trust-to-send-email-messages-to-groups
 + 7c338d0dc...1c8f343ff dev/rethink-mass-mentions-polymorphic -> origin/dev/rethink-mass-mentions-polymorphic  (forced update)
 * [new branch]          dev_no_preloaded_categories -> origin/dev_no_preloaded_categories
 * [new branch]          feature/paste-upload-on-first-upload-input-templates -> origin/feature/paste-upload-on-first-upload-input-templates
 * [new branch]          group_activity_posts_load_more -> origin/group_activity_posts_load_more
 * [new branch]          mobile-chat-redesign-header-updates -> origin/mobile-chat-redesign-header-updates
 * [new branch]          okay-proper-sm-fix   -> origin/okay-proper-sm-fix
 * [new branch]          pm-bubble-adjustment -> origin/pm-bubble-adjustment
   d7f7f79b6..69c095daf  stable               -> origin/stable
 * [new branch]          svg-ids              -> origin/svg-ids
   6ab1a19e9..d3625f228  tests-passed         -> origin/tests-passed
I, [2023-12-22T05:09:01.692449 #1]  INFO -- : 
I, [2023-12-22T05:09:01.692555 #1]  INFO -- : > cd /var/www/discourse && sudo -H -E -u discourse bash -c '
  set -o errexit
  if [[ $(git symbolic-ref --short HEAD) == tests-passed ]] ; then
      git pull
  else
      git -c advice.detachedHead=false checkout tests-passed
  fi
'
Switched to a new branch 'tests-passed'
I, [2023-12-22T05:09:03.953576 #1]  INFO -- : Branch 'tests-passed' set up to track remote branch 'tests-passed' from 'origin'.

I, [2023-12-22T05:09:03.953744 #1]  INFO -- : > cd /var/www/discourse && sudo -H -E -u discourse git config user.discourse-version tests-passed
I, [2023-12-22T05:09:03.961776 #1]  INFO -- : 
I, [2023-12-22T05:09:03.961833 #1]  INFO -- : > cd /var/www/discourse && mkdir -p tmp
I, [2023-12-22T05:09:03.964586 #1]  INFO -- : 
I, [2023-12-22T05:09:03.964723 #1]  INFO -- : > cd /var/www/discourse && chown discourse:www-data tmp
I, [2023-12-22T05:09:03.966979 #1]  INFO -- : 
I, [2023-12-22T05:09:03.967019 #1]  INFO -- : > cd /var/www/discourse && mkdir -p tmp/pids
I, [2023-12-22T05:09:03.969109 #1]  INFO -- : 
I, [2023-12-22T05:09:03.969183 #1]  INFO -- : > cd /var/www/discourse && mkdir -p tmp/sockets
I, [2023-12-22T05:09:03.971238 #1]  INFO -- : 
I, [2023-12-22T05:09:03.971359 #1]  INFO -- : > cd /var/www/discourse && touch tmp/.gitkeep
I, [2023-12-22T05:09:03.973173 #1]  INFO -- : 
I, [2023-12-22T05:09:03.973322 #1]  INFO -- : > cd /var/www/discourse && mkdir -p                    /shared/log/rails
I, [2023-12-22T05:09:03.975472 #1]  INFO -- : 
I, [2023-12-22T05:09:03.975517 #1]  INFO -- : > cd /var/www/discourse && bash -c "touch -a           /shared/log/rails/{production,production_errors,unicorn.stdout,unicorn.stderr,sidekiq}.log"
I, [2023-12-22T05:09:03.978876 #1]  INFO -- : 
I, [2023-12-22T05:09:03.978969 #1]  INFO -- : > cd /var/www/discourse && bash -c "ln    -s           /shared/log/rails/{production,production_errors,unicorn.stdout,unicorn.stderr,sidekiq}.log /var/www/discourse/log"
I, [2023-12-22T05:09:03.982494 #1]  INFO -- : 
I, [2023-12-22T05:09:03.982655 #1]  INFO -- : > cd /var/www/discourse && bash -c "mkdir -p           /shared/{uploads,backups}"
I, [2023-12-22T05:09:03.986479 #1]  INFO -- : 
I, [2023-12-22T05:09:03.986541 #1]  INFO -- : > cd /var/www/discourse && bash -c "ln    -s           /shared/{uploads,backups} /var/www/discourse/public"
I, [2023-12-22T05:09:03.990088 #1]  INFO -- : 
I, [2023-12-22T05:09:03.990173 #1]  INFO -- : > cd /var/www/discourse && bash -c "mkdir -p           /shared/tmp/{backups,restores}"
I, [2023-12-22T05:09:03.994222 #1]  INFO -- : 
I, [2023-12-22T05:09:03.994308 #1]  INFO -- : > cd /var/www/discourse && bash -c "ln    -s           /shared/tmp/{backups,restores} /var/www/discourse/tmp"
I, [2023-12-22T05:09:03.997951 #1]  INFO -- : 
I, [2023-12-22T05:09:03.998015 #1]  INFO -- : > cd /var/www/discourse && chown -R discourse:www-data /shared/log/rails /shared/uploads /shared/backups /shared/tmp
I, [2023-12-22T05:09:04.343687 #1]  INFO -- : 
I, [2023-12-22T05:09:04.343754 #1]  INFO -- : > cd /var/www/discourse && [ ! -d public/plugins ] || find public/plugins/ -maxdepth 1 -xtype l -delete
I, [2023-12-22T05:09:04.345501 #1]  INFO -- : 
I, [2023-12-22T05:09:04.345585 #1]  INFO -- : Replacing # redis with sv start redis || exit 1 in /etc/service/unicorn/run
I, [2023-12-22T05:09:04.345981 #1]  INFO -- : > cd /var/www/discourse/plugins && git clone https://github.com/discourse/docker_manager.git
Cloning into 'docker_manager'...
I, [2023-12-22T05:09:05.467693 #1]  INFO -- : 
I, [2023-12-22T05:09:05.467867 #1]  INFO -- : > cd /var/www/discourse/plugins && git clone https://github.com/discourse/discourse-solved.git
Cloning into 'discourse-solved'...
I, [2023-12-22T05:09:06.107093 #1]  INFO -- : 
I, [2023-12-22T05:09:06.107478 #1]  INFO -- : > cd /var/www/discourse/plugins && git clone https://github.com/discourse/discourse-data-explorer.git
Cloning into 'discourse-data-explorer'...
I, [2023-12-22T05:09:06.760695 #1]  INFO -- : 
I, [2023-12-22T05:09:06.897384 #1]  INFO -- : 
Cloning into 'discourse-setup'...
I, [2023-12-22T05:09:07.012410 #1]  INFO -- : 
I, [2023-12-22T05:09:07.012665 #1]  INFO -- : > cp /var/www/discourse/config/nginx.sample.conf /etc/nginx/conf.d/discourse.conf
I, [2023-12-22T05:09:07.015045 #1]  INFO -- : 
I, [2023-12-22T05:09:07.015152 #1]  INFO -- : > rm /etc/nginx/sites-enabled/default
I, [2023-12-22T05:09:07.016662 #1]  INFO -- : 
I, [2023-12-22T05:09:07.016728 #1]  INFO -- : > mkdir -p /var/nginx/cache
I, [2023-12-22T05:09:07.018369 #1]  INFO -- : 
I, [2023-12-22T05:09:07.018500 #1]  INFO -- : Replacing pid /run/nginx.pid; with daemon off; in /etc/nginx/nginx.conf
I, [2023-12-22T05:09:07.022473 #1]  INFO -- : Replacing (?m-ix:upstream[^\}]+\}) with upstream discourse { server 127.0.0.1:3000; } in /etc/nginx/conf.d/discourse.conf
I, [2023-12-22T05:09:07.022686 #1]  INFO -- : Replacing (?-mix:server_name.+$) with server_name _ ; in /etc/nginx/conf.d/discourse.conf
I, [2023-12-22T05:09:07.022868 #1]  INFO -- : Replacing (?-mix:client_max_body_size.+$) with client_max_body_size $upload_size ; in /etc/nginx/conf.d/discourse.conf
I, [2023-12-22T05:09:07.023045 #1]  INFO -- : Replacing (?-mix:worker_connections.+$) with worker_connections $nginx_worker_connections ; in /etc/nginx/nginx.conf
I, [2023-12-22T05:09:07.023215 #1]  INFO -- : > echo "done configuring web"
I, [2023-12-22T05:09:07.024506 #1]  INFO -- : done configuring web

I, [2023-12-22T05:09:07.024713 #1]  INFO -- : > cd /var/www/discourse && gem install bundler --conservative -v $(awk '/BUNDLED WITH/ { getline; gsub(/ /,""); print $0 }' Gemfile.lock)
I, [2023-12-22T05:09:07.169702 #1]  INFO -- : 
I, [2023-12-22T05:09:07.169855 #1]  INFO -- : > cd /var/www/discourse && find /var/www/discourse ! -user discourse -exec chown discourse {} \+
I, [2023-12-22T05:09:08.273126 #1]  INFO -- : 
I, [2023-12-22T05:09:08.273460 #1]  INFO -- : > cd /var/www/discourse && if [ "tests-passed" != "tests-passed" ]; then
  rm -rf app/assets/javascripts/node_modules
fi
I, [2023-12-22T05:09:08.275309 #1]  INFO -- : 
I, [2023-12-22T05:09:08.275360 #1]  INFO -- : > cd /var/www/discourse && su discourse -c 'yarn install --frozen-lockfile && yarn cache clean'
warning Resolution field "unset-value@2.0.1" is incompatible with requested version "unset-value@^1.0.0"
warning Pattern ["wrap-ansi@^7.0.0"] is trying to unpack in the same destination "/home/discourse/.cache/yarn/v6/npm-wrap-ansi-cjs-7.0.0-67e145cff510a6a6984bdf1152911d69d2eb9e43-integrity/node_modules/wrap-ansi-cjs" as pattern ["wrap-ansi-cjs@npm:wrap-ansi@^7.0.0"]. This could result in non-deterministic behavior, skipping.
warning " > discourse-markdown-it@1.0.0" has unmet peer dependency "xss@*".
warning "workspace-aggregator-1f6f037a-c78a-4d1d-ab71-c37fae45074d > discourse > @ember/legacy-built-in-components@0.5.0" has incorrect peer dependency "ember-source@>= 4.8".
warning "workspace-aggregator-1f6f037a-c78a-4d1d-ab71-c37fae45074d > discourse > @uppy/aws-s3@3.0.6" has incorrect peer dependency "@uppy/core@^3.1.2".
warning "workspace-aggregator-1f6f037a-c78a-4d1d-ab71-c37fae45074d > discourse > @uppy/aws-s3-multipart@3.1.3" has incorrect peer dependency "@uppy/core@^3.1.2".
warning "workspace-aggregator-1f6f037a-c78a-4d1d-ab71-c37fae45074d > discourse > @uppy/xhr-upload@3.1.1" has incorrect peer dependency "@uppy/core@^3.1.2".
warning "workspace-aggregator-1f6f037a-c78a-4d1d-ab71-c37fae45074d > discourse-plugins > ember-this-fallback@0.4.0" has unmet peer dependency "ember-source@^3.28.11 || ^4.0.0".
warning "workspace-aggregator-1f6f037a-c78a-4d1d-ab71-c37fae45074d > discourse > @uppy/aws-s3 > @uppy/xhr-upload@3.3.0" has incorrect peer dependency "@uppy/core@^3.2.1".
I, [2023-12-22T05:09:48.206439 #1]  INFO -- : yarn install v1.22.19
[1/5] Validating package.json...
[2/5] Resolving packages...
success Already up-to-date.
$ yarn --cwd app/assets/javascripts $(node -e 'const argv = JSON.parse(process.env.npm_config_argv).original; const passthrough = [`--frozen-lockfile`, `-s`].filter(arg => argv.includes(arg)); console.log(passthrough.join(` `));')
yarn install v1.22.19
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
$ ./run-patch-package
patch-package 8.0.0
Applying patches...
babel-plugin-debug-macros@0.3.4 ✔
content-tag@1.2.2 ✔
ember-cli@5.0.0 ✔
ember-this-fallback@0.4.0 (1 deprecation-name) ✔
ember-this-fallback@0.4.0 (2 themes) ✔
virtual-dom@2.1.1 ✔
Done in 36.46s.
Done in 36.94s.
yarn cache v1.22.19
success Cleared cache.
Done in 2.67s.

I, [2023-12-22T05:09:48.206835 #1]  INFO -- : > cd /var/www/discourse && su discourse -c 'bundle config --local deployment true'
I, [2023-12-22T05:09:48.330084 #1]  INFO -- : 
I, [2023-12-22T05:09:48.330188 #1]  INFO -- : > cd /var/www/discourse && su discourse -c 'bundle config --local without "development test"'
I, [2023-12-22T05:09:48.456606 #1]  INFO -- : You are replacing the current local value of without, which is currently "test:development"

I, [2023-12-22T05:09:48.456818 #1]  INFO -- : > cd /var/www/discourse && su discourse -c 'bundle install --retry 3 --jobs 4'
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

current directory:
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rbtrace-0.5.1/ext
/usr/local/bin/ruby extconf.rb
  -- tar zxvfo msgpack-1.1.0.tar.gz
  -- env CFLAGS=nil LDFLAGS=nil CC=nil
-- ./configure --disable-dependency-tracking --disable-shared --with-pic
--prefix=/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rbtrace-0.5.1/ext/dst/
--libdir=/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rbtrace-0.5.1/ext/dst/lib
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers.  Check the mkmf.log file for more details.  You may
need configuration options.

Provided configuration options:
	--with-opt-dir
	--without-opt-dir
	--with-opt-include
	--without-opt-include=${opt-dir}/include
	--with-opt-lib
	--without-opt-lib=${opt-dir}/lib
	--with-make-prog
	--without-make-prog
	--srcdir=.
	--curdir
	--ruby=/usr/local/bin/$(RUBY_BASE_NAME)
extconf.rb:6:in `sys': ./configure --disable-dependency-tracking
--disable-shared --with-pic
--prefix=/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rbtrace-0.5.1/ext/dst/
--libdir=/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rbtrace-0.5.1/ext/dst/lib
failed, please report to https://github.com/tmm1/rbtrace/issues (RuntimeError)
	from extconf.rb:42:in `block (2 levels) in <main>'
	from extconf.rb:34:in `chdir'
	from extconf.rb:34:in `block in <main>'
	from extconf.rb:30:in `chdir'
	from extconf.rb:30:in `<main>'

To see why this extension failed to compile, please check the mkmf.log which can
be found here:

/var/www/discourse/vendor/bundle/ruby/3.2.0/extensions/x86_64-linux/3.2.0/rbtrace-0.5.1/mkmf.log

extconf failed, exit code 1

Gem files will remain installed in
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rbtrace-0.5.1 for inspection.
Results logged to
/var/www/discourse/vendor/bundle/ruby/3.2.0/extensions/x86_64-linux/3.2.0/rbtrace-0.5.1/gem_make.out

  /usr/local/lib/ruby/site_ruby/3.2.0/rubygems/ext/builder.rb:125:in `run'
/usr/local/lib/ruby/site_ruby/3.2.0/rubygems/ext/ext_conf_builder.rb:28:in
`build'
/usr/local/lib/ruby/site_ruby/3.2.0/rubygems/ext/builder.rb:193:in
`build_extension'
/usr/local/lib/ruby/site_ruby/3.2.0/rubygems/ext/builder.rb:227:in `block in
build_extensions'
  /usr/local/lib/ruby/site_ruby/3.2.0/rubygems/ext/builder.rb:224:in `each'
/usr/local/lib/ruby/site_ruby/3.2.0/rubygems/ext/builder.rb:224:in
`build_extensions'
/usr/local/lib/ruby/site_ruby/3.2.0/rubygems/installer.rb:852:in
`build_extensions'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/rubygems_gem_installer.rb:72:in
`build_extensions'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/rubygems_gem_installer.rb:28:in
`install'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/source/rubygems.rb:198:in
`install'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/installer/gem_installer.rb:54:in
`install'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/installer/gem_installer.rb:16:in
`install_from_spec'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/installer/parallel_installer.rb:156:in
`do_install'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/installer/parallel_installer.rb:147:in
`block in worker_pool'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/worker.rb:62:in
`apply_func'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/worker.rb:57:in
`block in process_queue'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/worker.rb:54:in
`loop'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/worker.rb:54:in
`process_queue'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/worker.rb:90:in
`block (2 levels) in create_threads'

An error occurred while installing rbtrace (0.5.1), and Bundler cannot continue.

In Gemfile:
  rbtrace
I, [2023-12-22T05:09:50.952500 #1]  INFO -- : Fetching gem metadata from https://rubygems.org/.........
Using rake 13.1.0
Using minitest 5.20.0
Using builder 3.2.4
Using erubi 1.12.0
Using racc 1.7.3
Using concurrent-ruby 1.2.2
Using crass 1.0.6
Using rack 2.2.8
Using date 3.3.4
Using timeout 0.4.1
Using mini_mime 1.1.5
Using public_suffix 5.0.4
Using aws-eventstream 1.3.0
Using aws-partitions 1.583.0
Using jmespath 1.6.2
Using base64 0.2.0
Using bigdecimal 3.1.5
Using msgpack 1.7.2
Using bundler 2.4.13
Using cbor 0.5.9.6
Using cgi 0.4.1
Using chunky_png 1.4.0
Using coderay 1.1.3
Using colored2 4.0.0
Using byebug 11.1.3
Using connection_pool 2.4.1
Using openssl 3.2.0
Using cppjieba_rb 0.4.2
Using diffy 3.4.2
Using digest 3.1.1
Using discourse-fonts 0.0.9
Using email_reply_trimmer 0.1.13
Fetching excon 0.108.0
Using execjs 2.9.1
Using exifr 1.4.0
Using faraday-net_http 3.0.2
Using ruby2_keywords 0.0.5
Using fast_blank 1.0.1
Using fast_xs 0.8.0
Using fastimage 2.2.7
Using fspath 3.1.2
Using ffi 1.16.3
Using gc_tracer 1.5.1
Using google-protobuf 3.25.1 (x86_64-linux)
Using guess_html_encoding 0.0.11
Using hana 1.3.7
Using hashie 5.0.0
Using highline 2.1.0
Using htmlentities 4.3.4
Using http_accept_language 2.1.1
Using image_size 3.3.0
Using in_threads 1.6.0
Using json 2.7.1
Using progress 3.6.0
Using regexp_parser 2.8.3
Using unf_ext 0.0.9.1
Using jwt 2.7.1
Using kgio 2.11.4
Using libv8-node 18.16.0.0 (x86_64-linux)
Using method_source 1.0.0
Using thor 1.3.0
Using zeitwerk 2.6.12
Using logster 2.13.1
Using lru_redux 1.1.0
Using lz4-ruby 0.3.3
Using maxminddb 0.1.22
Using logstash-event 1.2.02
Using redis 4.8.1
Using mini_sql 1.5.0
Using multi_json 1.15.0
Using memory_profiler 1.0.1
Using mustache 1.1.1
Using uri 0.13.0
Using nio4r 2.7.0
Using version_gem 1.1.3
Using optimist 3.1.0
Using multi_xml 0.6.0
Using pg 1.5.4
Using raindrops 0.20.1
Using rchardet 1.8.0
Using rexml 3.2.6
Using rinku 2.0.6
Using rotp 6.3.0
Using rqrcode_core 1.2.0
Using sshkey 3.0.0
Using stackprof 0.2.25
Using rubyzip 2.3.2
Using nokogiri 1.15.5 (x86_64-linux)
Using rack-test 2.1.0
Using rack-mini-profiler 3.3.0
Using rack-protection 3.1.0
Using net-protocol 0.2.2
Using request_store 1.5.1
Using message_bus 4.3.8
Using addressable 2.8.6
Using i18n 1.14.1
Using tzinfo 2.0.6
Using sprockets 3.7.2 from https://github.com/rails/sprockets (at 3.x@f4d3dae)
Using oj 3.16.3
Using bootsnap 1.17.0
Using openssl-signature_algorithm 1.3.0
Fetching faraday 2.8.0
Using aws-sigv4 1.5.0
Using uglifier 4.2.0
Using mini_suffix 0.3.3
Using omniauth 1.9.2
Using sass-embedded 1.69.5 (x86_64-linux-gnu)
Using image_optim 0.31.3
Using unf 0.1.4
Using mini_racer 0.8.0
Using pry 0.14.2
Using web-push 3.0.1
Using logstash-logger 0.26.1
Using sidekiq 6.5.12
Using redis-namespace 1.11.0
Using net-http 0.4.0
Using oauth-tty 1.0.5
Using snaky_hash 2.0.1
Using puma 6.4.0
Fetching rbtrace 0.5.1
Using unicorn 6.1.0
Using rqrcode 2.2.0
Using rss 0.3.0
Using loofah 2.22.0
Using net-imap 0.4.8
Using net-pop 0.1.2
Using net-smtp 0.4.0
Using ruby-readability 0.7.0
Using sanitize 6.1.0
Using activesupport 7.0.7
Using cose 1.3.0
Using css_parser 1.16.0
Using tzinfo-data 1.2023.3
Using aws-sdk-core 3.130.2
Using sassc-embedded 1.68.6
Using simpleidn 0.2.1
Using rtlcss 0.2.1
Using pry-byebug 3.10.1
Using pry-rails 0.3.9
Using mini_scheduler 0.16.0
Using oauth 1.1.0
Using rails-html-sanitizer 1.6.0
Using rails-dom-testing 2.2.0
Using globalid 1.2.1
Using mail 2.8.1
Using activemodel 7.0.7
Using aws-sdk-kms 1.56.0
Using aws-sdk-sns 1.53.0
Using json_schemer 2.1.1
Using omniauth-oauth 1.2.0
Using actionview 7.0.7
Using activejob 7.0.7
Using active_model_serializers 0.8.4
Using activerecord 7.0.7
Using aws-sdk-s3 1.114.0
Using actionpack 7.0.7
Using actionview_precompiler 0.3.0
Using omniauth-twitter 1.4.0
Using discourse-seed-fu 2.3.12
Using actionmailer 7.0.7
Using railties 7.0.7
Using sprockets-rails 3.4.2
Using lograge 0.14.0
Using rails_failover 2.0.1
Using rails_multisite 5.0.0
Installing faraday 2.8.0
Installing excon 0.108.0
Installing rbtrace 0.5.1 with native extensions
Using faraday-retry 2.2.0
Using oauth2 1.4.11
Using omniauth-oauth2 1.7.3
Using omniauth-facebook 9.0.0
Using omniauth-github 1.4.0
Using omniauth-google-oauth2 0.8.2

I, [2023-12-22T05:09:50.952826 #1]  INFO -- : Terminating async processes
I, [2023-12-22T05:09:50.952888 #1]  INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main pid: 42
I, [2023-12-22T05:09:50.952921 #1]  INFO -- : Sending TERM to exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf pid: 111
2023-12-22 05:09:50.952 UTC [42] LOG:  received fast shutdown request
111:signal-handler (1703221790) Received SIGTERM scheduling shutdown...
111:M 22 Dec 2023 05:09:50.981 # User requested shutdown...
111:M 22 Dec 2023 05:09:50.981 * Saving the final RDB snapshot before exiting.
2023-12-22 05:09:51.360 UTC [42] LOG:  aborting any active transactions
2023-12-22 05:09:51.361 UTC [42] LOG:  background worker "logical replication launcher" (PID 51) exited with exit code 1
2023-12-22 05:09:51.362 UTC [46] LOG:  shutting down
111:M 22 Dec 2023 05:09:51.714 * DB saved on disk
111:M 22 Dec 2023 05:09:51.714 # Redis is now ready to exit, bye bye...
2023-12-22 05:09:52.503 UTC [42] LOG:  database system is shut down

Looks like the error is:

I, [2023-12-22T05:08:47.106408 #1]  INFO -- : 
I, [2023-12-22T05:08:47.106576 #1]  INFO -- : > su postgres -c 'createdb discourse' || true
2023-12-22 05:08:47.147 UTC [55] postgres@postgres ERROR:  database "discourse" already exists
2023-12-22 05:08:47.147 UTC [55] postgres@postgres STATEMENT:  CREATE DATABASE discourse;
createdb: error: database creation failed: ERROR:  database "discourse" already exists
I, [2023-12-22T05:08:47.148162 #1]  INFO -- : 
I, [2023-12-22T05:08:47.148325 #1]  INFO -- : > su postgres -c 'psql discourse -c "create user discourse;"' || true
2023-12-22 05:08:47.298 UTC [59] postgres@discourse ERROR:  role "discourse" already exists
2023-12-22 05:08:47.298 UTC [59] postgres@discourse STATEMENT:  create user discourse;
ERROR:  role "discourse" already exists

Am I missing something?

2

I seem to recall something similar happened with the PG12 update, and I’m not sure now, but I had to ./launcher stop app1/2/3 (all the other apps on the server as we run multiple containers) as that’s the only way the upgrade worked - but I’m not sure now.

Think I posted in the upgrade topic but all replies were auto-pruned - could a mod go through my replies there to see if that was the case then too please? (I did try stopping all other containers again, but it still doesn’t work unfortunately).

3

Nope, that’s perfectly normal!

Here’s the death:

2 Likes
4

From another topic on this exact issue:

2 Likes
5

Thanks, I saw that and wondered why it would fail since Discourse control the environment via docker - didn’t think an old docker install would cause the issue (wonder if it’s possible for Discourse to check docker version before attempting updates?)

So should a docker upgrade work? Or best to do what Jay did and move to a new server? I was actually planning on moving the sites to a new server but usually update the forums on the older server first…

6

You’d have to ask Jay, as I’m not sure where that ended.

1 Like
7

Any thoughts @pfaffman?

Or do any of the Discourse team have any thoughts on the best way to approach this?

9

The last one I fixed ended up being an issue with a broken index.

I’m mostly on vacation until the first of the year. I am not likely to have free time until after the first.

If it’s an emergency and you have a budget you can post in marketplace or contact me directly.

2 Likes
10

Thanks Jay. The forum is up (via ./launcher start app) so there’s no immediate rush.

I seem to recall having broken indexes before so will look through my old threads (though I’m sure that was highlighted by a related error message). I’m also fairly sure @sam said they were going to build in checks for the same thing not happening again in future (but I might be wrong).

How did you find the index was broken? Just by trying to reindex the schema?

1 Like
11

Can you please share your docker version, OS version and kernel version?

We have been getting a fair share of failures like this, but I’ve been unable to reproduce it in my droplets.

2 Likes
12

Docker version is 20.10.3 build 48d30b5, and the server is running CentOS 7, which doesn’t reach EOL until June next year but the kernel version (3.10.0-862.11.6.el7.x86_64) is one that doesn’t pass your checks as per this thread - though commenting the check out as per that thread has worked fine previously (even as recently as 3.2.0.beta4-dev (cdbe0f74e8) on a different forum on the same server). The forum with this issue is currently on 3.1.0.beta2 (I haven’t tried to upgrade the other forums which are on 3.2.0.beta4-dev yet).

The kernel is actually part of the reason why we’re moving to a new server, so shall I take a back up of this forum as it is (3.1.0.beta2) and restore that into a new Discourse install to see if that works? And then, if it’d be any help to you, come back to the old server and either upgrade Docker or the kernel to see whether the upgrade will work then?

Edit:

One thing I did notice was when I went to the admin/upgrade panel it took ages to refresh versions, then when I went to click on upgrade docker all I got was a blank screen (that’s when I tried ./launcher rebuild.

I’ve got another old discourse forum on this server and the same thing is happening here. This screengrab might be of help - it can’t find some docker files:

Screenshot 2023-12-22 at 23.46.12
Screenshot 2023-12-22 at 23.46.121920×1364 184 KB

Edit 2:

Doing the same on a more recent forum (on same server) I don’t get the same errors when I go to /upgrade > upgrade docker (just Failed to register Service Worker: TypeError: NetworkError: Load failed in the sources tab of the browser tools).

So I am guessing it’s because they are older versions (but I thought whenever they are older version you usually upgrade to the next newest until we get to the latest release : /)

13

Oh I see. If anyone can reproduce this in something like latest LTS Ubuntu or something more modern let me know.

1 Like
14

I managed to successfully move the forum to another server via import so can upgrade the kernel on the old server and try rebuilding again later if you like - however I suspect that is not the issue, the issue seems to be related to multiple containers, and where two were regularly kept updated, and two were not (it was the older two that failed to rebuild).

1 Like
15

What was the error? You probably need to share the full logs.

You’ll definitely need to rebuild rather than do the upgrade from the UX.

16

Do you mean the full output or something from log files? (If so which ones?)

1 Like
17

Were you able to build a container on the new server?

18

I installed a fresh copy of Discourse then carried out an import of the backup Jay (that’s how I usually move discourse forums to a new server).

19

So, use the new server?

20

Yes, that’s what I’ve done - just wondered whether @Falco wanted me to carry out any tests on the old server before decommissioning it, in case the DC team want to investigate why this happened.

1 Like
21

Since that is running some super old kernel version there is no need to keep it around, but thanks for the offer!