Stuck with 500 error after weird bugs and a rebuild

this time i created a new droplet, did the minimum it takes to get ready for a restore, then followed these instructions to restore the backup I managed to create after my site went down even though it was still accessible via command line: Restore a backup from the command line

Now I’m running into the below error. Perhaps my database got corrupted somehow? :thinking:

I can try restoring an earlier backup

./launcher rebuild app
x86_64 arch detected.
WARNING: containers/app.yml file is world-readable. You can secure this file by running: chmod o-rwx containers/app.yml
Ensuring launcher is up to date
Fetching origin
Launcher is up-to-date
Stopping old container
+ /usr/bin/docker stop -t 600 app
app
2.0.20231218-0429: Pulling from discourse/base
Digest: sha256:468f70b9bb4c6d0c6c2bbb3efc1a5e12d145eae57bdb6946b7fe5558beb52dc1
Status: Image is up to date for discourse/base:2.0.20231218-0429
docker.io/discourse/base:2.0.20231218-0429
/usr/local/lib/ruby/gems/3.2.0/gems/pups-1.2.1/lib/pups.rb
/usr/local/bin/pups --stdin
I, [2024-01-06T21:09:54.354218 #1]  INFO -- : Reading from stdin
I, [2024-01-06T21:09:54.365267 #1]  INFO -- : > locale-gen $LANG && update-locale
I, [2024-01-06T21:09:54.421443 #1]  INFO -- : Generating locales (this might take a while)...
Generation complete.

I, [2024-01-06T21:09:54.421795 #1]  INFO -- : > mkdir -p /shared/postgres_run
I, [2024-01-06T21:09:54.427121 #1]  INFO -- : 
I, [2024-01-06T21:09:54.427537 #1]  INFO -- : > chown postgres:postgres /shared/postgres_run
I, [2024-01-06T21:09:54.434259 #1]  INFO -- : 
I, [2024-01-06T21:09:54.434795 #1]  INFO -- : > chmod 775 /shared/postgres_run
I, [2024-01-06T21:09:54.441347 #1]  INFO -- : 
I, [2024-01-06T21:09:54.441696 #1]  INFO -- : > rm -fr /var/run/postgresql
I, [2024-01-06T21:09:54.448226 #1]  INFO -- : 
I, [2024-01-06T21:09:54.448601 #1]  INFO -- : > ln -s /shared/postgres_run /var/run/postgresql
I, [2024-01-06T21:09:54.454601 #1]  INFO -- : 
I, [2024-01-06T21:09:54.454965 #1]  INFO -- : > socat /dev/null UNIX-CONNECT:/shared/postgres_run/.s.PGSQL.5432 || exit 0 && echo postgres already running stop container ; exit 1
2024/01/06 21:09:54 socat[19] E connect(6, AF=1 "/shared/postgres_run/.s.PGSQL.5432", 36): Connection refused
I, [2024-01-06T21:09:54.470043 #1]  INFO -- : 
I, [2024-01-06T21:09:54.470647 #1]  INFO -- : > rm -fr /shared/postgres_run/.s*
I, [2024-01-06T21:09:54.478233 #1]  INFO -- : 
I, [2024-01-06T21:09:54.478786 #1]  INFO -- : > rm -fr /shared/postgres_run/*.pid
I, [2024-01-06T21:09:54.486059 #1]  INFO -- : 
I, [2024-01-06T21:09:54.486533 #1]  INFO -- : > mkdir -p /shared/postgres_run/13-main.pg_stat_tmp
I, [2024-01-06T21:09:54.491565 #1]  INFO -- : 
I, [2024-01-06T21:09:54.491920 #1]  INFO -- : > chown postgres:postgres /shared/postgres_run/13-main.pg_stat_tmp
I, [2024-01-06T21:09:54.496463 #1]  INFO -- : 
I, [2024-01-06T21:09:54.508217 #1]  INFO -- : File > /etc/service/postgres/run  chmod: +x  chown: 
I, [2024-01-06T21:09:54.519782 #1]  INFO -- : File > /etc/service/postgres/log/run  chmod: +x  chown: 
I, [2024-01-06T21:09:54.531602 #1]  INFO -- : File > /etc/runit/3.d/99-postgres  chmod: +x  chown: 
I, [2024-01-06T21:09:54.545224 #1]  INFO -- : File > /root/upgrade_postgres  chmod: +x  chown: 
I, [2024-01-06T21:09:54.545698 #1]  INFO -- : > chown -R root /var/lib/postgresql/13/main
I, [2024-01-06T21:09:57.594747 #1]  INFO -- : 
I, [2024-01-06T21:09:57.596324 #1]  INFO -- : > [ ! -e /shared/postgres_data ] && install -d -m 0755 -o postgres -g postgres /shared/postgres_data && sudo -E -u postgres /usr/lib/postgresql/13/bin/initdb -D /shared/postgres_data || exit 0
I, [2024-01-06T21:09:57.601398 #1]  INFO -- : 
I, [2024-01-06T21:09:57.601559 #1]  INFO -- : > chown -R postgres:postgres /shared/postgres_data
I, [2024-01-06T21:09:57.647669 #1]  INFO -- : 
I, [2024-01-06T21:09:57.648126 #1]  INFO -- : > chown -R postgres:postgres /var/run/postgresql
I, [2024-01-06T21:09:57.654847 #1]  INFO -- : 
I, [2024-01-06T21:09:57.655228 #1]  INFO -- : > /root/upgrade_postgres
I, [2024-01-06T21:09:57.671346 #1]  INFO -- : 
I, [2024-01-06T21:09:57.671840 #1]  INFO -- : > rm /root/upgrade_postgres
I, [2024-01-06T21:09:57.677190 #1]  INFO -- : 
I, [2024-01-06T21:09:57.678815 #1]  INFO -- : Replacing data_directory = '/var/lib/postgresql/13/main' with data_directory = '/shared/postgres_data' in /etc/postgresql/13/main/postgresql.conf
I, [2024-01-06T21:09:57.680570 #1]  INFO -- : Replacing (?-mix:#?listen_addresses *=.*) with listen_addresses = '*' in /etc/postgresql/13/main/postgresql.conf
I, [2024-01-06T21:09:57.681591 #1]  INFO -- : Replacing (?-mix:#?synchronous_commit *=.*) with synchronous_commit = $db_synchronous_commit in /etc/postgresql/13/main/postgresql.conf
I, [2024-01-06T21:09:57.693950 #1]  INFO -- : Replacing (?-mix:#?shared_buffers *=.*) with shared_buffers = $db_shared_buffers in /etc/postgresql/13/main/postgresql.conf
I, [2024-01-06T21:09:57.696644 #1]  INFO -- : Replacing (?-mix:#?work_mem *=.*) with work_mem = $db_work_mem in /etc/postgresql/13/main/postgresql.conf
I, [2024-01-06T21:09:57.699678 #1]  INFO -- : Replacing (?-mix:#?default_text_search_config *=.*) with default_text_search_config = '$db_default_text_search_config' in /etc/postgresql/13/main/postgresql.conf
I, [2024-01-06T21:09:57.706129 #1]  INFO -- : > install -d -m 0755 -o postgres -g postgres /shared/postgres_backup
I, [2024-01-06T21:09:57.716052 #1]  INFO -- : 
I, [2024-01-06T21:09:57.716636 #1]  INFO -- : Replacing (?-mix:#?checkpoint_segments *=.*) with checkpoint_segments = $db_checkpoint_segments in /etc/postgresql/13/main/postgresql.conf
I, [2024-01-06T21:09:57.717255 #1]  INFO -- : Replacing (?-mix:#?logging_collector *=.*) with logging_collector = $db_logging_collector in /etc/postgresql/13/main/postgresql.conf
I, [2024-01-06T21:09:57.721602 #1]  INFO -- : Replacing (?-mix:#?log_min_duration_statement *=.*) with log_min_duration_statement = $db_log_min_duration_statement in /etc/postgresql/13/main/postgresql.conf
I, [2024-01-06T21:09:57.727937 #1]  INFO -- : Replacing (?-mix:^#local +replication +postgres +peer$) with local replication postgres  peer in /etc/postgresql/13/main/pg_hba.conf
I, [2024-01-06T21:09:57.728569 #1]  INFO -- : Replacing (?-mix:^host.*all.*all.*127.*$) with host all all 0.0.0.0/0 md5 in /etc/postgresql/13/main/pg_hba.conf
I, [2024-01-06T21:09:57.732277 #1]  INFO -- : Replacing (?-mix:^host.*all.*all.*::1\/128.*$) with host all all ::/0 md5 in /etc/postgresql/13/main/pg_hba.conf
I, [2024-01-06T21:09:57.733290 #1]  INFO -- : > HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main
I, [2024-01-06T21:09:57.738665 #1]  INFO -- : > sleep 5
2024-01-06 21:09:57.897 UTC [42] LOG:  starting PostgreSQL 13.13 (Debian 13.13-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
2024-01-06 21:09:57.897 UTC [42] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2024-01-06 21:09:57.897 UTC [42] LOG:  listening on IPv6 address "::", port 5432
2024-01-06 21:09:57.904 UTC [42] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2024-01-06 21:09:57.914 UTC [45] LOG:  database system was interrupted; last known up at 2024-01-06 21:06:38 UTC
2024-01-06 21:09:58.287 UTC [45] LOG:  database system was not properly shut down; automatic recovery in progress
2024-01-06 21:09:58.300 UTC [45] LOG:  redo starts at 6/3F42F228
I, [2024-01-06T21:10:02.744704 #1]  INFO -- : 
I, [2024-01-06T21:10:02.745102 #1]  INFO -- : > su postgres -c 'createdb discourse' || true
2024-01-06 21:10:02.852 UTC [49] postgres@postgres FATAL:  the database system is starting up
2024-01-06 21:10:02.857 UTC [50] postgres@template1 FATAL:  the database system is starting up
createdb: error: could not connect to database template1: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL:  the database system is starting up
I, [2024-01-06T21:10:02.865413 #1]  INFO -- : 
I, [2024-01-06T21:10:02.867743 #1]  INFO -- : > su postgres -c 'psql discourse -c "create user discourse;"' || true
2024-01-06 21:10:02.984 UTC [54] postgres@discourse FATAL:  the database system is starting up
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL:  the database system is starting up
I, [2024-01-06T21:10:02.989529 #1]  INFO -- : 
I, [2024-01-06T21:10:02.989918 #1]  INFO -- : > su postgres -c 'psql discourse -c "grant all privileges on database discourse to discourse;"' || true
2024-01-06 21:10:03.100 UTC [58] postgres@discourse FATAL:  the database system is starting up
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL:  the database system is starting up
I, [2024-01-06T21:10:03.107339 #1]  INFO -- : 
I, [2024-01-06T21:10:03.107772 #1]  INFO -- : > su postgres -c 'psql discourse -c "alter schema public owner to discourse;"'
2024-01-06 21:10:03.200 UTC [62] postgres@discourse FATAL:  the database system is starting up
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL:  the database system is starting up
I, [2024-01-06T21:10:03.222053 #1]  INFO -- : 
I, [2024-01-06T21:10:03.222926 #1]  INFO -- : Terminating async processes
I, [2024-01-06T21:10:03.223018 #1]  INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main pid: 42
2024-01-06 21:10:03.238 UTC [42] LOG:  received fast shutdown request
2024-01-06 21:10:03.362 UTC [65] LOG:  shutting down
2024-01-06 21:10:03.602 UTC [42] LOG:  database system is shut down


FAILED
--------------------
Pups::ExecError: su postgres -c 'psql discourse -c "alter schema public owner to discourse;"' failed with return #<Process::Status: pid 59 exit 2>
Location of failure: /usr/local/lib/ruby/gems/3.2.0/gems/pups-1.2.1/lib/pups/exec_command.rb:132:in `spawn'
exec failed with the params "su postgres -c 'psql $db_name -c \"alter schema public owner to $db_user;\"'"
bootstrap failed with exit code 2
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.
a74fe31949946fbbd088418657abc63cca8f2d8ecda9d2d200979f275df55711