Discourse update doesn't wait for Postgress DB to shut down

Yesterday we tried rebuilding the forum several times with the same result - it failed, and log shows it is related to how Discourse shuts the database down. It looks like it doesn’t match the speed expectations set by rebuild process and that’s why it fails.

Last time it happened, we just tried to re-build once again and it worked. Yesterday, though, we tried that 3 times with the same result. Starting up the instance helped, but that doesn’t solve the problem of inability to upgrade the forum by rebuilding.

I’m including the rebuild log below. Hope someone will be able to help us getting to the real reason and how can we help Discourse with rebuilding itself.

Rebuild failed log
kinetiksoft:~$ sudo /usr/local/bin/discourse-tool git-pull
remote: Enumerating objects: 47, done.
remote: Counting objects: 100% (47/47), done.
remote: Compressing objects: 100% (14/14), done.
remote: Total 47 (delta 20), reused 41 (delta 19), pack-reused 0
Unpacking objects: 100% (47/47), done.
From https://github.com/discourse/discourse_docker
   02cd746..4cd6a5d  main       -> origin/main
Updating 02cd746..4cd6a5d
Fast-forward
 .github/workflows/build.yml                | 85 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--------------
 image/auto_build.rb                        |  4 ++--
 image/base/release.Dockerfile              | 11 +++++++++++
 image/base/{Dockerfile => slim.Dockerfile} |  8 +-------
 image/discourse_test/Dockerfile            | 40 +++++++++++++++++++++-------------------
 5 files changed, 106 insertions(+), 42 deletions(-)
 create mode 100644 image/base/release.Dockerfile
 rename image/base/{Dockerfile => slim.Dockerfile} (94%)
kinetiksoft:~$ sudo /usr/local/bin/discourse-tool rebuild
x86_64 arch detected.
Ensuring launcher is up to date
Fetching origin
Launcher is up-to-date
Stopping old container
+ /usr/bin/docker stop -t 60 app
app
cd /pups && git pull && git checkout v1.0.3 && /pups/bin/pups --stdin
Already up to date.
Note: checking out 'v1.0.3'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at d1db030 cut a new version
I, [2021-12-22T12:03:30.961687 #1]  INFO -- : Loading --stdin
I, [2021-12-22T12:03:30.965617 #1]  INFO -- : > locale-gen $LANG && update-locale
I, [2021-12-22T12:03:31.025039 #1]  INFO -- : Generating locales (this might take a while)...
Generation complete.

I, [2021-12-22T12:03:31.025205 #1]  INFO -- : > mkdir -p /shared/postgres_run
I, [2021-12-22T12:03:31.027383 #1]  INFO -- : 
I, [2021-12-22T12:03:31.027600 #1]  INFO -- : > chown postgres:postgres /shared/postgres_run
I, [2021-12-22T12:03:31.034332 #1]  INFO -- : 
I, [2021-12-22T12:03:31.034470 #1]  INFO -- : > chmod 775 /shared/postgres_run
I, [2021-12-22T12:03:31.035865 #1]  INFO -- : 
I, [2021-12-22T12:03:31.035982 #1]  INFO -- : > rm -fr /var/run/postgresql
I, [2021-12-22T12:03:31.037608 #1]  INFO -- : 
I, [2021-12-22T12:03:31.037725 #1]  INFO -- : > ln -s /shared/postgres_run /var/run/postgresql
I, [2021-12-22T12:03:31.041565 #1]  INFO -- : 
I, [2021-12-22T12:03:31.041696 #1]  INFO -- : > socat /dev/null UNIX-CONNECT:/shared/postgres_run/.s.PGSQL.5432 || exit 0 && echo postgres already running stop container ; exit 1
2021/12/22 12:03:31 socat[27] E connect(6, AF=1 "/shared/postgres_run/.s.PGSQL.5432", 36): Connection refused
I, [2021-12-22T12:03:31.053403 #1]  INFO -- : 
I, [2021-12-22T12:03:31.053535 #1]  INFO -- : > rm -fr /shared/postgres_run/.s*
I, [2021-12-22T12:03:31.055582 #1]  INFO -- : 
I, [2021-12-22T12:03:31.055693 #1]  INFO -- : > rm -fr /shared/postgres_run/*.pid
I, [2021-12-22T12:03:31.057512 #1]  INFO -- : 
I, [2021-12-22T12:03:31.057618 #1]  INFO -- : > mkdir -p /shared/postgres_run/13-main.pg_stat_tmp
I, [2021-12-22T12:03:31.058997 #1]  INFO -- : 
I, [2021-12-22T12:03:31.059130 #1]  INFO -- : > chown postgres:postgres /shared/postgres_run/13-main.pg_stat_tmp
I, [2021-12-22T12:03:31.060530 #1]  INFO -- : 
I, [2021-12-22T12:03:31.063566 #1]  INFO -- : File > /etc/service/postgres/run  chmod: +x  chown: 
I, [2021-12-22T12:03:31.066547 #1]  INFO -- : File > /etc/service/postgres/log/run  chmod: +x  chown: 
I, [2021-12-22T12:03:31.069511 #1]  INFO -- : File > /etc/runit/3.d/99-postgres  chmod: +x  chown: 
I, [2021-12-22T12:03:31.072380 #1]  INFO -- : File > /root/upgrade_postgres  chmod: +x  chown: 
I, [2021-12-22T12:03:31.072566 #1]  INFO -- : > chown -R root /var/lib/postgresql/13/main
I, [2021-12-22T12:04:08.248631 #1]  INFO -- : 
I, [2021-12-22T12:04:08.248799 #1]  INFO -- : > [ ! -e /shared/postgres_data ] && install -d -m 0755 -o postgres -g postgres /shared/postgres_data && sudo -E -u postgres /usr/lib/postgresql/13/bin/initdb -D /shared/postgres_data || exit 0
I, [2021-12-22T12:04:08.251062 #1]  INFO -- : 
I, [2021-12-22T12:04:08.251148 #1]  INFO -- : > chown -R postgres:postgres /shared/postgres_data
I, [2021-12-22T12:04:08.291023 #1]  INFO -- : 
I, [2021-12-22T12:04:08.291212 #1]  INFO -- : > chown -R postgres:postgres /var/run/postgresql
I, [2021-12-22T12:04:08.293474 #1]  INFO -- : 
I, [2021-12-22T12:04:08.293655 #1]  INFO -- : > /root/upgrade_postgres
I, [2021-12-22T12:04:08.301217 #1]  INFO -- : 
I, [2021-12-22T12:04:08.301351 #1]  INFO -- : > rm /root/upgrade_postgres
I, [2021-12-22T12:04:08.303272 #1]  INFO -- : 
I, [2021-12-22T12:04:08.310457 #1]  INFO -- : Replacing data_directory = '/var/lib/postgresql/13/main' with data_directory = '/shared/postgres_data' in /etc/postgresql/13/main/postgresql.conf
I, [2021-12-22T12:04:08.310879 #1]  INFO -- : Replacing (?-mix:#?listen_addresses *=.*) with listen_addresses = '*' in /etc/postgresql/13/main/postgresql.conf
I, [2021-12-22T12:04:08.318852 #1]  INFO -- : Replacing (?-mix:#?synchronous_commit *=.*) with synchronous_commit = $db_synchronous_commit in /etc/postgresql/13/main/postgresql.conf
I, [2021-12-22T12:04:08.319207 #1]  INFO -- : Replacing (?-mix:#?shared_buffers *=.*) with shared_buffers = $db_shared_buffers in /etc/postgresql/13/main/postgresql.conf
I, [2021-12-22T12:04:08.319512 #1]  INFO -- : Replacing (?-mix:#?work_mem *=.*) with work_mem = $db_work_mem in /etc/postgresql/13/main/postgresql.conf
I, [2021-12-22T12:04:08.319791 #1]  INFO -- : Replacing (?-mix:#?default_text_search_config *=.*) with default_text_search_config = '$db_default_text_search_config' in /etc/postgresql/13/main/postgresql.conf
I, [2021-12-22T12:04:08.320079 #1]  INFO -- : > install -d -m 0755 -o postgres -g postgres /shared/postgres_backup
I, [2021-12-22T12:04:08.330091 #1]  INFO -- : 
I, [2021-12-22T12:04:08.330365 #1]  INFO -- : Replacing (?-mix:#?checkpoint_segments *=.*) with checkpoint_segments = $db_checkpoint_segments in /etc/postgresql/13/main/postgresql.conf
I, [2021-12-22T12:04:08.330681 #1]  INFO -- : Replacing (?-mix:#?logging_collector *=.*) with logging_collector = $db_logging_collector in /etc/postgresql/13/main/postgresql.conf
I, [2021-12-22T12:04:08.331278 #1]  INFO -- : Replacing (?-mix:#?log_min_duration_statement *=.*) with log_min_duration_statement = $db_log_min_duration_statement in /etc/postgresql/13/main/postgresql.conf
I, [2021-12-22T12:04:08.335199 #1]  INFO -- : Replacing (?-mix:^#local +replication +postgres +peer$) with local replication postgres  peer in /etc/postgresql/13/main/pg_hba.conf
I, [2021-12-22T12:04:08.335399 #1]  INFO -- : Replacing (?-mix:^host.*all.*all.*127.*$) with host all all 0.0.0.0/0 md5 in /etc/postgresql/13/main/pg_hba.conf
I, [2021-12-22T12:04:08.343492 #1]  INFO -- : Replacing (?-mix:^host.*all.*all.*::1\/128.*$) with host all all ::/0 md5 in /etc/postgresql/13/main/pg_hba.conf
I, [2021-12-22T12:04:08.343701 #1]  INFO -- : > HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main
I, [2021-12-22T12:04:08.345307 #1]  INFO -- : > sleep 5
2021-12-22 12:04:08.666 UTC [50] LOG:  starting PostgreSQL 13.5 (Debian 13.5-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
2021-12-22 12:04:08.666 UTC [50] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2021-12-22 12:04:08.666 UTC [50] LOG:  listening on IPv6 address "::", port 5432
2021-12-22 12:04:08.732 UTC [50] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2021-12-22 12:04:08.808 UTC [53] LOG:  database system shutdown was interrupted; last known up at 2021-12-22 12:02:56 UTC
2021-12-22 12:04:09.928 UTC [53] LOG:  database system was not properly shut down; automatic recovery in progress
2021-12-22 12:04:09.972 UTC [53] LOG:  redo starts at 83/B07EB758
2021-12-22 12:04:10.123 UTC [53] LOG:  invalid record length at 83/B12EDED8: wanted 24, got 0
2021-12-22 12:04:10.123 UTC [53] LOG:  redo done at 83/B12EDEA0
I, [2021-12-22T12:04:13.347316 #1]  INFO -- : 
I, [2021-12-22T12:04:13.347566 #1]  INFO -- : > su postgres -c 'createdb discourse' || true
2021-12-22 12:04:13.521 UTC [57] postgres@postgres FATAL:  the database system is starting up
2021-12-22 12:04:13.522 UTC [58] postgres@template1 FATAL:  the database system is starting up
createdb: error: could not connect to database template1: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL:  the database system is starting up
I, [2021-12-22T12:04:13.523332 #1]  INFO -- : 
I, [2021-12-22T12:04:13.523518 #1]  INFO -- : > su postgres -c 'psql discourse -c "create user discourse;"' || true
2021-12-22 12:04:13.791 UTC [62] postgres@discourse FATAL:  the database system is starting up
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL:  the database system is starting up
I, [2021-12-22T12:04:13.792674 #1]  INFO -- : 
I, [2021-12-22T12:04:13.792923 #1]  INFO -- : > su postgres -c 'psql discourse -c "grant all privileges on database discourse to discourse;"' || true
2021-12-22 12:04:13.824 UTC [66] postgres@discourse FATAL:  the database system is starting up
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL:  the database system is starting up
I, [2021-12-22T12:04:13.825180 #1]  INFO -- : 
I, [2021-12-22T12:04:13.825452 #1]  INFO -- : > su postgres -c 'psql discourse -c "alter schema public owner to discourse;"'
2021-12-22 12:04:13.856 UTC [70] postgres@discourse FATAL:  the database system is starting up
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL:  the database system is starting up
I, [2021-12-22T12:04:13.857672 #1]  INFO -- : 
I, [2021-12-22T12:04:13.858028 #1]  INFO -- : Terminating async processes
I, [2021-12-22T12:04:13.858053 #1]  INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main pid: 50
2021-12-22 12:04:13.858 UTC [50] LOG:  received fast shutdown request
I, [2021-12-22T12:04:23.858557 #1]  INFO -- : HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main pid:50 did not terminate cleanly, forcing termination!


FAILED
--------------------
Pups::ExecError: su postgres -c 'psql discourse -c "alter schema public owner to discourse;"' failed with return #<Process::Status: pid 67 exit 2>
Location of failure: /pups/lib/pups/exec_command.rb:112:in `spawn'
exec failed with the params "su postgres -c 'psql $db_name -c \"alter schema public owner to $db_user;\"'"
08af256a113e238f69588774fcfed142d94742fc4f09c9182cb20c1901a7aa61
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.
kinetiksoft:~$ sudo /usr/local/bin/discourse-tool rebuild
x86_64 arch detected.
Ensuring launcher is up to date
Fetching origin
Launcher is up-to-date
Stopping old container
+ /usr/bin/docker stop -t 60 app
app
cd /pups && git pull && git checkout v1.0.3 && /pups/bin/pups --stdin
Already up to date.
Note: checking out 'v1.0.3'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at d1db030 cut a new version
I, [2021-12-22T12:05:48.204969 #1]  INFO -- : Loading --stdin
I, [2021-12-22T12:05:48.209104 #1]  INFO -- : > locale-gen $LANG && update-locale
I, [2021-12-22T12:05:48.229337 #1]  INFO -- : Generating locales (this might take a while)...
Generation complete.

I, [2021-12-22T12:05:48.229510 #1]  INFO -- : > mkdir -p /shared/postgres_run
I, [2021-12-22T12:05:48.231324 #1]  INFO -- : 
I, [2021-12-22T12:05:48.231513 #1]  INFO -- : > chown postgres:postgres /shared/postgres_run
I, [2021-12-22T12:05:48.233015 #1]  INFO -- : 
I, [2021-12-22T12:05:48.233122 #1]  INFO -- : > chmod 775 /shared/postgres_run
I, [2021-12-22T12:05:48.236585 #1]  INFO -- : 
I, [2021-12-22T12:05:48.236706 #1]  INFO -- : > rm -fr /var/run/postgresql
I, [2021-12-22T12:05:48.238156 #1]  INFO -- : 
I, [2021-12-22T12:05:48.238287 #1]  INFO -- : > ln -s /shared/postgres_run /var/run/postgresql
I, [2021-12-22T12:05:48.239526 #1]  INFO -- : 
I, [2021-12-22T12:05:48.239634 #1]  INFO -- : > socat /dev/null UNIX-CONNECT:/shared/postgres_run/.s.PGSQL.5432 || exit 0 && echo postgres already running stop container ; exit 1
2021/12/22 12:05:48 socat[27] E connect(6, AF=1 "/shared/postgres_run/.s.PGSQL.5432", 36): Connection refused
I, [2021-12-22T12:05:48.242590 #1]  INFO -- : 
I, [2021-12-22T12:05:48.242720 #1]  INFO -- : > rm -fr /shared/postgres_run/.s*
I, [2021-12-22T12:05:48.244501 #1]  INFO -- : 
I, [2021-12-22T12:05:48.244589 #1]  INFO -- : > rm -fr /shared/postgres_run/*.pid
I, [2021-12-22T12:05:48.246303 #1]  INFO -- : 
I, [2021-12-22T12:05:48.246397 #1]  INFO -- : > mkdir -p /shared/postgres_run/13-main.pg_stat_tmp
I, [2021-12-22T12:05:48.247818 #1]  INFO -- : 
I, [2021-12-22T12:05:48.248021 #1]  INFO -- : > chown postgres:postgres /shared/postgres_run/13-main.pg_stat_tmp
I, [2021-12-22T12:05:48.249386 #1]  INFO -- : 
I, [2021-12-22T12:05:48.252499 #1]  INFO -- : File > /etc/service/postgres/run  chmod: +x  chown: 
I, [2021-12-22T12:05:48.255331 #1]  INFO -- : File > /etc/service/postgres/log/run  chmod: +x  chown: 
I, [2021-12-22T12:05:48.258241 #1]  INFO -- : File > /etc/runit/3.d/99-postgres  chmod: +x  chown: 
I, [2021-12-22T12:05:48.261139 #1]  INFO -- : File > /root/upgrade_postgres  chmod: +x  chown: 
I, [2021-12-22T12:05:48.261348 #1]  INFO -- : > chown -R root /var/lib/postgresql/13/main
I, [2021-12-22T12:06:24.321828 #1]  INFO -- : 
I, [2021-12-22T12:06:24.321951 #1]  INFO -- : > [ ! -e /shared/postgres_data ] && install -d -m 0755 -o postgres -g postgres /shared/postgres_data && sudo -E -u postgres /usr/lib/postgresql/13/bin/initdb -D /shared/postgres_data || exit 0
I, [2021-12-22T12:06:24.323903 #1]  INFO -- : 
I, [2021-12-22T12:06:24.323952 #1]  INFO -- : > chown -R postgres:postgres /shared/postgres_data
I, [2021-12-22T12:06:24.339115 #1]  INFO -- : 
I, [2021-12-22T12:06:24.339247 #1]  INFO -- : > chown -R postgres:postgres /var/run/postgresql
I, [2021-12-22T12:06:24.340865 #1]  INFO -- : 
I, [2021-12-22T12:06:24.341031 #1]  INFO -- : > /root/upgrade_postgres
I, [2021-12-22T12:06:24.343922 #1]  INFO -- : 
I, [2021-12-22T12:06:24.344019 #1]  INFO -- : > rm /root/upgrade_postgres
I, [2021-12-22T12:06:24.345316 #1]  INFO -- : 
I, [2021-12-22T12:06:24.345502 #1]  INFO -- : Replacing data_directory = '/var/lib/postgresql/13/main' with data_directory = '/shared/postgres_data' in /etc/postgresql/13/main/postgresql.conf
I, [2021-12-22T12:06:24.345857 #1]  INFO -- : Replacing (?-mix:#?listen_addresses *=.*) with listen_addresses = '*' in /etc/postgresql/13/main/postgresql.conf
I, [2021-12-22T12:06:24.346104 #1]  INFO -- : Replacing (?-mix:#?synchronous_commit *=.*) with synchronous_commit = $db_synchronous_commit in /etc/postgresql/13/main/postgresql.conf
I, [2021-12-22T12:06:24.346356 #1]  INFO -- : Replacing (?-mix:#?shared_buffers *=.*) with shared_buffers = $db_shared_buffers in /etc/postgresql/13/main/postgresql.conf
I, [2021-12-22T12:06:24.346628 #1]  INFO -- : Replacing (?-mix:#?work_mem *=.*) with work_mem = $db_work_mem in /etc/postgresql/13/main/postgresql.conf
I, [2021-12-22T12:06:24.346891 #1]  INFO -- : Replacing (?-mix:#?default_text_search_config *=.*) with default_text_search_config = '$db_default_text_search_config' in /etc/postgresql/13/main/postgresql.conf
I, [2021-12-22T12:06:24.347165 #1]  INFO -- : > install -d -m 0755 -o postgres -g postgres /shared/postgres_backup
I, [2021-12-22T12:06:24.348910 #1]  INFO -- : 
I, [2021-12-22T12:06:24.349140 #1]  INFO -- : Replacing (?-mix:#?checkpoint_segments *=.*) with checkpoint_segments = $db_checkpoint_segments in /etc/postgresql/13/main/postgresql.conf
I, [2021-12-22T12:06:24.349383 #1]  INFO -- : Replacing (?-mix:#?logging_collector *=.*) with logging_collector = $db_logging_collector in /etc/postgresql/13/main/postgresql.conf
I, [2021-12-22T12:06:24.349678 #1]  INFO -- : Replacing (?-mix:#?log_min_duration_statement *=.*) with log_min_duration_statement = $db_log_min_duration_statement in /etc/postgresql/13/main/postgresql.conf
I, [2021-12-22T12:06:24.349972 #1]  INFO -- : Replacing (?-mix:^#local +replication +postgres +peer$) with local replication postgres  peer in /etc/postgresql/13/main/pg_hba.conf
I, [2021-12-22T12:06:24.350138 #1]  INFO -- : Replacing (?-mix:^host.*all.*all.*127.*$) with host all all 0.0.0.0/0 md5 in /etc/postgresql/13/main/pg_hba.conf
I, [2021-12-22T12:06:24.350303 #1]  INFO -- : Replacing (?-mix:^host.*all.*all.*::1\/128.*$) with host all all ::/0 md5 in /etc/postgresql/13/main/pg_hba.conf
I, [2021-12-22T12:06:24.350464 #1]  INFO -- : > HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main
I, [2021-12-22T12:06:24.351392 #1]  INFO -- : > sleep 5
2021-12-22 12:06:24.490 UTC [50] LOG:  starting PostgreSQL 13.5 (Debian 13.5-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
2021-12-22 12:06:24.490 UTC [50] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2021-12-22 12:06:24.490 UTC [50] LOG:  listening on IPv6 address "::", port 5432
2021-12-22 12:06:24.571 UTC [50] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2021-12-22 12:06:24.637 UTC [53] LOG:  database system shutdown was interrupted; last known up at 2021-12-22 12:04:10 UTC
2021-12-22 12:06:26.793 UTC [53] LOG:  database system was not properly shut down; automatic recovery in progress
2021-12-22 12:06:26.827 UTC [53] LOG:  redo starts at 83/B07EB758
2021-12-22 12:06:26.852 UTC [53] LOG:  invalid record length at 83/B12EDED8: wanted 24, got 0
2021-12-22 12:06:26.852 UTC [53] LOG:  redo done at 83/B12EDEA0
I, [2021-12-22T12:06:29.352869 #1]  INFO -- : 
I, [2021-12-22T12:06:29.353167 #1]  INFO -- : > su postgres -c 'createdb discourse' || true
2021-12-22 12:06:29.385 UTC [57] postgres@postgres FATAL:  the database system is starting up
2021-12-22 12:06:29.385 UTC [58] postgres@template1 FATAL:  the database system is starting up
createdb: error: could not connect to database template1: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL:  the database system is starting up
I, [2021-12-22T12:06:29.386701 #1]  INFO -- : 
I, [2021-12-22T12:06:29.386897 #1]  INFO -- : > su postgres -c 'psql discourse -c "create user discourse;"' || true
2021-12-22 12:06:29.417 UTC [62] postgres@discourse FATAL:  the database system is starting up
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL:  the database system is starting up
I, [2021-12-22T12:06:29.418609 #1]  INFO -- : 
I, [2021-12-22T12:06:29.418798 #1]  INFO -- : > su postgres -c 'psql discourse -c "grant all privileges on database discourse to discourse;"' || true
2021-12-22 12:06:29.453 UTC [66] postgres@discourse FATAL:  the database system is starting up
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL:  the database system is starting up
I, [2021-12-22T12:06:29.454838 #1]  INFO -- : 
I, [2021-12-22T12:06:29.455124 #1]  INFO -- : > su postgres -c 'psql discourse -c "alter schema public owner to discourse;"'
2021-12-22 12:06:29.486 UTC [70] postgres@discourse FATAL:  the database system is starting up
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL:  the database system is starting up
I, [2021-12-22T12:06:29.487598 #1]  INFO -- : 
I, [2021-12-22T12:06:29.487929 #1]  INFO -- : Terminating async processes
I, [2021-12-22T12:06:29.487965 #1]  INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main pid: 50
2021-12-22 12:06:29.488 UTC [50] LOG:  received fast shutdown request
I, [2021-12-22T12:06:39.488426 #1]  INFO -- : HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main pid:50 did not terminate cleanly, forcing termination!


FAILED
--------------------
Pups::ExecError: su postgres -c 'psql discourse -c "alter schema public owner to discourse;"' failed with return #<Process::Status: pid 67 exit 2>
Location of failure: /pups/lib/pups/exec_command.rb:112:in `spawn'
exec failed with the params "su postgres -c 'psql $db_name -c \"alter schema public owner to $db_user;\"'"
7fee51af158f0ed17bf212e936826b00650dd82d1825ff48889479f7c0d3fade
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.
kinetiksoft:~$ sudo /usr/local/bin/discourse-tool rebuild
x86_64 arch detected.
Ensuring launcher is up to date
Fetching origin
Launcher is up-to-date
Stopping old container
+ /usr/bin/docker stop -t 60 app
app
cd /pups && git pull && git checkout v1.0.3 && /pups/bin/pups --stdin
Already up to date.
Note: checking out 'v1.0.3'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at d1db030 cut a new version
I, [2021-12-22T12:08:33.310010 #1]  INFO -- : Loading --stdin
I, [2021-12-22T12:08:33.313979 #1]  INFO -- : > locale-gen $LANG && update-locale
I, [2021-12-22T12:08:33.333943 #1]  INFO -- : Generating locales (this might take a while)...
Generation complete.

I, [2021-12-22T12:08:33.334119 #1]  INFO -- : > mkdir -p /shared/postgres_run
I, [2021-12-22T12:08:33.336004 #1]  INFO -- : 
I, [2021-12-22T12:08:33.336149 #1]  INFO -- : > chown postgres:postgres /shared/postgres_run
I, [2021-12-22T12:08:33.337638 #1]  INFO -- : 
I, [2021-12-22T12:08:33.337777 #1]  INFO -- : > chmod 775 /shared/postgres_run
I, [2021-12-22T12:08:33.339209 #1]  INFO -- : 
I, [2021-12-22T12:08:33.339293 #1]  INFO -- : > rm -fr /var/run/postgresql
I, [2021-12-22T12:08:33.340714 #1]  INFO -- : 
I, [2021-12-22T12:08:33.340818 #1]  INFO -- : > ln -s /shared/postgres_run /var/run/postgresql
I, [2021-12-22T12:08:33.342042 #1]  INFO -- : 
I, [2021-12-22T12:08:33.342136 #1]  INFO -- : > socat /dev/null UNIX-CONNECT:/shared/postgres_run/.s.PGSQL.5432 || exit 0 && echo postgres already running stop container ; exit 1
2021/12/22 12:08:33 socat[27] E connect(6, AF=1 "/shared/postgres_run/.s.PGSQL.5432", 36): Connection refused
I, [2021-12-22T12:08:33.344996 #1]  INFO -- : 
I, [2021-12-22T12:08:33.345091 #1]  INFO -- : > rm -fr /shared/postgres_run/.s*
I, [2021-12-22T12:08:33.346977 #1]  INFO -- : 
I, [2021-12-22T12:08:33.347081 #1]  INFO -- : > rm -fr /shared/postgres_run/*.pid
I, [2021-12-22T12:08:33.348866 #1]  INFO -- : 
I, [2021-12-22T12:08:33.348953 #1]  INFO -- : > mkdir -p /shared/postgres_run/13-main.pg_stat_tmp
I, [2021-12-22T12:08:33.350474 #1]  INFO -- : 
I, [2021-12-22T12:08:33.350644 #1]  INFO -- : > chown postgres:postgres /shared/postgres_run/13-main.pg_stat_tmp
I, [2021-12-22T12:08:33.352043 #1]  INFO -- : 
I, [2021-12-22T12:08:33.355013 #1]  INFO -- : File > /etc/service/postgres/run  chmod: +x  chown: 
I, [2021-12-22T12:08:33.357838 #1]  INFO -- : File > /etc/service/postgres/log/run  chmod: +x  chown: 
I, [2021-12-22T12:08:33.360727 #1]  INFO -- : File > /etc/runit/3.d/99-postgres  chmod: +x  chown: 
I, [2021-12-22T12:08:33.364940 #1]  INFO -- : File > /root/upgrade_postgres  chmod: +x  chown: 
I, [2021-12-22T12:08:33.365242 #1]  INFO -- : > chown -R root /var/lib/postgresql/13/main
I, [2021-12-22T12:09:12.888922 #1]  INFO -- : 
I, [2021-12-22T12:09:12.889060 #1]  INFO -- : > [ ! -e /shared/postgres_data ] && install -d -m 0755 -o postgres -g postgres /shared/postgres_data && sudo -E -u postgres /usr/lib/postgresql/13/bin/initdb -D /shared/postgres_data || exit 0
I, [2021-12-22T12:09:12.890602 #1]  INFO -- : 
I, [2021-12-22T12:09:12.890637 #1]  INFO -- : > chown -R postgres:postgres /shared/postgres_data
I, [2021-12-22T12:09:12.905164 #1]  INFO -- : 
I, [2021-12-22T12:09:12.905282 #1]  INFO -- : > chown -R postgres:postgres /var/run/postgresql
I, [2021-12-22T12:09:12.907055 #1]  INFO -- : 
I, [2021-12-22T12:09:12.907187 #1]  INFO -- : > /root/upgrade_postgres
I, [2021-12-22T12:09:12.910221 #1]  INFO -- : 
I, [2021-12-22T12:09:12.910336 #1]  INFO -- : > rm /root/upgrade_postgres
I, [2021-12-22T12:09:12.911656 #1]  INFO -- : 
I, [2021-12-22T12:09:12.911836 #1]  INFO -- : Replacing data_directory = '/var/lib/postgresql/13/main' with data_directory = '/shared/postgres_data' in /etc/postgresql/13/main/postgresql.conf
I, [2021-12-22T12:09:12.912203 #1]  INFO -- : Replacing (?-mix:#?listen_addresses *=.*) with listen_addresses = '*' in /etc/postgresql/13/main/postgresql.conf
I, [2021-12-22T12:09:12.933927 #1]  INFO -- : Replacing (?-mix:#?synchronous_commit *=.*) with synchronous_commit = $db_synchronous_commit in /etc/postgresql/13/main/postgresql.conf
I, [2021-12-22T12:09:12.934365 #1]  INFO -- : Replacing (?-mix:#?shared_buffers *=.*) with shared_buffers = $db_shared_buffers in /etc/postgresql/13/main/postgresql.conf
I, [2021-12-22T12:09:12.934753 #1]  INFO -- : Replacing (?-mix:#?work_mem *=.*) with work_mem = $db_work_mem in /etc/postgresql/13/main/postgresql.conf
I, [2021-12-22T12:09:12.935135 #1]  INFO -- : Replacing (?-mix:#?default_text_search_config *=.*) with default_text_search_config = '$db_default_text_search_config' in /etc/postgresql/13/main/postgresql.conf
I, [2021-12-22T12:09:12.935459 #1]  INFO -- : > install -d -m 0755 -o postgres -g postgres /shared/postgres_backup
I, [2021-12-22T12:09:12.938583 #1]  INFO -- : 
I, [2021-12-22T12:09:12.938889 #1]  INFO -- : Replacing (?-mix:#?checkpoint_segments *=.*) with checkpoint_segments = $db_checkpoint_segments in /etc/postgresql/13/main/postgresql.conf
I, [2021-12-22T12:09:12.939246 #1]  INFO -- : Replacing (?-mix:#?logging_collector *=.*) with logging_collector = $db_logging_collector in /etc/postgresql/13/main/postgresql.conf
I, [2021-12-22T12:09:12.939664 #1]  INFO -- : Replacing (?-mix:#?log_min_duration_statement *=.*) with log_min_duration_statement = $db_log_min_duration_statement in /etc/postgresql/13/main/postgresql.conf
I, [2021-12-22T12:09:12.940011 #1]  INFO -- : Replacing (?-mix:^#local +replication +postgres +peer$) with local replication postgres  peer in /etc/postgresql/13/main/pg_hba.conf
I, [2021-12-22T12:09:12.940375 #1]  INFO -- : Replacing (?-mix:^host.*all.*all.*127.*$) with host all all 0.0.0.0/0 md5 in /etc/postgresql/13/main/pg_hba.conf
I, [2021-12-22T12:09:12.940711 #1]  INFO -- : Replacing (?-mix:^host.*all.*all.*::1\/128.*$) with host all all ::/0 md5 in /etc/postgresql/13/main/pg_hba.conf
I, [2021-12-22T12:09:12.941023 #1]  INFO -- : > HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main
I, [2021-12-22T12:09:12.942680 #1]  INFO -- : > sleep 5
2021-12-22 12:09:13.078 UTC [50] LOG:  starting PostgreSQL 13.5 (Debian 13.5-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
2021-12-22 12:09:13.078 UTC [50] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2021-12-22 12:09:13.078 UTC [50] LOG:  listening on IPv6 address "::", port 5432
2021-12-22 12:09:13.130 UTC [50] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2021-12-22 12:09:13.198 UTC [53] LOG:  database system shutdown was interrupted; last known up at 2021-12-22 12:06:26 UTC
2021-12-22 12:09:13.916 UTC [53] LOG:  database system was not properly shut down; automatic recovery in progress
2021-12-22 12:09:13.967 UTC [53] LOG:  redo starts at 83/B07EB758
2021-12-22 12:09:13.992 UTC [53] LOG:  invalid record length at 83/B12EDED8: wanted 24, got 0
2021-12-22 12:09:13.992 UTC [53] LOG:  redo done at 83/B12EDEA0
I, [2021-12-22T12:09:17.952534 #1]  INFO -- : 
I, [2021-12-22T12:09:17.952786 #1]  INFO -- : > su postgres -c 'createdb discourse' || true
2021-12-22 12:09:17.984 UTC [57] postgres@postgres FATAL:  the database system is starting up
2021-12-22 12:09:17.985 UTC [58] postgres@template1 FATAL:  the database system is starting up
createdb: error: could not connect to database template1: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL:  the database system is starting up
I, [2021-12-22T12:09:17.986173 #1]  INFO -- : 
I, [2021-12-22T12:09:17.986413 #1]  INFO -- : > su postgres -c 'psql discourse -c "create user discourse;"' || true
2021-12-22 12:09:18.016 UTC [62] postgres@discourse FATAL:  the database system is starting up
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL:  the database system is starting up
I, [2021-12-22T12:09:18.018083 #1]  INFO -- : 
I, [2021-12-22T12:09:18.018325 #1]  INFO -- : > su postgres -c 'psql discourse -c "grant all privileges on database discourse to discourse;"' || true
2021-12-22 12:09:18.049 UTC [66] postgres@discourse FATAL:  the database system is starting up
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL:  the database system is starting up
I, [2021-12-22T12:09:18.051088 #1]  INFO -- : 
I, [2021-12-22T12:09:18.051313 #1]  INFO -- : > su postgres -c 'psql discourse -c "alter schema public owner to discourse;"'
2021-12-22 12:09:18.081 UTC [70] postgres@discourse FATAL:  the database system is starting up
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL:  the database system is starting up
I, [2021-12-22T12:09:18.082181 #1]  INFO -- : 
I, [2021-12-22T12:09:18.082516 #1]  INFO -- : Terminating async processes
I, [2021-12-22T12:09:18.082549 #1]  INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main pid: 50
2021-12-22 12:09:18.082 UTC [50] LOG:  received fast shutdown request
I, [2021-12-22T12:09:28.083088 #1]  INFO -- : HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main pid:50 did not terminate cleanly, forcing termination!


FAILED
--------------------
Pups::ExecError: su postgres -c 'psql discourse -c "alter schema public owner to discourse;"' failed with return #<Process::Status: pid 67 exit 2>
Location of failure: /pups/lib/pups/exec_command.rb:112:in `spawn'
exec failed with the params "su postgres -c 'psql $db_name -c \"alter schema public owner to $db_user;\"'"
1c37ea92329dcf84d63aebbb7c326ab53b3bdae35f538566effc9009403369fd
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.
kinetiksoft:~$ sudo /usr/local/bin/discourse-tool rebuild
x86_64 arch detected.
Ensuring launcher is up to date
Fetching origin
Launcher is up-to-date
Stopping old container
+ /usr/bin/docker stop -t 60 app
app
cd /pups && git pull && git checkout v1.0.3 && /pups/bin/pups --stdin
Already up to date.
Note: checking out 'v1.0.3'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at d1db030 cut a new version
I, [2021-12-22T12:17:14.375608 #1]  INFO -- : Loading --stdin
I, [2021-12-22T12:17:14.380128 #1]  INFO -- : > locale-gen $LANG && update-locale
I, [2021-12-22T12:17:14.401507 #1]  INFO -- : Generating locales (this might take a while)...
Generation complete.

I, [2021-12-22T12:17:14.401656 #1]  INFO -- : > mkdir -p /shared/postgres_run
I, [2021-12-22T12:17:14.403445 #1]  INFO -- : 
I, [2021-12-22T12:17:14.403608 #1]  INFO -- : > chown postgres:postgres /shared/postgres_run
I, [2021-12-22T12:17:14.405191 #1]  INFO -- : 
I, [2021-12-22T12:17:14.405358 #1]  INFO -- : > chmod 775 /shared/postgres_run
I, [2021-12-22T12:17:14.406697 #1]  INFO -- : 
I, [2021-12-22T12:17:14.406802 #1]  INFO -- : > rm -fr /var/run/postgresql
I, [2021-12-22T12:17:14.408253 #1]  INFO -- : 
I, [2021-12-22T12:17:14.408361 #1]  INFO -- : > ln -s /shared/postgres_run /var/run/postgresql
I, [2021-12-22T12:17:14.412388 #1]  INFO -- : 
I, [2021-12-22T12:17:14.412496 #1]  INFO -- : > socat /dev/null UNIX-CONNECT:/shared/postgres_run/.s.PGSQL.5432 || exit 0 && echo postgres already running stop container ; exit 1
2021/12/22 12:17:14 socat[27] E connect(6, AF=1 "/shared/postgres_run/.s.PGSQL.5432", 36): Connection refused
I, [2021-12-22T12:17:14.415572 #1]  INFO -- : 
I, [2021-12-22T12:17:14.415677 #1]  INFO -- : > rm -fr /shared/postgres_run/.s*
I, [2021-12-22T12:17:14.417476 #1]  INFO -- : 
I, [2021-12-22T12:17:14.417589 #1]  INFO -- : > rm -fr /shared/postgres_run/*.pid
I, [2021-12-22T12:17:14.419317 #1]  INFO -- : 
I, [2021-12-22T12:17:14.419428 #1]  INFO -- : > mkdir -p /shared/postgres_run/13-main.pg_stat_tmp
I, [2021-12-22T12:17:14.420864 #1]  INFO -- : 
I, [2021-12-22T12:17:14.421005 #1]  INFO -- : > chown postgres:postgres /shared/postgres_run/13-main.pg_stat_tmp
I, [2021-12-22T12:17:14.422356 #1]  INFO -- : 
I, [2021-12-22T12:17:14.425488 #1]  INFO -- : File > /etc/service/postgres/run  chmod: +x  chown: 
I, [2021-12-22T12:17:14.428251 #1]  INFO -- : File > /etc/service/postgres/log/run  chmod: +x  chown: 
I, [2021-12-22T12:17:14.431100 #1]  INFO -- : File > /etc/runit/3.d/99-postgres  chmod: +x  chown: 
I, [2021-12-22T12:17:14.433890 #1]  INFO -- : File > /root/upgrade_postgres  chmod: +x  chown: 
I, [2021-12-22T12:17:14.434069 #1]  INFO -- : > chown -R root /var/lib/postgresql/13/main
I, [2021-12-22T12:17:56.907961 #1]  INFO -- : 
I, [2021-12-22T12:17:56.908121 #1]  INFO -- : > [ ! -e /shared/postgres_data ] && install -d -m 0755 -o postgres -g postgres /shared/postgres_data && sudo -E -u postgres /usr/lib/postgresql/13/bin/initdb -D /shared/postgres_data || exit 0
I, [2021-12-22T12:17:56.909780 #1]  INFO -- : 
I, [2021-12-22T12:17:56.909834 #1]  INFO -- : > chown -R postgres:postgres /shared/postgres_data
I, [2021-12-22T12:17:56.924298 #1]  INFO -- : 
I, [2021-12-22T12:17:56.924455 #1]  INFO -- : > chown -R postgres:postgres /var/run/postgresql
I, [2021-12-22T12:17:56.926015 #1]  INFO -- : 
I, [2021-12-22T12:17:56.926194 #1]  INFO -- : > /root/upgrade_postgres
I, [2021-12-22T12:17:56.929052 #1]  INFO -- : 
I, [2021-12-22T12:17:56.929149 #1]  INFO -- : > rm /root/upgrade_postgres
I, [2021-12-22T12:17:56.930512 #1]  INFO -- : 
I, [2021-12-22T12:17:56.930689 #1]  INFO -- : Replacing data_directory = '/var/lib/postgresql/13/main' with data_directory = '/shared/postgres_data' in /etc/postgresql/13/main/postgresql.conf
I, [2021-12-22T12:17:56.931044 #1]  INFO -- : Replacing (?-mix:#?listen_addresses *=.*) with listen_addresses = '*' in /etc/postgresql/13/main/postgresql.conf
I, [2021-12-22T12:17:56.931281 #1]  INFO -- : Replacing (?-mix:#?synchronous_commit *=.*) with synchronous_commit = $db_synchronous_commit in /etc/postgresql/13/main/postgresql.conf
I, [2021-12-22T12:17:56.931519 #1]  INFO -- : Replacing (?-mix:#?shared_buffers *=.*) with shared_buffers = $db_shared_buffers in /etc/postgresql/13/main/postgresql.conf
I, [2021-12-22T12:17:56.931779 #1]  INFO -- : Replacing (?-mix:#?work_mem *=.*) with work_mem = $db_work_mem in /etc/postgresql/13/main/postgresql.conf
I, [2021-12-22T12:17:56.932014 #1]  INFO -- : Replacing (?-mix:#?default_text_search_config *=.*) with default_text_search_config = '$db_default_text_search_config' in /etc/postgresql/13/main/postgresql.conf
I, [2021-12-22T12:17:56.932277 #1]  INFO -- : > install -d -m 0755 -o postgres -g postgres /shared/postgres_backup
I, [2021-12-22T12:17:56.934064 #1]  INFO -- : 
I, [2021-12-22T12:17:56.934292 #1]  INFO -- : Replacing (?-mix:#?checkpoint_segments *=.*) with checkpoint_segments = $db_checkpoint_segments in /etc/postgresql/13/main/postgresql.conf
I, [2021-12-22T12:17:56.934531 #1]  INFO -- : Replacing (?-mix:#?logging_collector *=.*) with logging_collector = $db_logging_collector in /etc/postgresql/13/main/postgresql.conf
I, [2021-12-22T12:17:56.934769 #1]  INFO -- : Replacing (?-mix:#?log_min_duration_statement *=.*) with log_min_duration_statement = $db_log_min_duration_statement in /etc/postgresql/13/main/postgresql.conf
I, [2021-12-22T12:17:56.935003 #1]  INFO -- : Replacing (?-mix:^#local +replication +postgres +peer$) with local replication postgres  peer in /etc/postgresql/13/main/pg_hba.conf
I, [2021-12-22T12:17:56.935192 #1]  INFO -- : Replacing (?-mix:^host.*all.*all.*127.*$) with host all all 0.0.0.0/0 md5 in /etc/postgresql/13/main/pg_hba.conf
I, [2021-12-22T12:17:56.935351 #1]  INFO -- : Replacing (?-mix:^host.*all.*all.*::1\/128.*$) with host all all ::/0 md5 in /etc/postgresql/13/main/pg_hba.conf
I, [2021-12-22T12:17:56.935501 #1]  INFO -- : > HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main
I, [2021-12-22T12:17:56.936492 #1]  INFO -- : > sleep 5
2021-12-22 12:17:57.065 UTC [50] LOG:  starting PostgreSQL 13.5 (Debian 13.5-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
2021-12-22 12:17:57.065 UTC [50] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2021-12-22 12:17:57.065 UTC [50] LOG:  listening on IPv6 address "::", port 5432
2021-12-22 12:17:57.133 UTC [50] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2021-12-22 12:17:57.201 UTC [53] LOG:  database system shutdown was interrupted; last known up at 2021-12-22 12:16:34 UTC
2021-12-22 12:17:57.810 UTC [53] LOG:  database system was not properly shut down; automatic recovery in progress
2021-12-22 12:17:57.845 UTC [53] LOG:  redo starts at 83/B12EDF50
2021-12-22 12:17:58.554 UTC [53] LOG:  invalid record length at 83/BA520118: wanted 24, got 0
2021-12-22 12:17:58.554 UTC [53] LOG:  redo done at 83/BA5200F0
I, [2021-12-22T12:18:01.937956 #1]  INFO -- : 
I, [2021-12-22T12:18:01.938274 #1]  INFO -- : > su postgres -c 'createdb discourse' || true
2021-12-22 12:18:01.968 UTC [57] postgres@postgres FATAL:  the database system is starting up
2021-12-22 12:18:01.969 UTC [58] postgres@template1 FATAL:  the database system is starting up
createdb: error: could not connect to database template1: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL:  the database system is starting up
I, [2021-12-22T12:18:01.970057 #1]  INFO -- : 
I, [2021-12-22T12:18:01.970258 #1]  INFO -- : > su postgres -c 'psql discourse -c "create user discourse;"' || true
2021-12-22 12:18:02.000 UTC [62] postgres@discourse FATAL:  the database system is starting up
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL:  the database system is starting up
I, [2021-12-22T12:18:02.001459 #1]  INFO -- : 
I, [2021-12-22T12:18:02.001709 #1]  INFO -- : > su postgres -c 'psql discourse -c "grant all privileges on database discourse to discourse;"' || true
2021-12-22 12:18:02.033 UTC [66] postgres@discourse FATAL:  the database system is starting up
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL:  the database system is starting up
I, [2021-12-22T12:18:02.034185 #1]  INFO -- : 
I, [2021-12-22T12:18:02.034419 #1]  INFO -- : > su postgres -c 'psql discourse -c "alter schema public owner to discourse;"'
2021-12-22 12:18:02.064 UTC [70] postgres@discourse FATAL:  the database system is starting up
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL:  the database system is starting up
I, [2021-12-22T12:18:02.065753 #1]  INFO -- : 
I, [2021-12-22T12:18:02.066059 #1]  INFO -- : Terminating async processes
I, [2021-12-22T12:18:02.066082 #1]  INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main pid: 50
2021-12-22 12:18:02.066 UTC [50] LOG:  received fast shutdown request
I, [2021-12-22T12:18:12.066665 #1]  INFO -- : HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main pid:50 did not terminate cleanly, forcing termination!


FAILED
--------------------
Pups::ExecError: su postgres -c 'psql discourse -c "alter schema public owner to discourse;"' failed with return #<Process::Status: pid 67 exit 2>
Location of failure: /pups/lib/pups/exec_command.rb:112:in `spawn'
exec failed with the params "su postgres -c 'psql $db_name -c \"alter schema public owner to $db_user;\"'"
c05642e47b7cc5a7871faa3bf73549342017cbe76da749c7fb23bcb40da0b2dc
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.
kinetiksoft:~$ 
Session was closed

It’s not shutting down, it’s not starting up properly because the redo log is damaged. If you have a backup you should consider recovering it on a blank instance.

But the instance is working fine overall, it starts behaving like that on rebuild only. What can we do to rebuild the forum just as we did multiple times before?

Whenever this happens, we usually (temporarily, as otherwise Git gets confused when updating the next time) replace the sleep 5 with a few more seconds so that Postgres has more time to settle/recover.

If you’re including a different postgres.template.yml, of course, edit that one instead.

1 Like

That worked flawlessly. Thank you!