Yesterday we tried rebuilding the forum several times with the same result - it failed, and log shows it is related to how Discourse shuts the database down. It looks like it doesn’t match the speed expectations set by rebuild process and that’s why it fails. Last time it happened, we just tried to re-build once again and it worked. Yesterday, though, we tried that 3 times with the same result. Starting up the instance helped, but that doesn’t solve the problem of inability to upgrade the forum by rebuilding. I’m including the rebuild log below. Hope someone will be able to help us getting to the real reason and how can we help Discourse with rebuilding itself. Rebuild failed log kinetiksoft:~$ sudo /usr/local/bin/discourse-tool git-pull remote: Enumerating objects: 47, done. remote: Counting objects: 100% (47/47), done. remote: Compressing objects: 100% (14/14), done. remote: Total 47 (delta 20), reused 41 (delta 19), pack-reused 0 Unpacking objects: 100% (47/47), done. From https://github.com/discourse/discourse_docker 02cd746..4cd6a5d main -> origin/main Updating 02cd746..4cd6a5d Fast-forward .github/workflows/build.yml | 85 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-------------- image/auto_build.rb | 4 ++-- image/base/release.Dockerfile | 11 +++++++++++ image/base/{Dockerfile => slim.Dockerfile} | 8 +------- image/discourse_test/Dockerfile | 40 +++++++++++++++++++++------------------- 5 files changed, 106 insertions(+), 42 deletions(-) create mode 100644 image/base/release.Dockerfile rename image/base/{Dockerfile => slim.Dockerfile} (94%) kinetiksoft:~$ sudo /usr/local/bin/discourse-tool rebuild x86_64 arch detected. Ensuring launcher is up to date Fetching origin Launcher is up-to-date Stopping old container + /usr/bin/docker stop -t 60 app app cd /pups && git pull && git checkout v1.0.3 && /pups/bin/pups --stdin Already up to date. Note: checking out 'v1.0.3'. You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by performing another checkout. If you want to create a new branch to retain commits you create, you may do so (now or later) by using -b with the checkout command again. Example: git checkout -b HEAD is now at d1db030 cut a new version I, [2021-12-22T12:03:30.961687 #1] INFO -- : Loading --stdin I, [2021-12-22T12:03:30.965617 #1] INFO -- : > locale-gen $LANG && update-locale I, [2021-12-22T12:03:31.025039 #1] INFO -- : Generating locales (this might take a while)... Generation complete. I, [2021-12-22T12:03:31.025205 #1] INFO -- : > mkdir -p /shared/postgres_run I, [2021-12-22T12:03:31.027383 #1] INFO -- : I, [2021-12-22T12:03:31.027600 #1] INFO -- : > chown postgres:postgres /shared/postgres_run I, [2021-12-22T12:03:31.034332 #1] INFO -- : I, [2021-12-22T12:03:31.034470 #1] INFO -- : > chmod 775 /shared/postgres_run I, [2021-12-22T12:03:31.035865 #1] INFO -- : I, [2021-12-22T12:03:31.035982 #1] INFO -- : > rm -fr /var/run/postgresql I, [2021-12-22T12:03:31.037608 #1] INFO -- : I, [2021-12-22T12:03:31.037725 #1] INFO -- : > ln -s /shared/postgres_run /var/run/postgresql I, [2021-12-22T12:03:31.041565 #1] INFO -- : I, [2021-12-22T12:03:31.041696 #1] INFO -- : > socat /dev/null UNIX-CONNECT:/shared/postgres_run/.s.PGSQL.5432 || exit 0 && echo postgres already running stop container ; exit 1 2021/12/22 12:03:31 socat[27] E connect(6, AF=1 "/shared/postgres_run/.s.PGSQL.5432", 36): Connection refused I, [2021-12-22T12:03:31.053403 #1] INFO -- : I, [2021-12-22T12:03:31.053535 #1] INFO -- : > rm -fr /shared/postgres_run/.s* I, [2021-12-22T12:03:31.055582 #1] INFO -- : I, [2021-12-22T12:03:31.055693 #1] INFO -- : > rm -fr /shared/postgres_run/*.pid I, [2021-12-22T12:03:31.057512 #1] INFO -- : I, [2021-12-22T12:03:31.057618 #1] INFO -- : > mkdir -p /shared/postgres_run/13-main.pg_stat_tmp I, [2021-12-22T12:03:31.058997 #1] INFO -- : I, [2021-12-22T12:03:31.059130 #1] INFO -- : > chown postgres:postgres /shared/postgres_run/13-main.pg_stat_tmp I, [2021-12-22T12:03:31.060530 #1] INFO -- : I, [2021-12-22T12:03:31.063566 #1] INFO -- : File > /etc/service/postgres/run chmod: +x chown: I, [2021-12-22T12:03:31.066547 #1] INFO -- : File > /etc/service/postgres/log/run chmod: +x chown: I, [2021-12-22T12:03:31.069511 #1] INFO -- : File > /etc/runit/3.d/99-postgres chmod: +x chown: I, [2021-12-22T12:03:31.072380 #1] INFO -- : File > /root/upgrade_postgres chmod: +x chown: I, [2021-12-22T12:03:31.072566 #1] INFO -- : > chown -R root /var/lib/postgresql/13/main I, [2021-12-22T12:04:08.248631 #1] INFO -- : I, [2021-12-22T12:04:08.248799 #1] INFO -- : > [ ! -e /shared/postgres_data ] && install -d -m 0755 -o postgres -g postgres /shared/postgres_data && sudo -E -u postgres /usr/lib/postgresql/13/bin/initdb -D /shared/postgres_data || exit 0 I, [2021-12-22T12:04:08.251062 #1] INFO -- : I, [2021-12-22T12:04:08.251148 #1] INFO -- : > chown -R postgres:postgres /shared/postgres_data I, [2021-12-22T12:04:08.291023 #1] INFO -- : I, [2021-12-22T12:04:08.291212 #1] INFO -- : > chown -R postgres:postgres /var/run/postgresql I, [2021-12-22T12:04:08.293474 #1] INFO -- : I, [2021-12-22T12:04:08.293655 #1] INFO -- : > /root/upgrade_postgres I, [2021-12-22T12:04:08.301217 #1] INFO -- : I, [2021-12-22T12:04:08.301351 #1] INFO -- : > rm /root/upgrade_postgres I, [2021-12-22T12:04:08.303272 #1] INFO -- : I, [2021-12-22T12:04:08.310457 #1] INFO -- : Replacing data_directory = '/var/lib/postgresql/13/main' with data_directory = '/shared/postgres_data' in /etc/postgresql/13/main/postgresql.conf I, [2021-12-22T12:04:08.310879 #1] INFO -- : Replacing (?-mix:#?listen_addresses *=.*) with listen_addresses = '*' in /etc/postgresql/13/main/postgresql.conf I, [2021-12-22T12:04:08.318852 #1] INFO -- : Replacing (?-mix:#?synchronous_commit *=.*) with synchronous_commit = $db_synchronous_commit in /etc/postgresql/13/main/postgresql.conf I, [2021-12-22T12:04:08.319207 #1] INFO -- : Replacing (?-mix:#?shared_buffers *=.*) with shared_buffers = $db_shared_buffers in /etc/postgresql/13/main/postgresql.conf I, [2021-12-22T12:04:08.319512 #1] INFO -- : Replacing (?-mix:#?work_mem *=.*) with work_mem = $db_work_mem in /etc/postgresql/13/main/postgresql.conf I, [2021-12-22T12:04:08.319791 #1] INFO -- : Replacing (?-mix:#?default_text_search_config *=.*) with default_text_search_config = '$db_default_text_search_config' in /etc/postgresql/13/main/postgresql.conf I, [2021-12-22T12:04:08.320079 #1] INFO -- : > install -d -m 0755 -o postgres -g postgres /shared/postgres_backup I, [2021-12-22T12:04:08.330091 #1] INFO -- : I, [2021-12-22T12:04:08.330365 #1] INFO -- : Replacing (?-mix:#?checkpoint_segments *=.*) with checkpoint_segments = $db_checkpoint_segments in /etc/postgresql/13/main/postgresql.conf I, [2021-12-22T12:04:08.330681 #1] INFO -- : Replacing (?-mix:#?logging_collector *=.*) with logging_collector = $db_logging_collector in /etc/postgresql/13/main/postgresql.conf I, [2021-12-22T12:04:08.331278 #1] INFO -- : Replacing (?-mix:#?log_min_duration_statement *=.*) with log_min_duration_statement = $db_log_min_duration_statement in /etc/postgresql/13/main/postgresql.conf I, [2021-12-22T12:04:08.335199 #1] INFO -- : Replacing (?-mix:^#local +replication +postgres +peer$) with local replication postgres peer in /etc/postgresql/13/main/pg_hba.conf I, [2021-12-22T12:04:08.335399 #1] INFO -- : Replacing (?-mix:^host.*all.*all.*127.*$) with host all all 0.0.0.0/0 md5 in /etc/postgresql/13/main/pg_hba.conf I, [2021-12-22T12:04:08.343492 #1] INFO -- : Replacing (?-mix:^host.*all.*all.*::1\/128.*$) with host all all ::/0 md5 in /etc/postgresql/13/main/pg_hba.conf I, [2021-12-22T12:04:08.343701 #1] INFO -- : > HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main I, [2021-12-22T12:04:08.345307 #1] INFO -- : > sleep 5 2021-12-22 12:04:08.666 UTC [50] LOG: starting PostgreSQL 13.5 (Debian 13.5-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit 2021-12-22 12:04:08.666 UTC [50] LOG: listening on IPv4 address "0.0.0.0", port 5432 2021-12-22 12:04:08.666 UTC [50] LOG: listening on IPv6 address "::", port 5432 2021-12-22 12:04:08.732 UTC [50] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" 2021-12-22 12:04:08.808 UTC [53] LOG: database system shutdown was interrupted; last known up at 2021-12-22 12:02:56 UTC 2021-12-22 12:04:09.928 UTC [53] LOG: database system was not properly shut down; automatic recovery in progress 2021-12-22 12:04:09.972 UTC [53] LOG: redo starts at 83/B07EB758 2021-12-22 12:04:10.123 UTC [53] LOG: invalid record length at 83/B12EDED8: wanted 24, got 0 2021-12-22 12:04:10.123 UTC [53] LOG: redo done at 83/B12EDEA0 I, [2021-12-22T12:04:13.347316 #1] INFO -- : I, [2021-12-22T12:04:13.347566 #1] INFO -- : > su postgres -c 'createdb discourse' || true 2021-12-22 12:04:13.521 UTC [57] postgres@postgres FATAL: the database system is starting up 2021-12-22 12:04:13.522 UTC [58] postgres@template1 FATAL: the database system is starting up createdb: error: could not connect to database template1: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL: the database system is starting up I, [2021-12-22T12:04:13.523332 #1] INFO -- : I, [2021-12-22T12:04:13.523518 #1] INFO -- : > su postgres -c 'psql discourse -c "create user discourse;"' || true 2021-12-22 12:04:13.791 UTC [62] postgres@discourse FATAL: the database system is starting up psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL: the database system is starting up I, [2021-12-22T12:04:13.792674 #1] INFO -- : I, [2021-12-22T12:04:13.792923 #1] INFO -- : > su postgres -c 'psql discourse -c "grant all privileges on database discourse to discourse;"' || true 2021-12-22 12:04:13.824 UTC [66] postgres@discourse FATAL: the database system is starting up psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL: the database system is starting up I, [2021-12-22T12:04:13.825180 #1] INFO -- : I, [2021-12-22T12:04:13.825452 #1] INFO -- : > su postgres -c 'psql discourse -c "alter schema public owner to discourse;"' 2021-12-22 12:04:13.856 UTC [70] postgres@discourse FATAL: the database system is starting up psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL: the database system is starting up I, [2021-12-22T12:04:13.857672 #1] INFO -- : I, [2021-12-22T12:04:13.858028 #1] INFO -- : Terminating async processes I, [2021-12-22T12:04:13.858053 #1] INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main pid: 50 2021-12-22 12:04:13.858 UTC [50] LOG: received fast shutdown request I, [2021-12-22T12:04:23.858557 #1] INFO -- : HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main pid:50 did not terminate cleanly, forcing termination! FAILED -------------------- Pups::ExecError: su postgres -c 'psql discourse -c "alter schema public owner to discourse;"' failed with return # Location of failure: /pups/lib/pups/exec_command.rb:112:in `spawn' exec failed with the params "su postgres -c 'psql $db_name -c \"alter schema public owner to $db_user;\"'" 08af256a113e238f69588774fcfed142d94742fc4f09c9182cb20c1901a7aa61 ** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one. ./discourse-doctor may help diagnose the problem. kinetiksoft:~$ sudo /usr/local/bin/discourse-tool rebuild x86_64 arch detected. Ensuring launcher is up to date Fetching origin Launcher is up-to-date Stopping old container + /usr/bin/docker stop -t 60 app app cd /pups && git pull && git checkout v1.0.3 && /pups/bin/pups --stdin Already up to date. Note: checking out 'v1.0.3'. You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by performing another checkout. If you want to create a new branch to retain commits you create, you may do so (now or later) by using -b with the checkout command again. Example: git checkout -b HEAD is now at d1db030 cut a new version I, [2021-12-22T12:05:48.204969 #1] INFO -- : Loading --stdin I, [2021-12-22T12:05:48.209104 #1] INFO -- : > locale-gen $LANG && update-locale I, [2021-12-22T12:05:48.229337 #1] INFO -- : Generating locales (this might take a while)... Generation complete. I, [2021-12-22T12:05:48.229510 #1] INFO -- : > mkdir -p /shared/postgres_run I, [2021-12-22T12:05:48.231324 #1] INFO -- : I, [2021-12-22T12:05:48.231513 #1] INFO -- : > chown postgres:postgres /shared/postgres_run I, [2021-12-22T12:05:48.233015 #1] INFO -- : I, [2021-12-22T12:05:48.233122 #1] INFO -- : > chmod 775 /shared/postgres_run I, [2021-12-22T12:05:48.236585 #1] INFO -- : I, [2021-12-22T12:05:48.236706 #1] INFO -- : > rm -fr /var/run/postgresql I, [2021-12-22T12:05:48.238156 #1] INFO -- : I, [2021-12-22T12:05:48.238287 #1] INFO -- : > ln -s /shared/postgres_run /var/run/postgresql I, [2021-12-22T12:05:48.239526 #1] INFO -- : I, [2021-12-22T12:05:48.239634 #1] INFO -- : > socat /dev/null UNIX-CONNECT:/shared/postgres_run/.s.PGSQL.5432 || exit 0 && echo postgres already running stop container ; exit 1 2021/12/22 12:05:48 socat[27] E connect(6, AF=1 "/shared/postgres_run/.s.PGSQL.5432", 36): Connection refused I, [2021-12-22T12:05:48.242590 #1] INFO -- : I, [2021-12-22T12:05:48.242720 #1] INFO -- : > rm -fr /shared/postgres_run/.s* I, [2021-12-22T12:05:48.244501 #1] INFO -- : I, [2021-12-22T12:05:48.244589 #1] INFO -- : > rm -fr /shared/postgres_run/*.pid I, [2021-12-22T12:05:48.246303 #1] INFO -- : I, [2021-12-22T12:05:48.246397 #1] INFO -- : > mkdir -p /shared/postgres_run/13-main.pg_stat_tmp I, [2021-12-22T12:05:48.247818 #1] INFO -- : I, [2021-12-22T12:05:48.248021 #1] INFO -- : > chown postgres:postgres /shared/postgres_run/13-main.pg_stat_tmp I, [2021-12-22T12:05:48.249386 #1] INFO -- : I, [2021-12-22T12:05:48.252499 #1] INFO -- : File > /etc/service/postgres/run chmod: +x chown: I, [2021-12-22T12:05:48.255331 #1] INFO -- : File > /etc/service/postgres/log/run chmod: +x chown: I, [2021-12-22T12:05:48.258241 #1] INFO -- : File > /etc/runit/3.d/99-postgres chmod: +x chown: I, [2021-12-22T12:05:48.261139 #1] INFO -- : File > /root/upgrade_postgres chmod: +x chown: I, [2021-12-22T12:05:48.261348 #1] INFO -- : > chown -R root /var/lib/postgresql/13/main I, [2021-12-22T12:06:24.321828 #1] INFO -- : I, [2021-12-22T12:06:24.321951 #1] INFO -- : > [ ! -e /shared/postgres_data ] && install -d -m 0755 -o postgres -g postgres /shared/postgres_data && sudo -E -u postgres /usr/lib/postgresql/13/bin/initdb -D /shared/postgres_data || exit 0 I, [2021-12-22T12:06:24.323903 #1] INFO -- : I, [2021-12-22T12:06:24.323952 #1] INFO -- : > chown -R postgres:postgres /shared/postgres_data I, [2021-12-22T12:06:24.339115 #1] INFO -- : I, [2021-12-22T12:06:24.339247 #1] INFO -- : > chown -R postgres:postgres /var/run/postgresql I, [2021-12-22T12:06:24.340865 #1] INFO -- : I, [2021-12-22T12:06:24.341031 #1] INFO -- : > /root/upgrade_postgres I, [2021-12-22T12:06:24.343922 #1] INFO -- : I, [2021-12-22T12:06:24.344019 #1] INFO -- : > rm /root/upgrade_postgres I, [2021-12-22T12:06:24.345316 #1] INFO -- : I, [2021-12-22T12:06:24.345502 #1] INFO -- : Replacing data_directory = '/var/lib/postgresql/13/main' with data_directory = '/shared/postgres_data' in /etc/postgresql/13/main/postgresql.conf I, [2021-12-22T12:06:24.345857 #1] INFO -- : Replacing (?-mix:#?listen_addresses *=.*) with listen_addresses = '*' in /etc/postgresql/13/main/postgresql.conf I, [2021-12-22T12:06:24.346104 #1] INFO -- : Replacing (?-mix:#?synchronous_commit *=.*) with synchronous_commit = $db_synchronous_commit in /etc/postgresql/13/main/postgresql.conf I, [2021-12-22T12:06:24.346356 #1] INFO -- : Replacing (?-mix:#?shared_buffers *=.*) with shared_buffers = $db_shared_buffers in /etc/postgresql/13/main/postgresql.conf I, [2021-12-22T12:06:24.346628 #1] INFO -- : Replacing (?-mix:#?work_mem *=.*) with work_mem = $db_work_mem in /etc/postgresql/13/main/postgresql.conf I, [2021-12-22T12:06:24.346891 #1] INFO -- : Replacing (?-mix:#?default_text_search_config *=.*) with default_text_search_config = '$db_default_text_search_config' in /etc/postgresql/13/main/postgresql.conf I, [2021-12-22T12:06:24.347165 #1] INFO -- : > install -d -m 0755 -o postgres -g postgres /shared/postgres_backup I, [2021-12-22T12:06:24.348910 #1] INFO -- : I, [2021-12-22T12:06:24.349140 #1] INFO -- : Replacing (?-mix:#?checkpoint_segments *=.*) with checkpoint_segments = $db_checkpoint_segments in /etc/postgresql/13/main/postgresql.conf I, [2021-12-22T12:06:24.349383 #1] INFO -- : Replacing (?-mix:#?logging_collector *=.*) with logging_collector = $db_logging_collector in /etc/postgresql/13/main/postgresql.conf I, [2021-12-22T12:06:24.349678 #1] INFO -- : Replacing (?-mix:#?log_min_duration_statement *=.*) with log_min_duration_statement = $db_log_min_duration_statement in /etc/postgresql/13/main/postgresql.conf I, [2021-12-22T12:06:24.349972 #1] INFO -- : Replacing (?-mix:^#local +replication +postgres +peer$) with local replication postgres peer in /etc/postgresql/13/main/pg_hba.conf I, [2021-12-22T12:06:24.350138 #1] INFO -- : Replacing (?-mix:^host.*all.*all.*127.*$) with host all all 0.0.0.0/0 md5 in /etc/postgresql/13/main/pg_hba.conf I, [2021-12-22T12:06:24.350303 #1] INFO -- : Replacing (?-mix:^host.*all.*all.*::1\/128.*$) with host all all ::/0 md5 in /etc/postgresql/13/main/pg_hba.conf I, [2021-12-22T12:06:24.350464 #1] INFO -- : > HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main I, [2021-12-22T12:06:24.351392 #1] INFO -- : > sleep 5 2021-12-22 12:06:24.490 UTC [50] LOG: starting PostgreSQL 13.5 (Debian 13.5-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit 2021-12-22 12:06:24.490 UTC [50] LOG: listening on IPv4 address "0.0.0.0", port 5432 2021-12-22 12:06:24.490 UTC [50] LOG: listening on IPv6 address "::", port 5432 2021-12-22 12:06:24.571 UTC [50] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" 2021-12-22 12:06:24.637 UTC [53] LOG: database system shutdown was interrupted; last known up at 2021-12-22 12:04:10 UTC 2021-12-22 12:06:26.793 UTC [53] LOG: database system was not properly shut down; automatic recovery in progress 2021-12-22 12:06:26.827 UTC [53] LOG: redo starts at 83/B07EB758 2021-12-22 12:06:26.852 UTC [53] LOG: invalid record length at 83/B12EDED8: wanted 24, got 0 2021-12-22 12:06:26.852 UTC [53] LOG: redo done at 83/B12EDEA0 I, [2021-12-22T12:06:29.352869 #1] INFO -- : I, [2021-12-22T12:06:29.353167 #1] INFO -- : > su postgres -c 'createdb discourse' || true 2021-12-22 12:06:29.385 UTC [57] postgres@postgres FATAL: the database system is starting up 2021-12-22 12:06:29.385 UTC [58] postgres@template1 FATAL: the database system is starting up createdb: error: could not connect to database template1: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL: the database system is starting up I, [2021-12-22T12:06:29.386701 #1] INFO -- : I, [2021-12-22T12:06:29.386897 #1] INFO -- : > su postgres -c 'psql discourse -c "create user discourse;"' || true 2021-12-22 12:06:29.417 UTC [62] postgres@discourse FATAL: the database system is starting up psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL: the database system is starting up I, [2021-12-22T12:06:29.418609 #1] INFO -- : I, [2021-12-22T12:06:29.418798 #1] INFO -- : > su postgres -c 'psql discourse -c "grant all privileges on database discourse to discourse;"' || true 2021-12-22 12:06:29.453 UTC [66] postgres@discourse FATAL: the database system is starting up psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL: the database system is starting up I, [2021-12-22T12:06:29.454838 #1] INFO -- : I, [2021-12-22T12:06:29.455124 #1] INFO -- : > su postgres -c 'psql discourse -c "alter schema public owner to discourse;"' 2021-12-22 12:06:29.486 UTC [70] postgres@discourse FATAL: the database system is starting up psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL: the database system is starting up I, [2021-12-22T12:06:29.487598 #1] INFO -- : I, [2021-12-22T12:06:29.487929 #1] INFO -- : Terminating async processes I, [2021-12-22T12:06:29.487965 #1] INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main pid: 50 2021-12-22 12:06:29.488 UTC [50] LOG: received fast shutdown request I, [2021-12-22T12:06:39.488426 #1] INFO -- : HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main pid:50 did not terminate cleanly, forcing termination! FAILED -------------------- Pups::ExecError: su postgres -c 'psql discourse -c "alter schema public owner to discourse;"' failed with return # Location of failure: /pups/lib/pups/exec_command.rb:112:in `spawn' exec failed with the params "su postgres -c 'psql $db_name -c \"alter schema public owner to $db_user;\"'" 7fee51af158f0ed17bf212e936826b00650dd82d1825ff48889479f7c0d3fade ** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one. ./discourse-doctor may help diagnose the problem. kinetiksoft:~$ sudo /usr/local/bin/discourse-tool rebuild x86_64 arch detected. Ensuring launcher is up to date Fetching origin Launcher is up-to-date Stopping old container + /usr/bin/docker stop -t 60 app app cd /pups && git pull && git checkout v1.0.3 && /pups/bin/pups --stdin Already up to date. Note: checking out 'v1.0.3'. You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by performing another checkout. If you want to create a new branch to retain commits you create, you may do so (now or later) by using -b with the checkout command again. Example: git checkout -b HEAD is now at d1db030 cut a new version I, [2021-12-22T12:08:33.310010 #1] INFO -- : Loading --stdin I, [2021-12-22T12:08:33.313979 #1] INFO -- : > locale-gen $LANG && update-locale I, [2021-12-22T12:08:33.333943 #1] INFO -- : Generating locales (this might take a while)... Generation complete. I, [2021-12-22T12:08:33.334119 #1] INFO -- : > mkdir -p /shared/postgres_run I, [2021-12-22T12:08:33.336004 #1] INFO -- : I, [2021-12-22T12:08:33.336149 #1] INFO -- : > chown postgres:postgres /shared/postgres_run I, [2021-12-22T12:08:33.337638 #1] INFO -- : I, [2021-12-22T12:08:33.337777 #1] INFO -- : > chmod 775 /shared/postgres_run I, [2021-12-22T12:08:33.339209 #1] INFO -- : I, [2021-12-22T12:08:33.339293 #1] INFO -- : > rm -fr /var/run/postgresql I, [2021-12-22T12:08:33.340714 #1] INFO -- : I, [2021-12-22T12:08:33.340818 #1] INFO -- : > ln -s /shared/postgres_run /var/run/postgresql I, [2021-12-22T12:08:33.342042 #1] INFO -- : I, [2021-12-22T12:08:33.342136 #1] INFO -- : > socat /dev/null UNIX-CONNECT:/shared/postgres_run/.s.PGSQL.5432 || exit 0 && echo postgres already running stop container ; exit 1 2021/12/22 12:08:33 socat[27] E connect(6, AF=1 "/shared/postgres_run/.s.PGSQL.5432", 36): Connection refused I, [2021-12-22T12:08:33.344996 #1] INFO -- : I, [2021-12-22T12:08:33.345091 #1] INFO -- : > rm -fr /shared/postgres_run/.s* I, [2021-12-22T12:08:33.346977 #1] INFO -- : I, [2021-12-22T12:08:33.347081 #1] INFO -- : > rm -fr /shared/postgres_run/*.pid I, [2021-12-22T12:08:33.348866 #1] INFO -- : I, [2021-12-22T12:08:33.348953 #1] INFO -- : > mkdir -p /shared/postgres_run/13-main.pg_stat_tmp I, [2021-12-22T12:08:33.350474 #1] INFO -- : I, [2021-12-22T12:08:33.350644 #1] INFO -- : > chown postgres:postgres /shared/postgres_run/13-main.pg_stat_tmp I, [2021-12-22T12:08:33.352043 #1] INFO -- : I, [2021-12-22T12:08:33.355013 #1] INFO -- : File > /etc/service/postgres/run chmod: +x chown: I, [2021-12-22T12:08:33.357838 #1] INFO -- : File > /etc/service/postgres/log/run chmod: +x chown: I, [2021-12-22T12:08:33.360727 #1] INFO -- : File > /etc/runit/3.d/99-postgres chmod: +x chown: I, [2021-12-22T12:08:33.364940 #1] INFO -- : File > /root/upgrade_postgres chmod: +x chown: I, [2021-12-22T12:08:33.365242 #1] INFO -- : > chown -R root /var/lib/postgresql/13/main I, [2021-12-22T12:09:12.888922 #1] INFO -- : I, [2021-12-22T12:09:12.889060 #1] INFO -- : > [ ! -e /shared/postgres_data ] && install -d -m 0755 -o postgres -g postgres /shared/postgres_data && sudo -E -u postgres /usr/lib/postgresql/13/bin/initdb -D /shared/postgres_data || exit 0 I, [2021-12-22T12:09:12.890602 #1] INFO -- : I, [2021-12-22T12:09:12.890637 #1] INFO -- : > chown -R postgres:postgres /shared/postgres_data I, [2021-12-22T12:09:12.905164 #1] INFO -- : I, [2021-12-22T12:09:12.905282 #1] INFO -- : > chown -R postgres:postgres /var/run/postgresql I, [2021-12-22T12:09:12.907055 #1] INFO -- : I, [2021-12-22T12:09:12.907187 #1] INFO -- : > /root/upgrade_postgres I, [2021-12-22T12:09:12.910221 #1] INFO -- : I, [2021-12-22T12:09:12.910336 #1] INFO -- : > rm /root/upgrade_postgres I, [2021-12-22T12:09:12.911656 #1] INFO -- : I, [2021-12-22T12:09:12.911836 #1] INFO -- : Replacing data_directory = '/var/lib/postgresql/13/main' with data_directory = '/shared/postgres_data' in /etc/postgresql/13/main/postgresql.conf I, [2021-12-22T12:09:12.912203 #1] INFO -- : Replacing (?-mix:#?listen_addresses *=.*) with listen_addresses = '*' in /etc/postgresql/13/main/postgresql.conf I, [2021-12-22T12:09:12.933927 #1] INFO -- : Replacing (?-mix:#?synchronous_commit *=.*) with synchronous_commit = $db_synchronous_commit in /etc/postgresql/13/main/postgresql.conf I, [2021-12-22T12:09:12.934365 #1] INFO -- : Replacing (?-mix:#?shared_buffers *=.*) with shared_buffers = $db_shared_buffers in /etc/postgresql/13/main/postgresql.conf I, [2021-12-22T12:09:12.934753 #1] INFO -- : Replacing (?-mix:#?work_mem *=.*) with work_mem = $db_work_mem in /etc/postgresql/13/main/postgresql.conf I, [2021-12-22T12:09:12.935135 #1] INFO -- : Replacing (?-mix:#?default_text_search_config *=.*) with default_text_search_config = '$db_default_text_search_config' in /etc/postgresql/13/main/postgresql.conf I, [2021-12-22T12:09:12.935459 #1] INFO -- : > install -d -m 0755 -o postgres -g postgres /shared/postgres_backup I, [2021-12-22T12:09:12.938583 #1] INFO -- : I, [2021-12-22T12:09:12.938889 #1] INFO -- : Replacing (?-mix:#?checkpoint_segments *=.*) with checkpoint_segments = $db_checkpoint_segments in /etc/postgresql/13/main/postgresql.conf I, [2021-12-22T12:09:12.939246 #1] INFO -- : Replacing (?-mix:#?logging_collector *=.*) with logging_collector = $db_logging_collector in /etc/postgresql/13/main/postgresql.conf I, [2021-12-22T12:09:12.939664 #1] INFO -- : Replacing (?-mix:#?log_min_duration_statement *=.*) with log_min_duration_statement = $db_log_min_duration_statement in /etc/postgresql/13/main/postgresql.conf I, [2021-12-22T12:09:12.940011 #1] INFO -- : Replacing (?-mix:^#local +replication +postgres +peer$) with local replication postgres peer in /etc/postgresql/13/main/pg_hba.conf I, [2021-12-22T12:09:12.940375 #1] INFO -- : Replacing (?-mix:^host.*all.*all.*127.*$) with host all all 0.0.0.0/0 md5 in /etc/postgresql/13/main/pg_hba.conf I, [2021-12-22T12:09:12.940711 #1] INFO -- : Replacing (?-mix:^host.*all.*all.*::1\/128.*$) with host all all ::/0 md5 in /etc/postgresql/13/main/pg_hba.conf I, [2021-12-22T12:09:12.941023 #1] INFO -- : > HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main I, [2021-12-22T12:09:12.942680 #1] INFO -- : > sleep 5 2021-12-22 12:09:13.078 UTC [50] LOG: starting PostgreSQL 13.5 (Debian 13.5-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit 2021-12-22 12:09:13.078 UTC [50] LOG: listening on IPv4 address "0.0.0.0", port 5432 2021-12-22 12:09:13.078 UTC [50] LOG: listening on IPv6 address "::", port 5432 2021-12-22 12:09:13.130 UTC [50] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" 2021-12-22 12:09:13.198 UTC [53] LOG: database system shutdown was interrupted; last known up at 2021-12-22 12:06:26 UTC 2021-12-22 12:09:13.916 UTC [53] LOG: database system was not properly shut down; automatic recovery in progress 2021-12-22 12:09:13.967 UTC [53] LOG: redo starts at 83/B07EB758 2021-12-22 12:09:13.992 UTC [53] LOG: invalid record length at 83/B12EDED8: wanted 24, got 0 2021-12-22 12:09:13.992 UTC [53] LOG: redo done at 83/B12EDEA0 I, [2021-12-22T12:09:17.952534 #1] INFO -- : I, [2021-12-22T12:09:17.952786 #1] INFO -- : > su postgres -c 'createdb discourse' || true 2021-12-22 12:09:17.984 UTC [57] postgres@postgres FATAL: the database system is starting up 2021-12-22 12:09:17.985 UTC [58] postgres@template1 FATAL: the database system is starting up createdb: error: could not connect to database template1: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL: the database system is starting up I, [2021-12-22T12:09:17.986173 #1] INFO -- : I, [2021-12-22T12:09:17.986413 #1] INFO -- : > su postgres -c 'psql discourse -c "create user discourse;"' || true 2021-12-22 12:09:18.016 UTC [62] postgres@discourse FATAL: the database system is starting up psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL: the database system is starting up I, [2021-12-22T12:09:18.018083 #1] INFO -- : I, [2021-12-22T12:09:18.018325 #1] INFO -- : > su postgres -c 'psql discourse -c "grant all privileges on database discourse to discourse;"' || true 2021-12-22 12:09:18.049 UTC [66] postgres@discourse FATAL: the database system is starting up psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL: the database system is starting up I, [2021-12-22T12:09:18.051088 #1] INFO -- : I, [2021-12-22T12:09:18.051313 #1] INFO -- : > su postgres -c 'psql discourse -c "alter schema public owner to discourse;"' 2021-12-22 12:09:18.081 UTC [70] postgres@discourse FATAL: the database system is starting up psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL: the database system is starting up I, [2021-12-22T12:09:18.082181 #1] INFO -- : I, [2021-12-22T12:09:18.082516 #1] INFO -- : Terminating async processes I, [2021-12-22T12:09:18.082549 #1] INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main pid: 50 2021-12-22 12:09:18.082 UTC [50] LOG: received fast shutdown request I, [2021-12-22T12:09:28.083088 #1] INFO -- : HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main pid:50 did not terminate cleanly, forcing termination! FAILED -------------------- Pups::ExecError: su postgres -c 'psql discourse -c "alter schema public owner to discourse;"' failed with return # Location of failure: /pups/lib/pups/exec_command.rb:112:in `spawn' exec failed with the params "su postgres -c 'psql $db_name -c \"alter schema public owner to $db_user;\"'" 1c37ea92329dcf84d63aebbb7c326ab53b3bdae35f538566effc9009403369fd ** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one. ./discourse-doctor may help diagnose the problem. kinetiksoft:~$ sudo /usr/local/bin/discourse-tool rebuild x86_64 arch detected. Ensuring launcher is up to date Fetching origin Launcher is up-to-date Stopping old container + /usr/bin/docker stop -t 60 app app cd /pups && git pull && git checkout v1.0.3 && /pups/bin/pups --stdin Already up to date. Note: checking out 'v1.0.3'. You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by performing another checkout. If you want to create a new branch to retain commits you create, you may do so (now or later) by using -b with the checkout command again. Example: git checkout -b HEAD is now at d1db030 cut a new version I, [2021-12-22T12:17:14.375608 #1] INFO -- : Loading --stdin I, [2021-12-22T12:17:14.380128 #1] INFO -- : > locale-gen $LANG && update-locale I, [2021-12-22T12:17:14.401507 #1] INFO -- : Generating locales (this might take a while)... Generation complete. I, [2021-12-22T12:17:14.401656 #1] INFO -- : > mkdir -p /shared/postgres_run I, [2021-12-22T12:17:14.403445 #1] INFO -- : I, [2021-12-22T12:17:14.403608 #1] INFO -- : > chown postgres:postgres /shared/postgres_run I, [2021-12-22T12:17:14.405191 #1] INFO -- : I, [2021-12-22T12:17:14.405358 #1] INFO -- : > chmod 775 /shared/postgres_run I, [2021-12-22T12:17:14.406697 #1] INFO -- : I, [2021-12-22T12:17:14.406802 #1] INFO -- : > rm -fr /var/run/postgresql I, [2021-12-22T12:17:14.408253 #1] INFO -- : I, [2021-12-22T12:17:14.408361 #1] INFO -- : > ln -s /shared/postgres_run /var/run/postgresql I, [2021-12-22T12:17:14.412388 #1] INFO -- : I, [2021-12-22T12:17:14.412496 #1] INFO -- : > socat /dev/null UNIX-CONNECT:/shared/postgres_run/.s.PGSQL.5432 || exit 0 && echo postgres already running stop container ; exit 1 2021/12/22 12:17:14 socat[27] E connect(6, AF=1 "/shared/postgres_run/.s.PGSQL.5432", 36): Connection refused I, [2021-12-22T12:17:14.415572 #1] INFO -- : I, [2021-12-22T12:17:14.415677 #1] INFO -- : > rm -fr /shared/postgres_run/.s* I, [2021-12-22T12:17:14.417476 #1] INFO -- : I, [2021-12-22T12:17:14.417589 #1] INFO -- : > rm -fr /shared/postgres_run/*.pid I, [2021-12-22T12:17:14.419317 #1] INFO -- : I, [2021-12-22T12:17:14.419428 #1] INFO -- : > mkdir -p /shared/postgres_run/13-main.pg_stat_tmp I, [2021-12-22T12:17:14.420864 #1] INFO -- : I, [2021-12-22T12:17:14.421005 #1] INFO -- : > chown postgres:postgres /shared/postgres_run/13-main.pg_stat_tmp I, [2021-12-22T12:17:14.422356 #1] INFO -- : I, [2021-12-22T12:17:14.425488 #1] INFO -- : File > /etc/service/postgres/run chmod: +x chown: I, [2021-12-22T12:17:14.428251 #1] INFO -- : File > /etc/service/postgres/log/run chmod: +x chown: I, [2021-12-22T12:17:14.431100 #1] INFO -- : File > /etc/runit/3.d/99-postgres chmod: +x chown: I, [2021-12-22T12:17:14.433890 #1] INFO -- : File > /root/upgrade_postgres chmod: +x chown: I, [2021-12-22T12:17:14.434069 #1] INFO -- : > chown -R root /var/lib/postgresql/13/main I, [2021-12-22T12:17:56.907961 #1] INFO -- : I, [2021-12-22T12:17:56.908121 #1] INFO -- : > [ ! -e /shared/postgres_data ] && install -d -m 0755 -o postgres -g postgres /shared/postgres_data && sudo -E -u postgres /usr/lib/postgresql/13/bin/initdb -D /shared/postgres_data || exit 0 I, [2021-12-22T12:17:56.909780 #1] INFO -- : I, [2021-12-22T12:17:56.909834 #1] INFO -- : > chown -R postgres:postgres /shared/postgres_data I, [2021-12-22T12:17:56.924298 #1] INFO -- : I, [2021-12-22T12:17:56.924455 #1] INFO -- : > chown -R postgres:postgres /var/run/postgresql I, [2021-12-22T12:17:56.926015 #1] INFO -- : I, [2021-12-22T12:17:56.926194 #1] INFO -- : > /root/upgrade_postgres I, [2021-12-22T12:17:56.929052 #1] INFO -- : I, [2021-12-22T12:17:56.929149 #1] INFO -- : > rm /root/upgrade_postgres I, [2021-12-22T12:17:56.930512 #1] INFO -- : I, [2021-12-22T12:17:56.930689 #1] INFO -- : Replacing data_directory = '/var/lib/postgresql/13/main' with data_directory = '/shared/postgres_data' in /etc/postgresql/13/main/postgresql.conf I, [2021-12-22T12:17:56.931044 #1] INFO -- : Replacing (?-mix:#?listen_addresses *=.*) with listen_addresses = '*' in /etc/postgresql/13/main/postgresql.conf I, [2021-12-22T12:17:56.931281 #1] INFO -- : Replacing (?-mix:#?synchronous_commit *=.*) with synchronous_commit = $db_synchronous_commit in /etc/postgresql/13/main/postgresql.conf I, [2021-12-22T12:17:56.931519 #1] INFO -- : Replacing (?-mix:#?shared_buffers *=.*) with shared_buffers = $db_shared_buffers in /etc/postgresql/13/main/postgresql.conf I, [2021-12-22T12:17:56.931779 #1] INFO -- : Replacing (?-mix:#?work_mem *=.*) with work_mem = $db_work_mem in /etc/postgresql/13/main/postgresql.conf I, [2021-12-22T12:17:56.932014 #1] INFO -- : Replacing (?-mix:#?default_text_search_config *=.*) with default_text_search_config = '$db_default_text_search_config' in /etc/postgresql/13/main/postgresql.conf I, [2021-12-22T12:17:56.932277 #1] INFO -- : > install -d -m 0755 -o postgres -g postgres /shared/postgres_backup I, [2021-12-22T12:17:56.934064 #1] INFO -- : I, [2021-12-22T12:17:56.934292 #1] INFO -- : Replacing (?-mix:#?checkpoint_segments *=.*) with checkpoint_segments = $db_checkpoint_segments in /etc/postgresql/13/main/postgresql.conf I, [2021-12-22T12:17:56.934531 #1] INFO -- : Replacing (?-mix:#?logging_collector *=.*) with logging_collector = $db_logging_collector in /etc/postgresql/13/main/postgresql.conf I, [2021-12-22T12:17:56.934769 #1] INFO -- : Replacing (?-mix:#?log_min_duration_statement *=.*) with log_min_duration_statement = $db_log_min_duration_statement in /etc/postgresql/13/main/postgresql.conf I, [2021-12-22T12:17:56.935003 #1] INFO -- : Replacing (?-mix:^#local +replication +postgres +peer$) with local replication postgres peer in /etc/postgresql/13/main/pg_hba.conf I, [2021-12-22T12:17:56.935192 #1] INFO -- : Replacing (?-mix:^host.*all.*all.*127.*$) with host all all 0.0.0.0/0 md5 in /etc/postgresql/13/main/pg_hba.conf I, [2021-12-22T12:17:56.935351 #1] INFO -- : Replacing (?-mix:^host.*all.*all.*::1\/128.*$) with host all all ::/0 md5 in /etc/postgresql/13/main/pg_hba.conf I, [2021-12-22T12:17:56.935501 #1] INFO -- : > HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main I, [2021-12-22T12:17:56.936492 #1] INFO -- : > sleep 5 2021-12-22 12:17:57.065 UTC [50] LOG: starting PostgreSQL 13.5 (Debian 13.5-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit 2021-12-22 12:17:57.065 UTC [50] LOG: listening on IPv4 address "0.0.0.0", port 5432 2021-12-22 12:17:57.065 UTC [50] LOG: listening on IPv6 address "::", port 5432 2021-12-22 12:17:57.133 UTC [50] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" 2021-12-22 12:17:57.201 UTC [53] LOG: database system shutdown was interrupted; last known up at 2021-12-22 12:16:34 UTC 2021-12-22 12:17:57.810 UTC [53] LOG: database system was not properly shut down; automatic recovery in progress 2021-12-22 12:17:57.845 UTC [53] LOG: redo starts at 83/B12EDF50 2021-12-22 12:17:58.554 UTC [53] LOG: invalid record length at 83/BA520118: wanted 24, got 0 2021-12-22 12:17:58.554 UTC [53] LOG: redo done at 83/BA5200F0 I, [2021-12-22T12:18:01.937956 #1] INFO -- : I, [2021-12-22T12:18:01.938274 #1] INFO -- : > su postgres -c 'createdb discourse' || true 2021-12-22 12:18:01.968 UTC [57] postgres@postgres FATAL: the database system is starting up 2021-12-22 12:18:01.969 UTC [58] postgres@template1 FATAL: the database system is starting up createdb: error: could not connect to database template1: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL: the database system is starting up I, [2021-12-22T12:18:01.970057 #1] INFO -- : I, [2021-12-22T12:18:01.970258 #1] INFO -- : > su postgres -c 'psql discourse -c "create user discourse;"' || true 2021-12-22 12:18:02.000 UTC [62] postgres@discourse FATAL: the database system is starting up psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL: the database system is starting up I, [2021-12-22T12:18:02.001459 #1] INFO -- : I, [2021-12-22T12:18:02.001709 #1] INFO -- : > su postgres -c 'psql discourse -c "grant all privileges on database discourse to discourse;"' || true 2021-12-22 12:18:02.033 UTC [66] postgres@discourse FATAL: the database system is starting up psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL: the database system is starting up I, [2021-12-22T12:18:02.034185 #1] INFO -- : I, [2021-12-22T12:18:02.034419 #1] INFO -- : > su postgres -c 'psql discourse -c "alter schema public owner to discourse;"' 2021-12-22 12:18:02.064 UTC [70] postgres@discourse FATAL: the database system is starting up psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL: the database system is starting up I, [2021-12-22T12:18:02.065753 #1] INFO -- : I, [2021-12-22T12:18:02.066059 #1] INFO -- : Terminating async processes I, [2021-12-22T12:18:02.066082 #1] INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main pid: 50 2021-12-22 12:18:02.066 UTC [50] LOG: received fast shutdown request I, [2021-12-22T12:18:12.066665 #1] INFO -- : HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main pid:50 did not terminate cleanly, forcing termination! FAILED -------------------- Pups::ExecError: su postgres -c 'psql discourse -c "alter schema public owner to discourse;"' failed with return # Location of failure: /pups/lib/pups/exec_command.rb:112:in `spawn' exec failed with the params "su postgres -c 'psql $db_name -c \"alter schema public owner to $db_user;\"'" c05642e47b7cc5a7871faa3bf73549342017cbe76da749c7fb23bcb40da0b2dc ** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one. ./discourse-doctor may help diagnose the problem. kinetiksoft:~$ Session was closed

Whenever this happens, we usually (temporarily, as otherwise Git gets confused when updating the next time) replace the sleep 5 with a few more seconds so that Postgres has more time to settle/recover. templates/postgres.template.yml 4cd6a5d96 # give db a few secs to start up - exec: "sleep 5" If you’re including a different postgres.template.yml, of course, edit that one instead.

[image] kinetiksoft: 2021-12-22 12:04:08.808 UTC [53] LOG: database system shutdown was interrupted; last known up at 2021-12-22 12:02:56 UTC 2021-12-22 12:04:09.928 UTC [53] LOG: database system was not properly shut down; automatic recovery in progress 2021-12-22 12:04:09.972 UTC [53] LOG: redo starts at 83/B07EB758 2021-12-22 12:04:10.123 UTC [53] LOG: invalid record length at 83/B12EDED8: wanted 24, got 0 2021-12-22 12:04:10.123 UTC [53] LOG: redo done at 83/B12EDEA0 It’s not shutting down, it’s not starting up properly because the redo log is damaged. If you have a backup you should consider recovering it on a blank instance.

But the instance is working fine overall, it starts behaving like that on rebuild only. What can we do to rebuild the forum just as we did multiple times before?

Discourse update doesn't wait for Postgress DB to shut down

Product Support Self-hosting

kinetiksoft December 29, 2021, 7:20pm 5

That worked flawlessly. Thank you!

Topic		Replies	Views
Postgres Errors on Rebuild Self-hosting	2	1753	September 16, 2015
Bootstrap failed, please help :( Self-hosting	6	1103	January 3, 2023
Upgrade failed due to unclean database shutdown Self-hosting	20	4500	December 30, 2016
Database system was not properly shut down error when rebuilding Self-hosting	3	7097	September 19, 2019
Discourse rebuild fails due to unclean shutdown Product Support	3	369	February 4, 2025

Discourse update doesn't wait for Postgress DB to shut down

Related topics