Continuing the discussion from PostgreSQL 15 update:
Hello, I was looking to rebuild my Discourse installation today following an addition of a plugin to my app.yml file. However, when I ran a rebuild the operation failed stating how the The source cluster was not shut down cleanly. I have followed the instructions to try and remediate the problem by starting and stopping the container, but it yields the same results.
My attempt at the FAQ Instructions:
root@anomicforum:/var/discourse# ./launcher start app
x86_64 arch detected.
starting up existing container
+ /usr/bin/docker start app
app
root@anomicforum:/var/discourse# ./launcher stop app
x86_64 arch detected.
+ /usr/bin/docker stop -t 600 app
app
My rebuild attempt:
x86_64 arch detected.
Ensuring launcher is up to date
Launcher is up-to-date
Stopping old container
+ /usr/bin/docker stop -t 600 app
app
2.0.20250129-0720: Pulling from discourse/base
Digest: sha256:01b8516e5504c0e9bc3707773015ff4407be03a89154194ff3b5b8699291bc26
Status: Image is up to date for discourse/base:2.0.20250129-0720
docker.io/discourse/base:2.0.20250129-0720
/usr/local/lib/ruby/gems/3.3.0/gems/pups-1.2.1/lib/pups.rb
/usr/local/bin/pups --stdin
I, [2025-02-03T21:14:22.601738 #1] INFO -- : Reading from stdin
I, [2025-02-03T21:14:22.615995 #1] INFO -- : File > /etc/service/postgres/run chmod: +x chown:
I, [2025-02-03T21:14:22.620264 #1] INFO -- : File > /etc/service/postgres/log/run chmod: +x chown:
I, [2025-02-03T21:14:22.624344 #1] INFO -- : File > /etc/runit/3.d/99-postgres chmod: +x chown:
I, [2025-02-03T21:14:22.628508 #1] INFO -- : File > /root/install_postgres chmod: +x chown:
I, [2025-02-03T21:14:22.636119 #1] INFO -- : File > /root/upgrade_postgres chmod: +x chown:
I, [2025-02-03T21:14:22.637957 #1] INFO -- : Replacing data_directory = '/var/lib/postgresql/15/main' with data_directory = '/shared/postgres_data' in /etc/postgresql/15/main/postgresql.conf
I, [2025-02-03T21:14:22.638736 #1] INFO -- : Replacing (?-mix:#?listen_addresses *=.*) with listen_addresses = '*' in /etc/postgresql/15/main/postgresql.conf
I, [2025-02-03T21:14:22.639276 #1] INFO -- : Replacing (?-mix:#?synchronous_commit *=.*) with synchronous_commit = $db_synchronous_commit in /etc/postgresql/15/main/postgresql.conf
I, [2025-02-03T21:14:22.640015 #1] INFO -- : Replacing (?-mix:#?shared_buffers *=.*) with shared_buffers = $db_shared_buffers in /etc/postgresql/15/main/postgresql.conf
I, [2025-02-03T21:14:22.641056 #1] INFO -- : Replacing (?-mix:#?work_mem *=.*) with work_mem = $db_work_mem in /etc/postgresql/15/main/postgresql.conf
I, [2025-02-03T21:14:22.641869 #1] INFO -- : Replacing (?-mix:#?default_text_search_config *=.*) with default_text_search_config = '$db_default_text_search_config' in /etc/postgresql/15/main/postgresql.conf
I, [2025-02-03T21:14:22.642730 #1] INFO -- : Replacing (?-mix:#?checkpoint_segments *=.*) with checkpoint_segments = $db_checkpoint_segments in /etc/postgresql/15/main/postgresql.conf
I, [2025-02-03T21:14:22.643579 #1] INFO -- : Replacing (?-mix:#?logging_collector *=.*) with logging_collector = $db_logging_collector in /etc/postgresql/15/main/postgresql.conf
I, [2025-02-03T21:14:22.644530 #1] INFO -- : Replacing (?-mix:#?log_min_duration_statement *=.*) with log_min_duration_statement = $db_log_min_duration_statement in /etc/postgresql/15/main/postgresql.conf
I, [2025-02-03T21:14:22.646187 #1] INFO -- : Replacing (?-mix:^#local +replication +postgres +peer$) with local replication postgres peer in /etc/postgresql/15/main/pg_hba.conf
I, [2025-02-03T21:14:22.646595 #1] INFO -- : Replacing (?-mix:^host.*all.*all.*127.*$) with host all all 0.0.0.0/0 md5 in /etc/postgresql/15/main/pg_hba.conf
I, [2025-02-03T21:14:22.646955 #1] INFO -- : Replacing (?-mix:^host.*all.*all.*::1\/128.*$) with host all all ::/0 md5 in /etc/postgresql/15/main/pg_hba.conf
I, [2025-02-03T21:14:22.647596 #1] INFO -- : > if [ -f /root/install_postgres ]; then
/root/install_postgres && rm -f /root/install_postgres
elif [ -e /shared/postgres_run/.s.PGSQL.5432 ]; then
socat /dev/null UNIX-CONNECT:/shared/postgres_run/.s.PGSQL.5432 || exit 0 && echo postgres already running stop container ; exit 1
fi
2025/02/03 21:14:25 socat[33] E connect(, AF=1 "/shared/postgres_run/.s.PGSQL.5432", 36): Connection refused
initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
W: https://dl.yarnpkg.com/debian/dists/stable/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
debconf: delaying package configuration, since apt-utils is not installed
I, [2025-02-03T21:14:59.545530 #1] INFO -- : Generating locales (this might take a while)...
en_US.UTF-8... done
Generation complete.
Upgrading PostgreSQL from version 13 to 15
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.UTF-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /shared/postgres_data_new ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
Success. You can now start the database server using:
/usr/lib/postgresql/15/bin/pg_ctl -D /shared/postgres_data_new -l logfile start
Get:1 http://deb.debian.org/debian bookworm-backports InRelease [59.0 kB]
Get:2 http://deb.debian.org/debian bookworm InRelease [151 kB]
Get:3 http://deb.debian.org/debian bookworm-updates InRelease [55.4 kB]
Get:4 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]
Get:5 https://deb.nodesource.com/node_22.x nodistro InRelease [12.1 kB]
Get:6 https://dl.yarnpkg.com/debian stable InRelease [17.1 kB]
Get:7 http://deb.debian.org/debian bookworm-backports/main amd64 Packages [280 kB]
Get:8 http://deb.debian.org/debian bookworm/main amd64 Packages [8,792 kB]
Get:9 https://apt.postgresql.org/pub/repos/apt bookworm-pgdg InRelease [129 kB]
Get:10 http://deb.debian.org/debian bookworm-updates/main amd64 Packages [13.5 kB]
Get:11 http://deb.debian.org/debian-security bookworm-security/main amd64 Packages [243 kB]
Get:12 https://deb.nodesource.com/node_22.x nodistro/main amd64 Packages [5,274 B]
Get:13 https://dl.yarnpkg.com/debian stable/main all Packages [10.9 kB]
Get:14 https://dl.yarnpkg.com/debian stable/main amd64 Packages [10.9 kB]
Get:15 https://apt.postgresql.org/pub/repos/apt bookworm-pgdg/main amd64 Packages [360 kB]
Fetched 10.2 MB in 4s (2,452 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
postgresql-client-13
Suggested packages:
postgresql-doc-13
The following NEW packages will be installed:
postgresql-13 postgresql-13-pgvector postgresql-client-13
0 upgraded, 3 newly installed, 0 to remove and 0 not upgraded.
Need to get 17.3 MB of archives.
After this operation, 56.7 MB of additional disk space will be used.
Get:1 https://apt.postgresql.org/pub/repos/apt bookworm-pgdg/main amd64 postgresql-client-13 amd64 13.18-1.pgdg120+1 [1,523 kB]
Get:2 https://apt.postgresql.org/pub/repos/apt bookworm-pgdg/main amd64 postgresql-13 amd64 13.18-1.pgdg120+1 [15.4 MB]
Get:3 https://apt.postgresql.org/pub/repos/apt bookworm-pgdg/main amd64 postgresql-13-pgvector amd64 0.8.0-1.pgdg120+1 [297 kB]
Fetched 17.3 MB in 1s (14.9 MB/s)
Selecting previously unselected package postgresql-client-13.
(Reading database ... 33363 files and directories currently installed.)
Preparing to unpack .../postgresql-client-13_13.18-1.pgdg120+1_amd64.deb ...
Unpacking postgresql-client-13 (13.18-1.pgdg120+1) ...
Selecting previously unselected package postgresql-13.
Preparing to unpack .../postgresql-13_13.18-1.pgdg120+1_amd64.deb ...
Unpacking postgresql-13 (13.18-1.pgdg120+1) ...
Selecting previously unselected package postgresql-13-pgvector.
Preparing to unpack .../postgresql-13-pgvector_0.8.0-1.pgdg120+1_amd64.deb ...
Unpacking postgresql-13-pgvector (0.8.0-1.pgdg120+1) ...
Setting up postgresql-client-13 (13.18-1.pgdg120+1) ...
Setting up postgresql-13 (13.18-1.pgdg120+1) ...
Creating new PostgreSQL cluster 13/main ...
/usr/lib/postgresql/13/bin/initdb -D /var/lib/postgresql/13/main --auth-local peer --auth-host md5
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "C.UTF-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /var/lib/postgresql/13/main ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
Success. You can now start the database server using:
pg_ctlcluster 13 main start
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Setting up postgresql-13-pgvector (0.8.0-1.pgdg120+1) ...
Processing triggers for postgresql-common (267.pgdg120+1) ...
Building PostgreSQL dictionaries from installed myspell/hunspell packages...
Removing obsolete dictionary files:
Stopping PostgreSQL 13 database server: main.
Stopping PostgreSQL 15 database server: main.
Performing Consistency Checks
-----------------------------
Checking cluster versions ok
The source cluster was not shut down cleanly.
Failure, exiting
-------------------------------------------------------------------------------------
UPGRADE OF POSTGRES FAILED
Please visit https://meta.discourse.org/t/postgresql-15-update/349515 for support.
You can run ./launcher start app to restart your app in the meanwhile
-------------------------------------------------------------------------------------
FAILED
--------------------
Pups::ExecError: if [ -f /root/install_postgres ]; then
/root/install_postgres && rm -f /root/install_postgres
elif [ -e /shared/postgres_run/.s.PGSQL.5432 ]; then
socat /dev/null UNIX-CONNECT:/shared/postgres_run/.s.PGSQL.5432 || exit 0 && echo postgres already running stop container ; exit 1
fi
failed with return #<Process::Status: pid 18 exit 1>
Location of failure: /usr/local/lib/ruby/gems/3.3.0/gems/pups-1.2.1/lib/pups/exec_command.rb:132:in `spawn'
exec failed with the params {"tag"=>"db", "cmd"=>"if [ -f /root/install_postgres ]; then\n /root/install_postgres && rm -f /root/install_postgres\nelif [ -e /shared/postgres_run/.s.PGSQL.5432 ]; then\n socat /dev/null UNIX-CONNECT:/shared/postgres_run/.s.PGSQL.5432 || exit 0 && echo postgres already running stop container ; exit 1\nfi\n"}
bootstrap failed with exit code 1
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.
9a16613f6ad358fc52b6d9498c29d61d30857e30528fab9a5e7dc72b1d0d6bf8
Anybody know a remedy to solve this? If I can’t find a solution, I’ll probably re-image the entire server this weekend and start fresh with a new Discourse installation and restore from a backup. Thanks in advance.
P.S. I have postponed the update by modifying the yaml file to use the postgres.13 template file and the forum is running normally on 13.
