PostgreSQL 15 update

Success. You can now start the database server using:

    pg_ctlcluster 13 main start

invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Setting up postgresql-13-pgvector (0.8.0-1.pgdg120+1) ...
Processing triggers for libc-bin (2.36-9+deb12u9) ...
Processing triggers for postgresql-common (267.pgdg120+1) ...
Building PostgreSQL dictionaries from installed myspell/hunspell packages...
Removing obsolete dictionary files:
Stopping PostgreSQL 13 database server: main.
Stopping PostgreSQL 15 database server: main.
Performing Consistency Checks
-----------------------------
Checking cluster versions                                   ok

The source cluster was not shut down cleanly.
Failure, exiting
-------------------------------------------------------------------------------------
UPGRADE OF POSTGRES FAILED

Please visit https://meta.discourse.org/t/postgresql-15-update/349515 for support.

You can run ./launcher start app to restart your app in the meanwhile
-------------------------------------------------------------------------------------



FAILED
--------------------
Pups::ExecError: if [ -f /root/install_postgres ]; then
  /root/install_postgres && rm -f /root/install_postgres
elif [ -e /shared/postgres_run/.s.PGSQL.5432 ]; then
  socat /dev/null UNIX-CONNECT:/shared/postgres_run/.s.PGSQL.5432 || exit 0 && echo postgres already running stop container ; exit 1
fi
 failed with return #<Process::Status: pid 17 exit 1>
Location of failure: /usr/local/lib/ruby/gems/3.3.0/gems/pups-1.2.1/lib/pups/exec_command.rb:132:in `spawn'
exec failed with the params {"tag"=>"db", "cmd"=>"if [ -f /root/install_postgres ]; then\n  /root/install_postgres && rm -f /root/install_postgres\nelif [ -e /shared/postgres_run/.s.PGSQL.5432 ]; then\n  socat /dev/null UNIX-CONNECT:/shared/postgres_run/.s.PGSQL.5432 || exit 0 && echo postgres already running stop container ; exit 1\nfi\n"}
bootstrap failed with exit code 1
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.
cbebd339ee3bd1e246300784f7825042702c1b5032877bfe580cb7f635fb852b

disk info:

========================================
Discourse 3.4.0.beta3
Discourse version at mydomian: Discourse 3.4.0.beta3
Discourse version at localhost: Discourse 3.4.0.beta3


==================== MEMORY INFORMATION ====================
RAM (MB): 2055

              total        used        free      shared  buff/cache   available
Mem:           1960        1463         100          43         396         255
Swap:          2047          66        1981

==================== DISK SPACE CHECK ====================
---------- OS Disk Space ----------
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        59G   45G   14G  77% /

---------- Container Disk Space ----------
Filesystem      Size  Used Avail Use% Mounted on
overlay          59G   45G   14G  77% /
/dev/root        59G   45G   14G  77% /shared
/dev/root        59G   45G   14G  77% /var/log

==================== DISK INFORMATION ====================
Disk /dev/loop0: 110.24 MiB, 115593216 bytes, 225768 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop2: 73.91 MiB, 77479936 bytes, 151328 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop3: 103.102 MiB, 109043712 bytes, 212976 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop4: 73.89 MiB, 77459456 bytes, 151288 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop5: 55.68 MiB, 58363904 bytes, 113992 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop6: 63.71 MiB, 66789376 bytes, 130448 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop7: 63.10 MiB, 67080192 bytes, 131016 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/xvda: 60 GiB, 64424509440 bytes, 125829120 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x15e6b846

Device     Boot Start       End   Sectors Size Id Type
/dev/xvda1 *     2048 125829086 125827039  60G 83 Linux


Disk /dev/loop8: 26.33 MiB, 27602944 bytes, 53912 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop9: 110.4 MiB, 115384320 bytes, 225360 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop10: 55.37 MiB, 58052608 bytes, 113384 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop11: 104.2 MiB, 109252608 bytes, 213384 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop13: 25.76 MiB, 26992640 bytes, 52720 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

==================== END DISK INFORMATION ====================

Does this mean I should upgrade Docker first if I encounter this issue while trying to rebuild?

I’m on 3.4.0.beta3-dev(93983286b5) now

Update: I rebuilt again, and this time it went through successfully.

2 Likes

There are specific instructions for that in the OP.

You should probably upgrade docker regardless of whether you have any issues.

2 Likes

For me, what finally turned out to help was to update docker.
I followed upgrade - Command for upgrading docker - Stack Overflow, and then ran:
sudo service docker start

Only then:

./launcher start app
./launcher stop app
./launcher rebuild app
3 Likes

I’m having significant problems doing this upgrade too. Similar @Vaping_Community, I have postgres on a different mounted volume (/mnt/volume_tor1_01/shared/standalone).

I am getting this error

mv: cannot move '/shared/postgres_data' to '/shared/postgres_data_old': Device or resource busy
mv: inter-device move failed: '/shared/postgres_data_new' to '/shared/postgres_data/postgres_data_new'; unable to remove target: Directory not empty

 [...]

Upgrade Complete
----------------
Optimizer statistics are not transferred by pg_upgrade.
Once you start the new server, consider running:
    /usr/lib/postgresql/15/bin/vacuumdb --all --analyze-in-stages

Running this script will delete the old cluster's data files:
    ./delete_old_cluster.sh
-------------------------------------------------------------------------------------
UPGRADE OF POSTGRES COMPLETE

Old 13 database is stored at /shared/postgres_data_old

To complete the upgrade, rebuild again using:

./launcher rebuild app
-------------------------------------------------------------------------------------

My understanding is that the solution is to basically manually rename
postgres_datapostgres_data_old
postgres_newpostgres_data

then run ./launcher rebuild app. Seems simple enough, but when I rebuild, I still am stuck in the loop and end up with the following directories:

  • postgres_data_older
  • postgres_data_old (I think it becomes postgres_data_older)
  • postgres_data
  • postgres_data_new

It seems like for some reason it’s not recognizing that postgres_data has updated, tries to do it again, fails when trying to move the directory.

$ cat postgres_data/PG_VERSION 
15

I tried the manual approach laid out in the OP but no luck either. Please help!

full log
x86_64 arch detected.
Ensuring launcher is up to date
Fetching origin
Launcher is up-to-date
Stopping old container
+ /usr/bin/docker stop -t 600 app
app
2.0.20250129-0720: Pulling from discourse/base
Digest: sha256:01b8516e5504c0e9bc3707773015ff4407be03a89154194ff3b5b8699291bc26
Status: Image is up to date for discourse/base:2.0.20250129-0720
docker.io/discourse/base:2.0.20250129-0720
/usr/local/lib/ruby/gems/3.3.0/gems/pups-1.2.1/lib/pups.rb
/usr/local/bin/pups --stdin
I, [2025-02-23T11:41:32.477354 #1]  INFO -- : Reading from stdin
I, [2025-02-23T11:41:32.504068 #1]  INFO -- : File > /etc/service/postgres/run  chmod: +x  chown: 
I, [2025-02-23T11:41:32.508552 #1]  INFO -- : File > /etc/service/postgres/log/run  chmod: +x  chown: 
I, [2025-02-23T11:41:32.513290 #1]  INFO -- : File > /etc/runit/3.d/99-postgres  chmod: +x  chown: 
I, [2025-02-23T11:41:32.517671 #1]  INFO -- : File > /root/install_postgres  chmod: +x  chown: 
I, [2025-02-23T11:41:32.522244 #1]  INFO -- : File > /root/upgrade_postgres  chmod: +x  chown: 
I, [2025-02-23T11:41:32.523354 #1]  INFO -- : Replacing data_directory = '/var/lib/postgresql/15/main' with data_directory = '/shared/postgres_data' in /etc/postgresql/15/main/postgresql.conf
I, [2025-02-23T11:41:32.523912 #1]  INFO -- : Replacing (?-mix:#?listen_addresses *=.*) with listen_addresses = '*' in /etc/postgresql/15/main/postgresql.conf
I, [2025-02-23T11:41:32.524756 #1]  INFO -- : Replacing (?-mix:#?synchronous_commit *=.*) with synchronous_commit = $db_synchronous_commit in /etc/postgresql/15/main/postgresql.conf
I, [2025-02-23T11:41:32.525365 #1]  INFO -- : Replacing (?-mix:#?shared_buffers *=.*) with shared_buffers = $db_shared_buffers in /etc/postgresql/15/main/postgresql.conf
I, [2025-02-23T11:41:32.526239 #1]  INFO -- : Replacing (?-mix:#?work_mem *=.*) with work_mem = $db_work_mem in /etc/postgresql/15/main/postgresql.conf
I, [2025-02-23T11:41:32.526873 #1]  INFO -- : Replacing (?-mix:#?default_text_search_config *=.*) with default_text_search_config = '$db_default_text_search_config' in /etc/postgresql/15/main/postgresql.conf
I, [2025-02-23T11:41:32.527613 #1]  INFO -- : Replacing (?-mix:#?checkpoint_segments *=.*) with checkpoint_segments = $db_checkpoint_segments in /etc/postgresql/15/main/postgresql.conf
I, [2025-02-23T11:41:32.531734 #1]  INFO -- : Replacing (?-mix:#?logging_collector *=.*) with logging_collector = $db_logging_collector in /etc/postgresql/15/main/postgresql.conf
I, [2025-02-23T11:41:32.532882 #1]  INFO -- : Replacing (?-mix:#?log_min_duration_statement *=.*) with log_min_duration_statement = $db_log_min_duration_statement in /etc/postgresql/15/main/postgresql.conf
I, [2025-02-23T11:41:32.534021 #1]  INFO -- : Replacing (?-mix:^#local +replication +postgres +peer$) with local replication postgres  peer in /etc/postgresql/15/main/pg_hba.conf
I, [2025-02-23T11:41:32.534342 #1]  INFO -- : Replacing (?-mix:^host.*all.*all.*127.*$) with host all all 0.0.0.0/0 md5 in /etc/postgresql/15/main/pg_hba.conf
I, [2025-02-23T11:41:32.534935 #1]  INFO -- : Replacing (?-mix:^host.*all.*all.*::1\/128.*$) with host all all ::/0 md5 in /etc/postgresql/15/main/pg_hba.conf
I, [2025-02-23T11:41:32.535476 #1]  INFO -- : > if [ -f /root/install_postgres ]; then
  /root/install_postgres && rm -f /root/install_postgres
elif [ -e /shared/postgres_run/.s.PGSQL.5432 ]; then
  socat /dev/null UNIX-CONNECT:/shared/postgres_run/.s.PGSQL.5432 || exit 0 && echo postgres already running stop container ; exit 1
fi

initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
W: https://dl.yarnpkg.com/debian/dists/stable/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
debconf: delaying package configuration, since apt-utils is not installed
mv: cannot move '/shared/postgres_data' to '/shared/postgres_data_old': Device or resource busy
mv: inter-device move failed: '/shared/postgres_data_new' to '/shared/postgres_data/postgres_data_new'; unable to remove target: Directory not empty
I, [2025-02-23T11:42:52.303584 #1]  INFO -- : Generating locales (this might take a while)...
  en_US.UTF-8... done
Generation complete.
Upgrading PostgreSQL from version 13 to 15
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /shared/postgres_data_new ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok


Success. You can now start the database server using:

    /usr/lib/postgresql/15/bin/pg_ctl -D /shared/postgres_data_new -l logfile start

Get:1 http://deb.debian.org/debian bookworm-backports InRelease [59.0 kB]
Get:2 http://deb.debian.org/debian bookworm InRelease [151 kB]
Get:3 http://deb.debian.org/debian bookworm-updates InRelease [55.4 kB]
Get:4 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]
Get:5 https://dl.yarnpkg.com/debian stable InRelease [17.1 kB]
Get:6 https://deb.nodesource.com/node_22.x nodistro InRelease [12.1 kB]
Get:7 http://deb.debian.org/debian bookworm-backports/main amd64 Packages [283 kB]
Get:8 http://deb.debian.org/debian bookworm/main amd64 Packages [8,792 kB]
Get:9 http://deb.debian.org/debian bookworm-updates/main amd64 Packages [13.5 kB]
Get:10 http://deb.debian.org/debian-security bookworm-security/main amd64 Packages [246 kB]
Get:11 https://dl.yarnpkg.com/debian stable/main amd64 Packages [10.9 kB]
Get:12 https://dl.yarnpkg.com/debian stable/main all Packages [10.9 kB]
Get:13 https://apt.postgresql.org/pub/repos/apt bookworm-pgdg InRelease [129 kB]
Get:14 https://deb.nodesource.com/node_22.x nodistro/main amd64 Packages [5,636 B]
Get:15 https://apt.postgresql.org/pub/repos/apt bookworm-pgdg/main amd64 Packages [360 kB]
Fetched 10.2 MB in 2s (6,490 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  libllvm19 postgresql-client-13
Suggested packages:
  postgresql-doc-13
The following NEW packages will be installed:
  libllvm19 postgresql-13 postgresql-13-pgvector postgresql-client-13
0 upgraded, 4 newly installed, 0 to remove and 12 not upgraded.
Need to get 41.6 MB of archives.
After this operation, 184 MB of additional disk space will be used.
Get:1 http://deb.debian.org/debian bookworm/main amd64 libllvm19 amd64 1:19.1.4-1~deb12u1 [25.9 MB]
Get:2 https://apt.postgresql.org/pub/repos/apt bookworm-pgdg/main amd64 postgresql-client-13 amd64 13.20-1.pgdg120+1 [1,538 kB]
Get:3 https://apt.postgresql.org/pub/repos/apt bookworm-pgdg/main amd64 postgresql-13 amd64 13.20-1.pgdg120+1 [13.9 MB]
Get:4 https://apt.postgresql.org/pub/repos/apt bookworm-pgdg/main amd64 postgresql-13-pgvector amd64 0.8.0-1.pgdg120+1 [297 kB]
Fetched 41.6 MB in 2s (18.8 MB/s)
Selecting previously unselected package libllvm19:amd64.
(Reading database ... 33363 files and directories currently installed.)
Preparing to unpack .../libllvm19_1%3a19.1.4-1~deb12u1_amd64.deb ...
Unpacking libllvm19:amd64 (1:19.1.4-1~deb12u1) ...
Selecting previously unselected package postgresql-client-13.
Preparing to unpack .../postgresql-client-13_13.20-1.pgdg120+1_amd64.deb ...
Unpacking postgresql-client-13 (13.20-1.pgdg120+1) ...
Selecting previously unselected package postgresql-13.
Preparing to unpack .../postgresql-13_13.20-1.pgdg120+1_amd64.deb ...
Unpacking postgresql-13 (13.20-1.pgdg120+1) ...
Selecting previously unselected package postgresql-13-pgvector.
Preparing to unpack .../postgresql-13-pgvector_0.8.0-1.pgdg120+1_amd64.deb ...
Unpacking postgresql-13-pgvector (0.8.0-1.pgdg120+1) ...
Setting up libllvm19:amd64 (1:19.1.4-1~deb12u1) ...
Setting up postgresql-client-13 (13.20-1.pgdg120+1) ...
Setting up postgresql-13 (13.20-1.pgdg120+1) ...
Creating new PostgreSQL cluster 13/main ...
/usr/lib/postgresql/13/bin/initdb -D /var/lib/postgresql/13/main --auth-local peer --auth-host md5
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "C.UTF-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/13/main ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

Success. You can now start the database server using:

    pg_ctlcluster 13 main start

invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Setting up postgresql-13-pgvector (0.8.0-1.pgdg120+1) ...
Processing triggers for libc-bin (2.36-9+deb12u9) ...
Processing triggers for postgresql-common (267.pgdg120+1) ...
Building PostgreSQL dictionaries from installed myspell/hunspell packages...
Removing obsolete dictionary files:
Stopping PostgreSQL 13 database server: main.
Stopping PostgreSQL 15 database server: main.
Performing Consistency Checks
-----------------------------
Checking cluster versions                                   ok
Checking database user is the install user                  ok
Checking database connection settings                       ok
Checking for prepared transactions                          ok
Checking for system-defined composite types in user tables  ok
Checking for reg* data types in user tables                 ok
Checking for contrib/isn with bigint-passing mismatch       ok
Checking for user-defined encoding conversions              ok
Checking for user-defined postfix operators                 ok
Checking for incompatible polymorphic functions             ok
Creating dump of global objects                             ok
Creating dump of database schemas                           ok
Checking for presence of required libraries                 ok
Checking database user is the install user                  ok
Checking for prepared transactions                          ok
Checking for new cluster tablespace directories             ok

If pg_upgrade fails after this point, you must re-initdb the
new cluster before continuing.

Performing Upgrade
------------------
Analyzing all rows in the new cluster                       ok
Freezing all rows in the new cluster                        ok
Deleting files from new pg_xact                             ok
Copying old pg_xact to new server                           ok
Setting oldest XID for new cluster                          ok
Setting next transaction ID and epoch for new cluster       ok
Deleting files from new pg_multixact/offsets                ok
Copying old pg_multixact/offsets to new server              ok
Deleting files from new pg_multixact/members                ok
Copying old pg_multixact/members to new server              ok
Setting next multixact ID and offset for new cluster        ok
Resetting WAL archives                                      ok
Setting frozenxid and minmxid counters in new cluster       ok
Restoring global objects in the new cluster                 ok
Restoring database schemas in the new cluster               ok
Copying user relation files                                 ok
Setting next OID for new cluster                            ok
Sync data directory to disk                                 ok
Creating script to delete old cluster                       ok
Checking for extension updates                              notice

Your installation contains extensions that should be updated
with the ALTER EXTENSION command.  The file
    update_extensions.sql
when executed by psql by the database superuser will update
these extensions.


Upgrade Complete
----------------
Optimizer statistics are not transferred by pg_upgrade.
Once you start the new server, consider running:
    /usr/lib/postgresql/15/bin/vacuumdb --all --analyze-in-stages

Running this script will delete the old cluster's data files:
    ./delete_old_cluster.sh
-------------------------------------------------------------------------------------
UPGRADE OF POSTGRES COMPLETE

Old 13 database is stored at /shared/postgres_data_old

To complete the upgrade, rebuild again using:

./launcher rebuild app
-------------------------------------------------------------------------------------
1 Like

Upgrade on ARM (aarch64) with non default locale

I just spent some time fixing a faulty upgrade on a single server with two container setup.

  1. Its CPU is ARM
  2. Discourse was configured with en_UK.UTF-8 locale

Step by step, I figured that:

  1. tianon/postgres-upgrade:13-to-15 does not work on ARM
  2. locale configuration is stored in the share/${CONTAINER}/postgres_data/postgresql.conf

I started with commenting out the env: part of the data container, to ensure it uses default locales.

So, to get out of the upgrade mess, I took a dangerous step, assuming collation differences between en_GB and en_US were negligible, and I ruthlessly updated the whole cluster to en_US:

UPDATE pg_database SET datctype='en_US.UTF-8', datcollate='en_US.UTF-8';

I’m not sure this step is required, YMMV. I leave it here for others more knowledgeable to comment, and I do not recommend it at all for different languages!

Then I replaced the locale in postgresql.conf:

sed -i -e 's/en_GB/en_US/g' shared/data/postgres_data/postgresql.conf
sudo -H -u discourse ./launcher rebuild data
...
UPGRADE OF POSTGRES COMPLETE

:tada:

Do not think it was easy: I’m not mentioning the long path of exploring options, especially as when I intervened, the admin interface was unavailable, and the last backup was made 6 days ago. I had to fiddle around before I could get to a proper – or at least, working – resolution.

I hope this will save some people some time.

5 Likes

Are you still seeing this in your logs for every rebuild? If so, I suspect you were somehow looking at (thus manually renamed) the wrong directories here:

If the container’s /shared volume is mapped to /mnt/volume_tor1_01/shared/standalone on the host, then cat /mnt/volume_tor1_01/shared/standalone/postgres_data/PG_VERSION should return 13 instead of 15.

Can you share your sanitized app.yml file?

4 Likes

I think you correctly identified the issue. Here’s what I have in my app.yml:

## The Docker container is stateless; all data is stored in /shared
volumes:
  - volume:
      host: /mnt/volume_tor1_01/shared/standalone
      guest: /shared
  - volume:
      host: /mnt/volume_tor1_01/shared/standalone/log/var-log
      guest: /var/log
  - volume:
      host: /var/postgres_data_discourse
      guest: /shared/postgres_data

(based on this guide Move Uploads and Backups to DigitalOcean Block Storage)

$ cat postgres_data/PG_VERSION 
15
$ cat /var/postgres_data_discourse/PG_VERSION 
13

So I think the problem is indeed solved by this approach PostgreSQL 15 update - #125 by Vaping_Community

I think what stumped me was that I had multiple postgres_data folders on the mnt volume (because I reran the rebuild) and didn’t realize the source folder was actually in /var/

I’ll try it out later tonight

2 Likes

It’d be much easier to spin up a new VM and restore a backup there. That way you completely skip the PG upgrade step and can’t break your old site before the new one works.

8 Likes

It worked :slight_smile:

2 Likes

Note that the UIDs change!

/launcher enter app
x86_64 arch detected.
grep postgres /etc/passwd
postgres:x:101:104:PostgreSQL administrator,,,:/var/lib/postgresql:/bin/bash

This just caught me out…

Yeah, it worked for me but better to use updated instruction from OP-post,

I can’t edit my post to add a note.

2 Likes

Failing for me too.

more /var/discourse/shared/standalone/postgres_data_new/pg_upgrade_output.d/20250416T103311.340/log/pg_upgrade_server.log

-----------------------------------------------------------------
  pg_upgrade run on Wed Apr 16 10:33:11 2025
-----------------------------------------------------------------

command: "/usr/lib/postgresql/13/bin/pg_ctl" -w -l "/shared/postgres_data_new/pg_upgrade_output.d/20250416T103311.340/log/pg_upgrade_server.log" -D "/shared/postgres_data" -o 
"-p 50432 -b  -c listen_addresses='' -c unix_socket_permissions=0700 -c unix_socket_directories='/var/lib/postgresql'" start >> "/shared/postgres_data_new/pg_upgrade_output.d/
20250416T103311.340/log/pg_upgrade_server.log" 2>&1
waiting for server to start....2025-04-16 10:33:11.515 UTC [1887] LOG:  starting PostgreSQL 13.20 (Debian 13.20-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12
.2.0-14) 12.2.0, 64-bit
2025-04-16 10:33:11.517 UTC [1887] LOG:  listening on Unix socket "/var/lib/postgresql/.s.PGSQL.50432"
2025-04-16 10:33:11.525 UTC [1887] LOG:  could not open configuration file "/etc/postgresql/13/main/pg_hba.conf": No such file or directory
2025-04-16 10:33:11.525 UTC [1887] FATAL:  could not load pg_hba.conf
2025-04-16 10:33:11.526 UTC [1887] LOG:  database system is shut down

It looks like it doesn’t find /etc/postgresql/13/main/pg_hba.conf, and it’s right because it’s not there. It is is multiple locations though:

/var/discourse/shared/standalone/postgres_data_older/pg_hba.conf
/var/discourse/shared/standalone/postgres_data_new/pg_hba.conf
/var/discourse/shared/standalone/postgres_data/pg_hba.conf
/var/lib/docker/overlay2/6514156241efd883dd59b27a0275ba1cc73a7365fb78c5782536a39ee2d6d641/diff/etc/postgresql/13/main/pg_hba.conf
/var/lib/docker/overlay2/8afed861b039e5a366dc9a5513cd7df5a958d86a41f05063ce5256fb9f8fdfec/diff/etc/postgresql/13/main/pg_hba.conf
/var/lib/docker/overlay2/fa8893e9f714439469d391635d8882b345e90705a886d74cf6fea8d5460df3f0/diff/etc/postgresql/15/main/pg_hba.conf
/var/lib/docker/overlay2/e4eec92eb7b181648ca2bc6744ac3af0935745b75b1abe89170fc4b82a8eca42/diff/etc/postgresql/13/main/pg_hba.conf
/var/lib/docker/overlay2/4cd3189b9d554b1c60891dc5d48cc670e5d7d847ec5a0f2dfa91ad7f66b3a211/diff/etc/postgresql/13/main/pg_hba.conf
/var/lib/docker/overlay2/0db2210fc76fce6f41912183f744517154f46af15a0b1a1e976999eabbd801d9/diff/etc/postgresql/15/main/pg_hba.conf
/var/postgres_data/postgres_data_new/pg_hba.conf
/var/postgres_data/pg_hba.conf

Any hints?

I tried your docker run… command and I get:

Checking cluster versions
Old cluster data and binary directories are from different major versions.

What’s your docker version? What’s your os version? I’ve found that moving to a new vm is often easier.

Docker Version: 28.0.4
Linux Version: 5.15.0-131-generic

You can try changing to the postgres 13 template in your app.yml.

That’s inside the container. You’re looking outside the container.

I did it just to upgrade Discourse, and it went through, now I have:

templates:
  - "templates/postgres.13.template.yml"

Doesn’t this mean I can’t upgrade Postgres though?

I think you can now change it back to the regular template and try again. What I would do, is move to a new VM, but you can try it again if you want.

I upgraded Postgres manually as described in the first post, anyway I can’t set the template back to postgres.template.yml because it won’t rebuild for the same problem:

2025-04-16 13:40:07.815 UTC [1888] LOG: could not open configuration file “/etc/postgresql/13/main/pg_hba.conf”: No such file or directory
2025-04-16 13:40:07.815 UTC [1888] FATAL: could not load pg_hba.conf

I’ll stick to the version 13 hoping it won’t break in the future.
Thank you @pfaffman.