Postgres upgrade success loop due to prior postgres 8 to 10 migration

In a loop of:

To complete the upgrade, rebuild again using: ./launcher rebuild app

I must be missing something. Complete log below. Appreciate any help.

root@discuss:/var/discourse# ./launcher rebuild app
Ensuring launcher is up to date
Fetching origin
Launcher is up-to-date
Stopping old container
+ /usr/bin/docker stop -t 60 app
app
cd /pups && git pull && /pups/bin/pups --stdin
Already up to date.
I, [2020-05-26T20:43:00.459915 #1]  INFO -- : Loading --stdin
I, [2020-05-26T20:43:00.466076 #1]  INFO -- : > locale-gen $LANG && update-locale
I, [2020-05-26T20:43:00.497010 #1]  INFO -- : Generating locales (this might take a while)...
Generation complete.

I, [2020-05-26T20:43:00.497661 #1]  INFO -- : > mkdir -p /shared/postgres_run
I, [2020-05-26T20:43:00.500629 #1]  INFO -- :
I, [2020-05-26T20:43:00.501124 #1]  INFO -- : > chown postgres:postgres /shared/postgres_run
I, [2020-05-26T20:43:00.503417 #1]  INFO -- :
I, [2020-05-26T20:43:00.503867 #1]  INFO -- : > chmod 775 /shared/postgres_run
I, [2020-05-26T20:43:00.505796 #1]  INFO -- :
I, [2020-05-26T20:43:00.506199 #1]  INFO -- : > rm -fr /var/run/postgresql
I, [2020-05-26T20:43:00.508341 #1]  INFO -- :
I, [2020-05-26T20:43:00.508752 #1]  INFO -- : > ln -s /shared/postgres_run /var/run/postgresql
I, [2020-05-26T20:43:00.510785 #1]  INFO -- :
I, [2020-05-26T20:43:00.511166 #1]  INFO -- : > socat /dev/null UNIX-CONNECT:/shared/postgres_run/.s.PGSQL.5432 || exit 0 && echo postgres already running stop container ; exit 1
2020/05/26 20:43:00 socat[26] E connect(6, AF=1 "/shared/postgres_run/.s.PGSQL.5432", 36): No such file or directory
I, [2020-05-26T20:43:00.515245 #1]  INFO -- :
I, [2020-05-26T20:43:00.515542 #1]  INFO -- : > rm -fr /shared/postgres_run/.s*
I, [2020-05-26T20:43:00.518166 #1]  INFO -- :
I, [2020-05-26T20:43:00.518509 #1]  INFO -- : > rm -fr /shared/postgres_run/*.pid
I, [2020-05-26T20:43:00.520876 #1]  INFO -- :
I, [2020-05-26T20:43:00.521215 #1]  INFO -- : > mkdir -p /shared/postgres_run/12-main.pg_stat_tmp
I, [2020-05-26T20:43:00.523438 #1]  INFO -- :
I, [2020-05-26T20:43:00.523931 #1]  INFO -- : > chown postgres:postgres /shared/postgres_run/12-main.pg_stat_tmp
I, [2020-05-26T20:43:00.526034 #1]  INFO -- :
I, [2020-05-26T20:43:00.530519 #1]  INFO -- : File > /etc/service/postgres/run  chmod: +x  chown:
I, [2020-05-26T20:43:00.534602 #1]  INFO -- : File > /etc/service/postgres/log/run  chmod: +x  chown:
I, [2020-05-26T20:43:00.538528 #1]  INFO -- : File > /etc/runit/3.d/99-postgres  chmod: +x  chown:
I, [2020-05-26T20:43:00.542481 #1]  INFO -- : File > /root/upgrade_postgres  chmod: +x  chown:
I, [2020-05-26T20:43:00.542896 #1]  INFO -- : > chown -R root /var/lib/postgresql/12/main
I, [2020-05-26T20:43:00.638174 #1]  INFO -- :
I, [2020-05-26T20:43:00.638520 #1]  INFO -- : > [ ! -e /shared/postgres_data ] && install -d -m 0755 -o postgres -g postgres /shared/postgres_data && sudo -E -u postgres /usr/lib/postgresql/12/bin/initdb -D /shared/postgres_data || exit 0
I, [2020-05-26T20:43:00.640938 #1]  INFO -- :
I, [2020-05-26T20:43:00.641292 #1]  INFO -- : > chown -R postgres:postgres /shared/postgres_data
I, [2020-05-26T20:43:00.672801 #1]  INFO -- :
I, [2020-05-26T20:43:00.673456 #1]  INFO -- : > chown -R postgres:postgres /var/run/postgresql
I, [2020-05-26T20:43:00.676098 #1]  INFO -- :
I, [2020-05-26T20:43:00.676587 #1]  INFO -- : > /root/upgrade_postgres
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
debconf: delaying package configuration, since apt-utils is not installed
mv: cannot move '/shared/postgres_data' to '/shared/postgres_data_old/postgres_data': Directory not empty
mv: cannot move '/shared/postgres_data_new' to '/shared/postgres_data/postgres_data_new': Directory not empty
I, [2020-05-26T20:43:15.530374 #1]  INFO -- : Upgrading PostgreSQL from version 10 to 12
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /shared/postgres_data_new ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok


Success. You can now start the database server using:

/usr/lib/postgresql/12/bin/pg_ctl -D /shared/postgres_data_new -l logfile start

Hit:1 http://deb.debian.org/debian buster InRelease
Get:2 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
Get:3 http://deb.debian.org/debian buster-updates InRelease [49.3 kB]
Get:4 http://apt.postgresql.org/pub/repos/apt buster-pgdg InRelease [84.6 kB]
Get:5 http://security.debian.org/debian-security buster/updates/main amd64 Packages [201 kB]
Hit:6 https://deb.nodesource.com/node_10.x buster InRelease
Get:7 http://apt.postgresql.org/pub/repos/apt buster-pgdg/main amd64 Packages [171 kB]
Fetched 571 kB in 1s (810 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  postgresql-client-10
Suggested packages:
  postgresql-doc-10
The following NEW packages will be installed:
  postgresql-10 postgresql-client-10
0 upgraded, 2 newly installed, 0 to remove and 9 not upgraded.
Need to get 6,390 kB of archives.
After this operation, 30.6 MB of additional disk space will be used.
Get:1 http://apt.postgresql.org/pub/repos/apt buster-pgdg/main amd64 postgresql-client-10 amd64 10.13-1.pgdg100+1 [1,428 kB]
Get:2 http://apt.postgresql.org/pub/repos/apt buster-pgdg/main amd64 postgresql-10 amd64 10.13-1.pgdg100+1 [4,961 kB]
Fetched 6,390 kB in 2s (4,017 kB/s)
Selecting previously unselected package postgresql-client-10.
(Reading database ... 43929 files and directories currently installed.)
Preparing to unpack .../postgresql-client-10_10.13-1.pgdg100+1_amd64.deb ...
Unpacking postgresql-client-10 (10.13-1.pgdg100+1) ...
Selecting previously unselected package postgresql-10.
Preparing to unpack .../postgresql-10_10.13-1.pgdg100+1_amd64.deb ...
Unpacking postgresql-10 (10.13-1.pgdg100+1) ...
Setting up postgresql-client-10 (10.13-1.pgdg100+1) ...
update-alternatives: warning: forcing reinstallation of alternative /usr/share/postgresql/12/man/man1/psql.1.gz because link group psql.1.gz is broken
Setting up postgresql-10 (10.13-1.pgdg100+1) ...
Creating new PostgreSQL cluster 10/main ...
/usr/lib/postgresql/10/bin/initdb -D /var/lib/postgresql/10/main --auth-local peer --auth-host md5
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "C.UTF-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/10/main ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default timezone ... Etc/UTC
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

Success. You can now start the database server using:

pg_ctlcluster 10 main start

Warning: The selected stats_temp_directory /var/run/postgresql/10-main.pg_stat_tmp
is not writable for the cluster owner. Not adding this setting in
postgresql.conf.
Ver Cluster Port Status Owner    Data directory              Log file
10  main    5433 down   postgres /var/lib/postgresql/10/main /var/log/postgresql/postgresql-10-main.log
update-alternatives: warning: forcing reinstallation of alternative /usr/share/postgresql/12/man/man1/postmaster.1.gz because link group postmaster.1.gz is broken
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Processing triggers for postgresql-common (213.pgdg100+1) ...
Building PostgreSQL dictionaries from installed myspell/hunspell packages...
Removing obsolete dictionary files:
Stopping PostgreSQL 10 database server: main.
Stopping PostgreSQL 12 database server: main.
Performing Consistency Checks
-----------------------------
Checking cluster versions                                   ok
Checking database user is the install user                  ok
Checking database connection settings                       ok
Checking for prepared transactions                          ok
Checking for reg* data types in user tables                 ok
Checking for contrib/isn with bigint-passing mismatch       ok
Checking for tables WITH OIDS                               ok
Checking for invalid "sql_identifier" user columns          ok
Creating dump of global objects                             ok
Creating dump of database schemas
  discourse
  postgres
  template1
                                                        ok
Checking for presence of required libraries                 ok
Checking database user is the install user                  ok
Checking for prepared transactions                          ok

If pg_upgrade fails after this point, you must re-initdb the
new cluster before continuing.

Performing Upgrade
------------------
Analyzing all rows in the new cluster                       ok
Freezing all rows in the new cluster                        ok
Deleting files from new pg_xact                             ok
Copying old pg_xact to new server                           ok
Setting next transaction ID and epoch for new cluster       ok
Deleting files from new pg_multixact/offsets                ok
Copying old pg_multixact/offsets to new server              ok
Deleting files from new pg_multixact/members                ok
Copying old pg_multixact/members to new server              ok
Setting next multixact ID and offset for new cluster        ok
Resetting WAL archives                                      ok
Setting frozenxid and minmxid counters in new cluster       ok
Restoring global objects in the new cluster                 ok
Restoring database schemas in the new cluster
  template1
  discourse
  postgres
                                                        ok
Copying user relation files
  /shared/postgres_data/base/16400/2613
  /shared/postgres_data/base/16400/2683
  /shared/postgres_data/base/16400/16728
  /shared/postgres_data/base/16400/16728_fsm
  /shared/postgres_data/base/16400/16728_vm
  /shared/postgres_data/base/16400/16731
  /shared/postgres_data/base/16400/84163
  /shared/postgres_data/base/16400/84163_fsm
  /shared/postgres_data/base/16400/84163_vm
……
                                                        ok
Setting next OID for new cluster                            ok
Sync data directory to disk                                 ok
Creating script to analyze new cluster                      ok
Creating script to delete old cluster                       ok

Upgrade Complete
----------------
Optimizer statistics are not transferred by pg_upgrade so,
once you start the new server, consider running:
./analyze_new_cluster.sh

Running this script will delete the old cluster's data files:
./delete_old_cluster.sh
-------------------------------------------------------------------------------------
UPGRADE OF POSTGRES COMPLETE

Old 10 database is stored at /shared/postgres_data_old

To complete the upgrade, rebuild again using:

./launcher rebuild app
-------------------------------------------------------------------------------------
2 Likes

Looks like there are some files remaining from 2 years ago since the last upgrade:

mv: cannot move '/shared/postgres_data' to '/shared/postgres_data_old/postgres_data': Directory not empty
mv: cannot move '/shared/postgres_data_new' to '/shared/postgres_data/postgres_data_new': Directory not empty

Can you move those files elsewhere before rebuilding?

3 Likes

That was it! Thank you, @Falco.

2 Likes

Thanks, added this to the FAQ. I will also add detection to this state in the upgrade script.

3 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.