Site Won't Start After Update

Here's the output of my rebuild:

Ensuring launcher is up to date
Fetching origin
Launcher is up-to-date
Stopping old container
+ /usr/bin/docker stop -t 10 app
app
cd /pups && git pull && /pups/bin/pups --stdin
Already up-to-date.
I, [2018-07-28T22:17:48.984452 #14]  INFO -- : Loading --stdin
I, [2018-07-28T22:17:48.994487 #14]  INFO -- : > echo cron is now included in base image, remove from templates
I, [2018-07-28T22:17:48.997417 #14]  INFO -- : cron is now included in base image, remove from templates

I, [2018-07-28T22:17:48.998028 #14]  INFO -- : > locale-gen $LANG && update-locale
I, [2018-07-28T22:17:49.953451 #14]  INFO -- : Generating locales (this might take a while)...
  en_US.UTF-8... done
Generation complete.

I, [2018-07-28T22:17:49.954153 #14]  INFO -- : > mkdir -p /shared/postgres_run
I, [2018-07-28T22:17:49.957081 #14]  INFO -- : 
I, [2018-07-28T22:17:49.957637 #14]  INFO -- : > chown postgres:postgres /shared/postgres_run
I, [2018-07-28T22:17:49.960692 #14]  INFO -- : 
I, [2018-07-28T22:17:49.961242 #14]  INFO -- : > chmod 775 /shared/postgres_run
I, [2018-07-28T22:17:49.963779 #14]  INFO -- : 
I, [2018-07-28T22:17:49.964349 #14]  INFO -- : > rm -fr /var/run/postgresql
I, [2018-07-28T22:17:49.967345 #14]  INFO -- : 
I, [2018-07-28T22:17:49.968003 #14]  INFO -- : > ln -s /shared/postgres_run /var/run/postgresql
I, [2018-07-28T22:17:49.970526 #14]  INFO -- : 
I, [2018-07-28T22:17:49.971036 #14]  INFO -- : > socat /dev/null UNIX-CONNECT:/shared/postgres_run/.s.PGSQL.5432 || exit 0 && echo postgres already running stop container ; exit 1
2018/07/28 22:17:49 socat[47] E connect(6, AF=1 "/shared/postgres_run/.s.PGSQL.5432", 36): No such file or directory
I, [2018-07-28T22:17:49.981306 #14]  INFO -- : 
I, [2018-07-28T22:17:49.981691 #14]  INFO -- : > rm -fr /shared/postgres_run/.s*
I, [2018-07-28T22:17:49.984271 #14]  INFO -- : 
I, [2018-07-28T22:17:49.984679 #14]  INFO -- : > rm -fr /shared/postgres_run/*.pid
I, [2018-07-28T22:17:49.987225 #14]  INFO -- : 
I, [2018-07-28T22:17:49.987689 #14]  INFO -- : > mkdir -p /shared/postgres_run/10-main.pg_stat_tmp
I, [2018-07-28T22:17:49.991726 #14]  INFO -- : 
I, [2018-07-28T22:17:49.992307 #14]  INFO -- : > chown postgres:postgres /shared/postgres_run/10-main.pg_stat_tmp
I, [2018-07-28T22:17:49.994447 #14]  INFO -- : 
I, [2018-07-28T22:17:49.998945 #14]  INFO -- : File > /etc/service/postgres/run  chmod: +x
I, [2018-07-28T22:17:50.002648 #14]  INFO -- : File > /etc/service/postgres/log/run  chmod: +x
I, [2018-07-28T22:17:50.007015 #14]  INFO -- : File > /etc/runit/3.d/99-postgres  chmod: +x
I, [2018-07-28T22:17:50.010727 #14]  INFO -- : File > /root/upgrade_postgres  chmod: +x
I, [2018-07-28T22:17:50.011198 #14]  INFO -- : > chown -R root /var/lib/postgresql/10/main
I, [2018-07-28T22:17:50.359304 #14]  INFO -- : 
I, [2018-07-28T22:17:50.359656 #14]  INFO -- : > [ ! -e /shared/postgres_data ] && install -d -m 0755 -o postgres -g postgres /shared/postgres_data && sudo -E -u postgres /usr/lib/postgresql/10/bin/initdb -D /shared/postgres_data || exit 0
I, [2018-07-28T22:17:50.362080 #14]  INFO -- : 
I, [2018-07-28T22:17:50.362645 #14]  INFO -- : > chown -R postgres:postgres /shared/postgres_data
I, [2018-07-28T22:17:50.428220 #14]  INFO -- : 
I, [2018-07-28T22:17:50.429117 #14]  INFO -- : > chown -R postgres:postgres /var/run/postgresql
I, [2018-07-28T22:17:50.431922 #14]  INFO -- : 
I, [2018-07-28T22:17:50.432440 #14]  INFO -- : > /root/upgrade_postgres

WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
dpkg-preconfigure: unable to re-open stdin: 
mv: cannot move '/shared/postgres_data' to '/shared/postgres_data_old/postgres_data': Directory not empty
mv: cannot move '/shared/postgres_data_new' to '/shared/postgres_data/postgres_data_new': Directory not empty
I, [2018-07-28T22:18:11.539427 #14]  INFO -- : Upgrading PostgreSQL from version 9.5 to 10
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /shared/postgres_data_new ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

Success. You can now start the database server using:

    /usr/lib/postgresql/10/bin/pg_ctl -D /shared/postgres_data_new -l logfile start

Hit:1 http://archive.ubuntu.com/ubuntu xenial InRelease
Hit:2 https://deb.nodesource.com/node_8.x xenial InRelease
Get:3 http://archive.ubuntu.com/ubuntu xenial-updates InRelease [109 kB]
Get:4 http://apt.postgresql.org/pub/repos/apt xenial-pgdg InRelease [51.4 kB]
Get:5 http://archive.ubuntu.com/ubuntu xenial-security InRelease [107 kB]
Get:6 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages [1,058 kB]
Get:7 http://apt.postgresql.org/pub/repos/apt xenial-pgdg/main amd64 Packages [179 kB]
Get:8 http://archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages [868 kB]
Get:9 http://archive.ubuntu.com/ubuntu xenial-security/main amd64 Packages [675 kB]
Get:10 http://archive.ubuntu.com/ubuntu xenial-security/universe amd64 Packages [457 kB]
Fetched 3,505 kB in 1s (2,588 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  postgresql-client-9.5 postgresql-contrib-9.5
Suggested packages:
  locales-all postgresql-doc-9.5 libdbd-pg-perl
The following NEW packages will be installed:
  postgresql-9.5 postgresql-client-9.5 postgresql-contrib-9.5
0 upgraded, 3 newly installed, 0 to remove and 15 not upgraded.
Need to get 5,610 kB of archives.
After this operation, 26.3 MB of additional disk space will be used.
Get:1 http://apt.postgresql.org/pub/repos/apt xenial-pgdg/main amd64 postgresql-client-9.5 amd64 9.5.13-2.pgdg16.04+1 [1,194 kB]
Get:2 http://apt.postgresql.org/pub/repos/apt xenial-pgdg/main amd64 postgresql-9.5 amd64 9.5.13-2.pgdg16.04+1 [3,961 kB]
Get:3 http://apt.postgresql.org/pub/repos/apt xenial-pgdg/main amd64 postgresql-contrib-9.5 amd64 9.5.13-2.pgdg16.04+1 [455 kB]
Fetched 5,610 kB in 1s (4,395 kB/s)
Selecting previously unselected package postgresql-client-9.5.
(Reading database ... 36351 files and directories currently installed.)
Preparing to unpack .../postgresql-client-9.5_9.5.13-2.pgdg16.04+1_amd64.deb ...
Unpacking postgresql-client-9.5 (9.5.13-2.pgdg16.04+1) ...
Selecting previously unselected package postgresql-9.5.
Preparing to unpack .../postgresql-9.5_9.5.13-2.pgdg16.04+1_amd64.deb ...
Unpacking postgresql-9.5 (9.5.13-2.pgdg16.04+1) ...
Selecting previously unselected package postgresql-contrib-9.5.
Preparing to unpack .../postgresql-contrib-9.5_9.5.13-2.pgdg16.04+1_amd64.deb ...
Unpacking postgresql-contrib-9.5 (9.5.13-2.pgdg16.04+1) ...
Processing triggers for postgresql-common (191.pgdg16.04+1) ...
Building PostgreSQL dictionaries from installed myspell/hunspell packages...
Removing obsolete dictionary files:
Setting up postgresql-client-9.5 (9.5.13-2.pgdg16.04+1) ...
update-alternatives: warning: forcing reinstallation of alternative /usr/share/postgresql/10/man/man1/psql.1.gz because link group psql.1.gz is broken
Setting up postgresql-9.5 (9.5.13-2.pgdg16.04+1) ...
Creating new PostgreSQL cluster 9.5/main ...
/usr/lib/postgresql/9.5/bin/initdb -D /var/lib/postgresql/9.5/main --auth-local peer --auth-host md5
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "C.UTF-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/9.5/main ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
creating template1 database in /var/lib/postgresql/9.5/main/base/1 ... ok
initializing pg_authid ... ok
initializing dependencies ... ok
creating system views ... ok
loading system objects' descriptions ... ok
creating collations ... ok
creating conversions ... ok
creating dictionaries ... ok
setting privileges on built-in objects ... ok
creating information schema ... ok
loading PL/pgSQL server-side language ... ok
vacuuming database template1 ... ok
copying template1 to template0 ... ok
copying template1 to postgres ... ok
syncing data to disk ... ok

Success. You can now start the database server using:

    /usr/lib/postgresql/9.5/bin/pg_ctl -D /var/lib/postgresql/9.5/main -l logfile start

Ver Cluster Port Status Owner    Data directory               Log file
9.5 main    5433 down   postgres /var/lib/postgresql/9.5/main /var/log/postgresql/postgresql-9.5-main.log
update-alternatives: warning: forcing reinstallation of alternative /usr/share/postgresql/10/man/man1/postmaster.1.gz because link group postmaster.1.gz is broken
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Setting up postgresql-contrib-9.5 (9.5.13-2.pgdg16.04+1) ...
 * Stopping PostgreSQL 10 database server
   ...done.
 * Stopping PostgreSQL 9.5 database server
   ...done.
Performing Consistency Checks
-----------------------------
Checking cluster versions                                   ok
Checking database user is the install user                  ok
Checking database connection settings                       ok
Checking for prepared transactions                          ok
Checking for reg* data types in user tables                 ok
Checking for contrib/isn with bigint-passing mismatch       ok
Checking for invalid "unknown" user columns                 ok
Checking for roles starting with "pg_"                      ok
Creating dump of global objects                             ok
Creating dump of database schemas
  discourse
  postgres
  template1
                                                            ok
Checking for presence of required libraries                 ok
Checking database user is the install user                  ok
Checking for prepared transactions                          ok

If pg_upgrade fails after this point, you must re-initdb the
new cluster before continuing.

Performing Upgrade
------------------
Analyzing all rows in the new cluster                       ok
Freezing all rows in the new cluster                        ok
Deleting files from new pg_xact                             ok
Copying old pg_clog to new server                           ok
Setting next transaction ID and epoch for new cluster       ok
Deleting files from new pg_multixact/offsets                ok
Copying old pg_multixact/offsets to new server              ok
Deleting files from new pg_multixact/members                ok
Copying old pg_multixact/members to new server              ok
Setting next multixact ID and offset for new cluster        ok
Resetting WAL archives                                      ok
Setting frozenxid and minmxid counters in new cluster       ok
Restoring global objects in the new cluster                 ok
Restoring database schemas in the new cluster
  discourse
  postgres
  template1
                                                            ok
Copying user relation files
 ...
                                                            ok
Setting next OID for new cluster                            ok
Sync data directory to disk                                 ok
Creating script to analyze new cluster                      ok
Creating script to delete old cluster                       ok
Checking for hash indexes                                   ok

Upgrade Complete
----------------
Optimizer statistics are not transferred by pg_upgrade so,
once you start the new server, consider running:
    ./analyze_new_cluster.sh

Running this script will delete the old cluster's data files:
    ./delete_old_cluster.sh
-------------------------------------------------------------------------------------
UPGRADE OF POSTGRES COMPLETE

Old 9.5 database is stored at /shared/postgres_data_old

To complete the upgrade, rebuild again using:

./launcher rebuild app
-------------------------------------------------------------------------------------

120fbc5275200c1b02d58a46336721f82bff056f832acd22256050a57e2e0d33

Now when I navigate to the website in a browser, I just get a ERR_CONNECTION_REFUSED. How do I fix it?

I did ./launcher start app after that and it seems to be working now? But if I reboot it’s not working again. What went wrong here?

Did you do this a second time like it says?

1 Like

Yea, I’ve done it like 5 times.

1 Like

What does this mean during ./launcher app rebuild?

WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
dpkg-preconfigure: unable to re-open stdin:

Is your database somehow inside your container, what is your app.yml.

Hey thanks for responding, but I punted and just created a new droplet, reinstalled docker and discourse, and restored from a backup of discourse from the broken droplet. Seems good now, though not sure what went wrong.

3 Likes