Error message after rebuild app (UPGRADE OF POSTGRES FAILED)


(Markus) #1

It seems, my instance is broken after trying to update / rebuild app in console.

Any ideas how to fix this very quickly?
BIG THANKS IN ADVANCE!

I need this instance running in the next few hours. :thermometer_face:

root@Ubuntu-1510-wily-64-minimal /var/discourse # ./launcher rebuild app
Ensuring launcher is up to date
Fetching origin
Launcher is up-to-date
Stopping old container
+ /usr/bin/docker stop -t 10 app
app
cd /pups && git pull && /pups/bin/pups --stdin
Already up-to-date.
I, [2016-07-01T07:07:14.703329 #15]  INFO -- : Loading --stdin
I, [2016-07-01T07:07:14.717498 #15]  INFO -- : > mkdir -p /shared/postgres_run
I, [2016-07-01T07:07:14.721683 #15]  INFO -- : 
I, [2016-07-01T07:07:14.722510 #15]  INFO -- : > chown postgres:postgres /shared/postgres_run
I, [2016-07-01T07:07:14.733969 #15]  INFO -- : 
I, [2016-07-01T07:07:14.735065 #15]  INFO -- : > chmod 775 /shared/postgres_run
I, [2016-07-01T07:07:14.740404 #15]  INFO -- : 
I, [2016-07-01T07:07:14.741202 #15]  INFO -- : > rm -fr /var/run/postgresql
I, [2016-07-01T07:07:14.744714 #15]  INFO -- : 
I, [2016-07-01T07:07:14.745543 #15]  INFO -- : > ln -s /shared/postgres_run /var/run/postgresql
I, [2016-07-01T07:07:14.749176 #15]  INFO -- : 
I, [2016-07-01T07:07:14.750030 #15]  INFO -- : > socat /dev/null UNIX-CONNECT:/shared/postgres_run/.s.PGSQL.5432 || exit 0 && echo postgres already running stop container ; exit 1
2016/07/01 07:07:14 socat[23] E connect(6, AF=1 "/shared/postgres_run/.s.PGSQL.5432", 36): No such file or directory
I, [2016-07-01T07:07:14.793527 #15]  INFO -- : 
I, [2016-07-01T07:07:14.794009 #15]  INFO -- : > rm -fr /shared/postgres_run/.s*
I, [2016-07-01T07:07:14.801778 #15]  INFO -- : 
I, [2016-07-01T07:07:14.802733 #15]  INFO -- : > rm -fr /shared/postgres_run/*.pid
I, [2016-07-01T07:07:14.814025 #15]  INFO -- : 
I, [2016-07-01T07:07:14.815190 #15]  INFO -- : > mkdir -p /shared/postgres_run/9.5-main.pg_stat_tmp
I, [2016-07-01T07:07:14.818464 #15]  INFO -- : 
I, [2016-07-01T07:07:14.819381 #15]  INFO -- : > chown postgres:postgres /shared/postgres_run/9.5-main.pg_stat_tmp
I, [2016-07-01T07:07:14.832486 #15]  INFO -- : 
I, [2016-07-01T07:07:14.839736 #15]  INFO -- : File > /etc/service/postgres/run  chmod: +x
I, [2016-07-01T07:07:14.854120 #15]  INFO -- : File > /etc/runit/3.d/99-postgres  chmod: +x
I, [2016-07-01T07:07:14.859639 #15]  INFO -- : File > /root/upgrade_postgres  chmod: +x
I, [2016-07-01T07:07:14.861734 #15]  INFO -- : > chown -R root /var/lib/postgresql/9.5/main
I, [2016-07-01T07:07:15.098538 #15]  INFO -- : 
I, [2016-07-01T07:07:15.099222 #15]  INFO -- : > [ ! -e /shared/postgres_data ] && install -d -m 0755 -o postgres -g postgres /shared/postgres_data && sudo -E -u postgres /usr/lib/postgresql/9.5/bin/initdb -D /shared/postgres_data || exit 0
I, [2016-07-01T07:07:15.102704 #15]  INFO -- : 
I, [2016-07-01T07:07:15.103474 #15]  INFO -- : > chown -R postgres:postgres /shared/postgres_data
I, [2016-07-01T07:07:15.145029 #15]  INFO -- : 
I, [2016-07-01T07:07:15.146012 #15]  INFO -- : > chown -R postgres:postgres /var/run/postgresql
I, [2016-07-01T07:07:15.151614 #15]  INFO -- : 
I, [2016-07-01T07:07:15.157353 #15]  INFO -- : > /root/upgrade_postgres

WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
dpkg-preconfigure: unable to re-open stdin: 
I, [2016-07-01T07:08:00.957927 #15]  INFO -- : Upgrading PostgreSQL to from version 9.3 to 9.5
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /shared/postgres_data_new ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
creating template1 database in /shared/postgres_data_new/base/1 ... ok
initializing pg_authid ... ok
initializing dependencies ... ok
creating system views ... ok
loading system objects' descriptions ... ok
creating collations ... ok
creating conversions ... ok
creating dictionaries ... ok
setting privileges on built-in objects ... ok
creating information schema ... ok
loading PL/pgSQL server-side language ... ok
vacuuming database template1 ... ok
copying template1 to template0 ... ok
copying template1 to postgres ... ok
syncing data to disk ... ok

Success. You can now start the database server using:

    /usr/lib/postgresql/9.5/bin/pg_ctl -D /shared/postgres_data_new -l logfile start

Get:1 http://apt.postgresql.org/pub/repos/apt xenial-pgdg InRelease [26.5 kB]
Hit:2 http://archive.ubuntu.com/ubuntu xenial InRelease
Get:3 http://archive.ubuntu.com/ubuntu xenial-updates InRelease [94.5 kB]
Get:4 http://archive.ubuntu.com/ubuntu xenial-security InRelease [94.5 kB]
Get:5 http://apt.postgresql.org/pub/repos/apt xenial-pgdg/main amd64 Packages [64.9 kB]
Get:6 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages [303 kB]
Get:7 http://archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages [157 kB]
Get:8 http://archive.ubuntu.com/ubuntu xenial-security/main amd64 Packages [141 kB]
Get:9 http://archive.ubuntu.com/ubuntu xenial-security/universe amd64 Packages [34.3 kB]
Get:10 https://deb.nodesource.com/node_4.x xenial InRelease [3,914 B]
Get:11 https://deb.nodesource.com/node_4.x xenial/main Sources [762 B]
Get:12 https://deb.nodesource.com/node_4.x xenial/main amd64 Packages [961 B]
Fetched 921 kB in 1s (624 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  libossp-uuid16 postgresql-client-9.3 postgresql-contrib-9.3
Suggested packages:
  uuid locales-all postgresql-doc-9.3 libdbd-pg-perl
The following NEW packages will be installed:
  libossp-uuid16 postgresql-9.3 postgresql-client-9.3 postgresql-contrib-9.3
0 upgraded, 4 newly installed, 0 to remove and 26 not upgraded.
Need to get 4,990 kB of archives.
After this operation, 23.9 MB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu xenial/universe amd64 libossp-uuid16 amd64 1.6.2-1.5build2 [28.6 kB]
Get:2 http://apt.postgresql.org/pub/repos/apt xenial-pgdg/main amd64 postgresql-client-9.3 amd64 9.3.13-1.pgdg16.04+1 [1,038 kB]
Get:3 http://apt.postgresql.org/pub/repos/apt xenial-pgdg/main amd64 postgresql-9.3 amd64 9.3.13-1.pgdg16.04+1 [3,515 kB]
Get:4 http://apt.postgresql.org/pub/repos/apt xenial-pgdg/main amd64 postgresql-contrib-9.3 amd64 9.3.13-1.pgdg16.04+1 [409 kB]
Fetched 4,990 kB in 0s (8,259 kB/s)
Selecting previously unselected package libossp-uuid16:amd64.
(Reading database ... 34207 files and directories currently installed.)
Preparing to unpack .../libossp-uuid16_1.6.2-1.5build2_amd64.deb ...
Unpacking libossp-uuid16:amd64 (1.6.2-1.5build2) ...
Selecting previously unselected package postgresql-client-9.3.
Preparing to unpack .../postgresql-client-9.3_9.3.13-1.pgdg16.04+1_amd64.deb ...
Unpacking postgresql-client-9.3 (9.3.13-1.pgdg16.04+1) ...
Selecting previously unselected package postgresql-9.3.
Preparing to unpack .../postgresql-9.3_9.3.13-1.pgdg16.04+1_amd64.deb ...
Unpacking postgresql-9.3 (9.3.13-1.pgdg16.04+1) ...
Selecting previously unselected package postgresql-contrib-9.3.
Preparing to unpack .../postgresql-contrib-9.3_9.3.13-1.pgdg16.04+1_amd64.deb ...
Unpacking postgresql-contrib-9.3 (9.3.13-1.pgdg16.04+1) ...
Processing triggers for libc-bin (2.23-0ubuntu3) ...
Processing triggers for postgresql-common (174.pgdg16.04+1) ...
Building PostgreSQL dictionaries from installed myspell/hunspell packages...
Removing obsolete dictionary files:
Setting up libossp-uuid16:amd64 (1.6.2-1.5build2) ...
Setting up postgresql-client-9.3 (9.3.13-1.pgdg16.04+1) ...
update-alternatives: warning: forcing reinstallation of alternative /usr/share/postgresql/9.5/man/man1/psql.1.gz because link group psql.1.gz is broken
Setting up postgresql-9.3 (9.3.13-1.pgdg16.04+1) ...
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Creating new cluster 9.3/main ...
  config /etc/postgresql/9.3/main
  data   /var/lib/postgresql/9.3/main
  locale C
  socket /var/run/postgresql
  port   5433
Warning: The selected stats_temp_directory /var/run/postgresql/9.3-main.pg_stat_tmp
is not writable for the cluster owner. Not adding this setting in
postgresql.conf.
update-alternatives: warning: forcing reinstallation of alternative /usr/share/postgresql/9.5/man/man1/postmaster.1.gz because link group postmaster.1.gz is broken
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Setting up postgresql-contrib-9.3 (9.3.13-1.pgdg16.04+1) ...
Processing triggers for libc-bin (2.23-0ubuntu3) ...
 * Stopping PostgreSQL 9.3 database server
   ...done.
 * Stopping PostgreSQL 9.5 database server
 * Error: Config owner (postgres:107) and data owner (root:0) do not match, and config owner is not root
   ...fail!
Performing Consistency Checks
-----------------------------
Checking cluster versions                                   ok
Checking database user is the install user                  ok
Checking database connection settings                       ok
Checking for prepared transactions                          ok
Checking for reg* system OID user data types                ok
Checking for contrib/isn with bigint-passing mismatch       ok
Checking for invalid "line" user columns                    ok
Creating dump of global objects                             ok
Creating dump of database schemas
  discourse
  postgres
  template1
                                                            ok

encodings for database "postgres" do not match:  old "SQL_ASCII", new "UTF8"
Failure, exiting
-------------------------------------------------------------------------------------
UPGRADE OF POSTGRES FAILED

You are going to need to export your data and import into a clean instance:

In containers/app.yml: Change "templates/postgres.template.yml" TO "templates/postgres.9.3.template.yml"

Run ./launcher rebuild app again

When your instance is running:
Run ./launcher enter app
Run cd /shared/postgres_backup && sudo -u postgres pg_dump discourse > backup.db

Undo the base_image and postgres template from your container config
Run: ./launcher stop app
Run: sudo mv /var/discourse/shared/standalone/postgres_data /var/discourse/shared/standalone/postgres_data_old
Run: ./launcher rebuild app

Run: ./launcher enter app
Run: cd /shared/postgres_backup
Run: sv stop unicorn
Run: sudo -iu postgres dropdb discourse
Run: sudo -iu postgres createdb discourse
Run: sudo -iu postgres psql discourse < backup.db
Run: exit
Run: ./launcher rebuild app



FAILED
--------------------
Pups::ExecError: /root/upgrade_postgres failed with return #<Process::Status: pid 40 exit 1>
Location of failure: /pups/lib/pups/exec_command.rb:108:in `spawn'
exec failed with the params "/root/upgrade_postgres"
65c5e38e8c961e602da5ca96e994573e183d5cdecc814278a28546b48c8ca565
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one
root@Ubuntu-1510-wily-64-minimal /var/discourse #

(Alan Tan) #2
-------------------------------------------------------------------------------------
UPGRADE OF POSTGRES FAILED

You are going to need to export your data and import into a clean instance:

In containers/app.yml: Change "templates/postgres.template.yml" TO "templates/postgres.9.3.template.yml"

Run ./launcher rebuild app again

When your instance is running:
Run ./launcher enter app
Run cd /shared/postgres_backup && sudo -u postgres pg_dump discourse > backup.db

Undo the base_image and postgres template from your container config
Run: ./launcher stop app
Run: sudo mv /var/discourse/shared/standalone/postgres_data /var/discourse/shared/standalone/postgres_data_old
Run: ./launcher rebuild app

Run: ./launcher enter app
Run: cd /shared/postgres_backup
Run: sv stop unicorn
Run: sudo -iu postgres dropdb discourse
Run: sudo -iu postgres createdb discourse
Run: sudo -iu postgres psql discourse < backup.db
Run: exit
Run: ./launcher rebuild app

Follow the instructions in the output :slight_smile:


Upgrade from 1.7.0 beta 5 to beta 7 failed. Postgresql error
Trouble while upgrading docker
Upgrade from v1.7.0.beta2
(Markus) #3
e4da6f64afa6d35e348e7d93734e984894acb26041bc4910ce2ebedb39fb3b67
Removing old container
+ /usr/bin/docker rm app
Failed to remove container (app): Error response from daemon: Conflict, You cannot remove a running container. Stop the container before attempting removal or use -f
5fa0ff93a364
Nothing to do, your container has already started!
root@Ubuntu-1510-wily-64-minimal /var/discourse # 

hmm… :frowning:

The instance isn’t running.


(Markus) #4

After reboot and rebuild, it’s working again :slight_smile: puuuhhh


(Peter N Lewis) #5

Just got this myself. I have to say - quiet terrifying.

Hopefully it will all work our eventually, but this sort of thing is very stressful!

Update: Followed the instructions and all is well again. Sheesh, just doing a rebuild to update the SSL certificate should not cause this much anxiety! And now it tells me “Your server is using default ruby garbage collection parameters, which will not give you the best performance” - can’t wait to delve down that rabbit hole!


(Peter N Lewis) #6

An interesting side affect of this process seems to be to generate a bunch of emails. I received a new “Welcome to Discourse” PM email, as well as emails about the lounge, staff, Assets, meta category, READ ME FIRST Admin Quick Start Guide and Welcome to the Keyboard Maestro Forum.

I suspect it may be related to “mailing list mode” and/or being admin/staff whatever. I know at least one of my other moderators who is in mailing list mode got all the emails too. I’m not sure if others did as well.


(Markus) #7

Nope. Followed the instructions and everything was fine again, Just the error handling and reboot is improvable :wink: