Errors were encountered while processing: postgresql-contrib-9.5

CamilleRoux · 2016-11-08 16:53:54 UTC

Hi!

My instance failed to bootstrap (beta Git version).

Here is the templates I’m using

templates:
  - "templates/postgres.9.3.template.yml"
  - "templates/redis.template.yml"
  - "templates/sshd.template.yml"
  - "templates/web.template.yml"
  - "templates/web.ssl.template.yml"
  - "templates/web.letsencrypt.ssl.template.yml"
  #- "templates/web.ratelimited.template.yml"

Here is what I get when I execute ./launcher rebuild app

Ensuring launcher is up to date
Fetching origin
Launcher is up-to-date
Stopping old container
+ /usr/bin/docker stop -t 10 app
app
cd /pups && git pull && /pups/bin/pups --stdin
Already up-to-date.
I, [2016-11-08T16:42:43.632690 #15]  INFO -- : Loading --stdin
I, [2016-11-08T16:42:43.641225 #15]  INFO -- : > apt-get remove -y postgresql-9.5 postgresql-client-9.5 postgresql-contrib-9.5
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
	LANGUAGE = (unset),
	LC_ALL = (unset),
	LANG = "fr_FR.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
E: Sub-process /usr/bin/dpkg returned an error code (1)
I, [2016-11-08T16:42:45.908508 #15]  INFO -- : Reading package lists...
Building dependency tree...
Reading state information...
The following packages will be REMOVED:
  postgresql-9.5 postgresql-client-9.5 postgresql-contrib-9.5
0 upgraded, 0 newly installed, 3 to remove and 0 not upgraded.
After this operation, 26.1 MB disk space will be freed.
(Reading database ... 34341 files and directories currently installed.)
Removing postgresql-contrib-9.5 (9.5.5-1.pgdg16.04+1) ...
update-alternatives: warning: forcing reinstallation of alternative /usr/share/postgresql/9.5/man/man1/postmaster.1.gz because link group postmaster.1.gz is broken
update-alternatives: error: error creating symbolic link '/usr/share/man/man1/postmaster.1.gz.dpkg-tmp': No such file or directory
dpkg: error processing package postgresql-contrib-9.5 (--remove):
 subprocess installed pre-removal script returned error exit status 2
dpkg: postgresql-9.5: dependency problems, but removing anyway as you requested:
 postgresql-contrib-9.5 depends on postgresql-9.5 (= 9.5.5-1.pgdg16.04+1).

Removing postgresql-9.5 (9.5.5-1.pgdg16.04+1) ...
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of stop.
Removing postgresql-client-9.5 (9.5.5-1.pgdg16.04+1) ...
Processing triggers for postgresql-common (177.pgdg16.04+1) ...
Building PostgreSQL dictionaries from installed myspell/hunspell packages...
Removing obsolete dictionary files:
Errors were encountered while processing:
 postgresql-contrib-9.5



FAILED
--------------------
Pups::ExecError: apt-get remove -y postgresql-9.5 postgresql-client-9.5 postgresql-contrib-9.5 failed with return #<Process::Status: pid 17 exit 100>
Location of failure: /pups/lib/pups/exec_command.rb:108:in `spawn'
exec failed with the params "apt-get remove -y postgresql-9.5 postgresql-client-9.5 postgresql-contrib-9.5"
4a144bfb01e96e35da31aa0dae839f3e344cc9e0b667e20e6ae355f36251c673
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one

Falco · 2016-11-08 17:29:02 UTC

A current app.yml should be using:

templates:
  - "templates/postgres.template.yml"
  - "templates/redis.template.yml"
  - "templates/web.template.yml"
  - "templates/web.ratelimited.template.yml"
## Uncomment these two lines if you wish to add Lets Encrypt (https)
  - "templates/web.ssl.template.yml"
  - "templates/web.letsencrypt.ssl.template.yml"

Why are you using PostgreSQL 9.3? Take a backup, fix templates, rebuild and, if needed, restore the backup.

CamilleRoux · 2016-11-08 17:05:32 UTC

I choose the 9.3 because there was some bug with the LANG value.
Now if I’m using templates/postgres.template.yml, I get the following message in logs:

Failure, exiting
-------------------------------------------------------------------------------------
UPGRADE OF POSTGRES FAILED

You are going to need to export your data and import into a clean instance:

In containers/app.yml: Change "templates/postgres.template.yml" TO "templates/postgres.9.3.template.yml"

Run ./launcher rebuild app again

When your instance is running:
Run ./launcher enter app
Run cd /shared/postgres_backup && sudo -u postgres pg_dump discourse > backup.db

Undo the postgres template in your container config
Run: ./launcher stop app
Run: sudo mv /var/discourse/shared/standalone/postgres_data /var/discourse/shared/standalone/postgres_data_old
Run: ./launcher rebuild app

Run: ./launcher enter app
Run: cd /shared/postgres_backup
Run: sv stop unicorn
Run: sudo -iu postgres dropdb discourse
Run: sudo -iu postgres createdb discourse
Run: sudo -iu postgres psql discourse < backup.db
Run: exit
Run: ./launcher rebuild app



FAILED
--------------------
Pups::ExecError: /root/upgrade_postgres failed with return #<Process::Status: pid 39 exit 1>
Location of failure: /pups/lib/pups/exec_command.rb:108:in `spawn'
exec failed with the params "/root/upgrade_postgres"
ebd56652d59e7b6edf01ea0d356cf73719b37fc918a87d6f6db292bf18e6aa68
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one

But, because rebuild is failing with the template postgres.9.3.template.yml, I don’t know what to do…

sphyden · 2016-12-05 17:03:33 UTC

Im running into the same catch 22. The failed bootstrap give the instructions, but instructions can’t be run because the app will not rebuild.

Falco · 2016-12-05 17:08:23 UTC

First of all, take a backup and download it!

Then follow this:

cd /var/discourse
mv shared shared_old

Then edit app.yml, removing postgres version from it. After the editing the first lines should look like this:

## this is the all-in-one, standalone Discourse Docker container template
##
## After making changes to this file, you MUST rebuild
## /var/discourse/launcher rebuild app
##
## BE *VERY* CAREFUL WHEN EDITING!
## YAML FILES ARE SUPER SUPER SENSITIVE TO MISTAKES IN WHITESPACE OR ALIGNMENT!
## visit http://www.yamllint.com/ to validate this file as needed
templates:
  - "templates/postgres.template.yml"
  - "templates/redis.template.yml"
  - "templates/web.template.yml"
  - "templates/web.ratelimited.template.yml"
## Uncomment these two lines if you wish to add Lets Encrypt (https)
...

Then ./launcher rebuild app, create an admin, upload a backup and restore. Just make sure to have plenty of space.

This procedure isn’t the fastest, but will work to restore a broken Discourse from any point to a usable and working install.

sphyden · 2016-12-07 19:45:55 UTC

I am migrating my companies instance of discourse from CentOS 6 to CentOS 7 host. I can no longer update Discourse on Cent 6 because there is no new Docker versions. The first thing that I tried is what you mention above. I can import my backup, and successfully restore my data. The catch is I am no longer able to log into the site. Its in some weird state that the new instance thinks that I am logged in from the old instance, causing a log in loop on the new instance. I tried to use the api gem to force log myself out, but you need admin priv’s to do this. I cannot log in, in any form or fashion on the new instance to use the log out API call.

After that, I decided to attempt to simply stand up a new instance, and sync my standalone dir. on top of the new one, considering you offer a psql 9.3 template. Unfortunately I cannot bootstrap with the 9.3 template due to the apt-get errors mentioned above. If I bootstrap with the vanilla postgres template, I get schema errors because my old instance was running on 9.3.

I guess my next option will be to import the old Docker container via registry, and sync over the entire old “var/docker” directory to my new server, then fingers crossed when I attempt to actually update the instance.

Please let me know if there is anything else I can try here.

Falco · 2016-12-07 20:04:42 UTC

Try to logout trough rails console:

u = User.find_by_id(YOUR_ID_HERE)
u.update_attribute auth_token: nil

sphyden · 2016-12-08 14:40:03 UTC

Is there a way that I can do this will all users simultaneously? Also, I can’t seem to find userids on our current production instance.

Falco · 2016-12-08 16:31:08 UTC

Something like

User.update_all(auth_token: nil)

should work.

sphyden · 2016-12-08 17:20:49 UTC

I just stood up a new instance of Discourse, imported and restored to a backup of my production instance via the web Admin panel. At that point, all of my data shows up and the forum looks as it should, BUT I get stuck in the log in look explained earlier.

I tried the rails command quoted above, with the following return:

[1] pry(main)> User.update_all(auth_token: nil)
=> 1884

After that, I attempt to log in to the forum via web again, and again get the log in loop. It successfully authenticates, but never acts as If i am logged in.

Falco · 2016-12-08 17:45:37 UTC

Do you have some sort of outside reverse proxy? Is SSL enabled?

sphyden · 2016-12-08 18:21:04 UTC

I have SSL enabled on my production instance, but not the current test instance. The web.ssl.template is being used in production, and commented out in test.

sphyden · 2016-12-08 20:01:41 UTC

I’ve confirmed that the auth_token for at least my user is nil. Still unable to log in.

 => #<User:0x007f16814c8968
  id: 92,
  username: "steven",
  created_at: Tue, 05 Aug 2014 15:10:41 UTC +00:00,
  updated_at: Sat, 18 Apr 2015 07:09:54 UTC +00:00,
  name: "Steven",
  seen_notification_id: 0,
  last_posted_at: Sat, 18 Apr 2015 07:09:54 UTC +00:00,
  email: "shyden@goldenfrog.com",

  active: true,
  username_lower: "steven",
  auth_token: nil,

Falco · 2016-12-08 21:04:53 UTC

Can you share a link?

If you had HSTS before this can wreak havoc.

sphyden · 2016-12-08 21:50:09 UTC

I will work on getting the site opened up. In the mean time, I did find this in my production log, from attempted log in’s:

  Started GET "/session/csrf?_=1481229888046" for ip at 2016-12-08 20:46:39 +0000
 Processing by SessionController#csrf as */*
  Parameters: {"_"=>"1481229888046"}
Completed 200 OK in 1ms (Views: 0.2ms | ActiveRecord: 0.0ms)
Started POST "/session" for ip at 2016-12-08 20:46:39 +0000
Processing by SessionController#create as */*
  Parameters: {"login"=>"chu@phmgmt.com", "password"=>"[FILTERED]"}
Can't verify CSRF token authenticity
  Rendered text template (0.0ms)
Filter chain halted as :verify_authenticity_token rendered or redirected
Completed 403 Forbidden in 1ms (Views: 0.3ms | ActiveRecord: 0.0ms)

sphyden · 2016-12-13 16:52:15 UTC

I ended up getting Discourse over to my Cent 7 server, by pushing the current container to our registry, and copying the entire /var/docker directory over. I am able to get a functioning version of our Discourse working that way. I am still unable to update discourse due to the following error:

Pups::ExecError: /root/upgrade_postgres failed with return #<Process::Status: pid 40 exit 1>
Location of failure: /pups/lib/pups/exec_command.rb:108:in `spawn'
exec failed with the params "/root/upgrade_postgres"
e68c256f43b7101b5c7003b8db31b80ac83e845999e9f15bec369592d9d04649
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one

This using the update method:

./launcher stop discourse
git pull
./launcher rebuild discourse

This of course leads me back to my loop placing the 9.3 postgres template in my discourse.yml and bootstrap failing again due to apt errors.

I am attempting to disable HSTS now to see if I can get the app to rebuild that way.

sam · 2016-12-14 07:23:46 UTC

You got to follow the manual backup / restore step documented there if the automatic upgrade fails.