Errors were encountered while processing: postgresql-contrib-9.5

Hi!

My instance failed to bootstrap (beta Git version).

Here is the templates I’m using

templates:
  - "templates/postgres.9.3.template.yml"
  - "templates/redis.template.yml"
  - "templates/sshd.template.yml"
  - "templates/web.template.yml"
  - "templates/web.ssl.template.yml"
  - "templates/web.letsencrypt.ssl.template.yml"
  #- "templates/web.ratelimited.template.yml"

Here is what I get when I execute ./launcher rebuild app

Ensuring launcher is up to date
Fetching origin
Launcher is up-to-date
Stopping old container
+ /usr/bin/docker stop -t 10 app
app
cd /pups && git pull && /pups/bin/pups --stdin
Already up-to-date.
I, [2016-11-08T16:42:43.632690 #15]  INFO -- : Loading --stdin
I, [2016-11-08T16:42:43.641225 #15]  INFO -- : > apt-get remove -y postgresql-9.5 postgresql-client-9.5 postgresql-contrib-9.5
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
	LANGUAGE = (unset),
	LC_ALL = (unset),
	LANG = "fr_FR.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
E: Sub-process /usr/bin/dpkg returned an error code (1)
I, [2016-11-08T16:42:45.908508 #15]  INFO -- : Reading package lists...
Building dependency tree...
Reading state information...
The following packages will be REMOVED:
  postgresql-9.5 postgresql-client-9.5 postgresql-contrib-9.5
0 upgraded, 0 newly installed, 3 to remove and 0 not upgraded.
After this operation, 26.1 MB disk space will be freed.
(Reading database ... 34341 files and directories currently installed.)
Removing postgresql-contrib-9.5 (9.5.5-1.pgdg16.04+1) ...
update-alternatives: warning: forcing reinstallation of alternative /usr/share/postgresql/9.5/man/man1/postmaster.1.gz because link group postmaster.1.gz is broken
update-alternatives: error: error creating symbolic link '/usr/share/man/man1/postmaster.1.gz.dpkg-tmp': No such file or directory
dpkg: error processing package postgresql-contrib-9.5 (--remove):
 subprocess installed pre-removal script returned error exit status 2
dpkg: postgresql-9.5: dependency problems, but removing anyway as you requested:
 postgresql-contrib-9.5 depends on postgresql-9.5 (= 9.5.5-1.pgdg16.04+1).

Removing postgresql-9.5 (9.5.5-1.pgdg16.04+1) ...
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of stop.
Removing postgresql-client-9.5 (9.5.5-1.pgdg16.04+1) ...
Processing triggers for postgresql-common (177.pgdg16.04+1) ...
Building PostgreSQL dictionaries from installed myspell/hunspell packages...
Removing obsolete dictionary files:
Errors were encountered while processing:
 postgresql-contrib-9.5



FAILED
--------------------
Pups::ExecError: apt-get remove -y postgresql-9.5 postgresql-client-9.5 postgresql-contrib-9.5 failed with return #<Process::Status: pid 17 exit 100>
Location of failure: /pups/lib/pups/exec_command.rb:108:in `spawn'
exec failed with the params "apt-get remove -y postgresql-9.5 postgresql-client-9.5 postgresql-contrib-9.5"
4a144bfb01e96e35da31aa0dae839f3e344cc9e0b667e20e6ae355f36251c673
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one

A current app.yml should be using:

templates:
  - "templates/postgres.template.yml"
  - "templates/redis.template.yml"
  - "templates/web.template.yml"
  - "templates/web.ratelimited.template.yml"
## Uncomment these two lines if you wish to add Lets Encrypt (https)
  - "templates/web.ssl.template.yml"
  - "templates/web.letsencrypt.ssl.template.yml"

Why are you using PostgreSQL 9.3? Take a backup, fix templates, rebuild and, if needed, restore the backup.

3 Likes

I choose the 9.3 because there was some bug with the LANG value.
Now if I’m using templates/postgres.template.yml, I get the following message in logs:

Failure, exiting
-------------------------------------------------------------------------------------
UPGRADE OF POSTGRES FAILED

You are going to need to export your data and import into a clean instance:

In containers/app.yml: Change "templates/postgres.template.yml" TO "templates/postgres.9.3.template.yml"

Run ./launcher rebuild app again

When your instance is running:
Run ./launcher enter app
Run cd /shared/postgres_backup && sudo -u postgres pg_dump discourse > backup.db

Undo the postgres template in your container config
Run: ./launcher stop app
Run: sudo mv /var/discourse/shared/standalone/postgres_data /var/discourse/shared/standalone/postgres_data_old
Run: ./launcher rebuild app

Run: ./launcher enter app
Run: cd /shared/postgres_backup
Run: sv stop unicorn
Run: sudo -iu postgres dropdb discourse
Run: sudo -iu postgres createdb discourse
Run: sudo -iu postgres psql discourse < backup.db
Run: exit
Run: ./launcher rebuild app



FAILED
--------------------
Pups::ExecError: /root/upgrade_postgres failed with return #<Process::Status: pid 39 exit 1>
Location of failure: /pups/lib/pups/exec_command.rb:108:in `spawn'
exec failed with the params "/root/upgrade_postgres"
ebd56652d59e7b6edf01ea0d356cf73719b37fc918a87d6f6db292bf18e6aa68
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one

But, because rebuild is failing with the template postgres.9.3.template.yml, I don’t know what to do…

Im running into the same catch 22. The failed bootstrap give the instructions, but instructions can’t be run because the app will not rebuild.

First of all, take a backup and download it!

Then follow this:

cd /var/discourse
mv shared shared_old

Then edit app.yml, removing postgres version from it. After the editing the first lines should look like this:

## this is the all-in-one, standalone Discourse Docker container template
##
## After making changes to this file, you MUST rebuild
## /var/discourse/launcher rebuild app
##
## BE *VERY* CAREFUL WHEN EDITING!
## YAML FILES ARE SUPER SUPER SENSITIVE TO MISTAKES IN WHITESPACE OR ALIGNMENT!
## visit http://www.yamllint.com/ to validate this file as needed
templates:
  - "templates/postgres.template.yml"
  - "templates/redis.template.yml"
  - "templates/web.template.yml"
  - "templates/web.ratelimited.template.yml"
## Uncomment these two lines if you wish to add Lets Encrypt (https)
...

Then ./launcher rebuild app, create an admin, upload a backup and restore. Just make sure to have plenty of space.

This procedure isn’t the fastest, but will work to restore a broken Discourse from any point to a usable and working install.

1 Like

I am migrating my companies instance of discourse from CentOS 6 to CentOS 7 host. I can no longer update Discourse on Cent 6 because there is no new Docker versions. The first thing that I tried is what you mention above. I can import my backup, and successfully restore my data. The catch is I am no longer able to log into the site. Its in some weird state that the new instance thinks that I am logged in from the old instance, causing a log in loop on the new instance. I tried to use the api gem to force log myself out, but you need admin priv’s to do this. I cannot log in, in any form or fashion on the new instance to use the log out API call.

After that, I decided to attempt to simply stand up a new instance, and sync my standalone dir. on top of the new one, considering you offer a psql 9.3 template. Unfortunately I cannot bootstrap with the 9.3 template due to the apt-get errors mentioned above. If I bootstrap with the vanilla postgres template, I get schema errors because my old instance was running on 9.3.

I guess my next option will be to import the old Docker container via registry, and sync over the entire old “var/docker” directory to my new server, then fingers crossed when I attempt to actually update the instance.

Please let me know if there is anything else I can try here.

Try to logout trough rails console:

u = User.find_by_id(YOUR_ID_HERE)
u.update_attribute auth_token: nil
1 Like

Is there a way that I can do this will all users simultaneously? Also, I can’t seem to find userids on our current production instance.

Something like

User.update_all(auth_token: nil)

should work.

1 Like

I just stood up a new instance of Discourse, imported and restored to a backup of my production instance via the web Admin panel. At that point, all of my data shows up and the forum looks as it should, BUT I get stuck in the log in look explained earlier.

I tried the rails command quoted above, with the following return:

[1] pry(main)> User.update_all(auth_token: nil)
=> 1884

After that, I attempt to log in to the forum via web again, and again get the log in loop. It successfully authenticates, but never acts as If i am logged in.

Do you have some sort of outside reverse proxy? Is SSL enabled?

I have SSL enabled on my production instance, but not the current test instance. The web.ssl.template is being used in production, and commented out in test.

I’ve confirmed that the auth_token for at least my user is nil. Still unable to log in.

 => #<User:0x007f16814c8968
  id: 92,
  username: "steven",
  created_at: Tue, 05 Aug 2014 15:10:41 UTC +00:00,
  updated_at: Sat, 18 Apr 2015 07:09:54 UTC +00:00,
  name: "Steven",
  seen_notification_id: 0,
  last_posted_at: Sat, 18 Apr 2015 07:09:54 UTC +00:00,
  email: "shyden@goldenfrog.com",

  active: true,
  username_lower: "steven",
  auth_token: nil,

Can you share a link?

If you had HSTS before this can wreak havoc.

I will work on getting the site opened up. In the mean time, I did find this in my production log, from attempted log in’s:

  Started GET "/session/csrf?_=1481229888046" for ip at 2016-12-08 20:46:39 +0000
 Processing by SessionController#csrf as */*
  Parameters: {"_"=>"1481229888046"}
Completed 200 OK in 1ms (Views: 0.2ms | ActiveRecord: 0.0ms)
Started POST "/session" for ip at 2016-12-08 20:46:39 +0000
Processing by SessionController#create as */*
  Parameters: {"login"=>"chu@phmgmt.com", "password"=>"[FILTERED]"}
Can't verify CSRF token authenticity
  Rendered text template (0.0ms)
Filter chain halted as :verify_authenticity_token rendered or redirected
Completed 403 Forbidden in 1ms (Views: 0.3ms | ActiveRecord: 0.0ms)

I ended up getting Discourse over to my Cent 7 server, by pushing the current container to our registry, and copying the entire /var/docker directory over. I am able to get a functioning version of our Discourse working that way. I am still unable to update discourse due to the following error:

Pups::ExecError: /root/upgrade_postgres failed with return #<Process::Status: pid 40 exit 1>
Location of failure: /pups/lib/pups/exec_command.rb:108:in `spawn'
exec failed with the params "/root/upgrade_postgres"
e68c256f43b7101b5c7003b8db31b80ac83e845999e9f15bec369592d9d04649
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one

This using the update method:

./launcher stop discourse
git pull
./launcher rebuild discourse

This of course leads me back to my loop placing the 9.3 postgres template in my discourse.yml and bootstrap failing again due to apt errors.

I am attempting to disable HSTS now to see if I can get the app to rebuild that way.

You got to follow the manual backup / restore step documented there if the automatic upgrade fails.

2 Likes