Major woes upgrading from 1.6.0 to 1.8.0 -- broken site

This is the first major problem I’ve ran into while maintaining two Discourse sites. The (currently) broken forum had two particularities but installation was done by the book. One of these special settings is Spanish locale, configured like this:

params:                                                                                         
  db_default_text_search_config: "pg_catalog.spanish"                                                                                                                                                                                                  
                                                                                                
env:                                                                                            
  LANG: es_ES.UTF-8                                                                             
  DISCOURSE_DEFAULT_LOCALE: es                                                                  

This was running smoothly for a year or so, until I couldn’t do an automatic upgrade. Fine, I thought, let’s go manual. That’s when ./launcher rebuild app started complaining about missing locales:

2017-03-28 12:08:55 UTC [48-2] FATAL:  database files are incompatible with server
2017-03-28 12:08:55 UTC [48-3] DETAIL:  The data directory was initialized by PostgreSQL version 9.3, which is not compatible with this version 9.5.5.

I did my homework and studied several reports of similar issues in this community. There’s too many posts to link to but, suffice to say, no suggested solution works. It would seem as if something else is going on here. First, I made the following change in my app.yml:

env:                                                                                            
  LANG: en_US.UTF-8                                                                             
  DISCOURSE_DEFAULT_LOCALE: es  

Which was a bit of progress. Here’s the relevant error:

update-alternatives: warning: forcing reinstallation of alternative /usr/share/postgresql/9.5/man/man1/psql.1.gz because link group psql.1.gz is broken
update-alternatives: error: error creating symbolic link '/usr/share/man/man1/psql.1.gz.dpkg-tmp': No such file or directory
dpkg: error processing package postgresql-client-9.3 (--configure):
 subprocess installed post-installation script returned error exit status 2
dpkg: dependency problems prevent configuration of postgresql-9.3:
 postgresql-9.3 depends on postgresql-client-9.3; however:
  Package postgresql-client-9.3 is not configured yet.

And the final recommendation to export a backup and start anew:

UPGRADE OF POSTGRES FAILED

You are going to need to export your data and import into a clean instance:

In containers/app.yml: Change "templates/postgres.template.yml" TO "templates/postgres.9.3.template.yml"

Fine, I can live with that. So I made the suggested change:

templates:
  - "templates/postgres.9.3.template.yml"
  - "templates/redis.template.yml"
  - "templates/web.template.yml"
  - "templates/sshd.template.yml"
  - "templates/web.ratelimited.template.yml"

And now I’m effectively stumped:

Removing postgresql-contrib-9.5 (9.5.5-1.pgdg16.04+1) ...
update-alternatives: warning: forcing reinstallation of alternative /usr/share/postgresql/9.5/man/man1/postmaster.1.gz because link group postmaster.1.gz is broken
update-alternatives: error: error creating symbolic link '/usr/share/man/man1/postmaster.1.gz.dpkg-tmp': No such file or directory
dpkg: error processing package postgresql-contrib-9.5 (--remove):
 subprocess installed pre-removal script returned error exit status 2
dpkg: postgresql-9.5: dependency problems, but removing anyway as you requested:
 postgresql-contrib-9.5 depends on postgresql-9.5 (= 9.5.5-1.pgdg16.04+1).

Removing postgresql-9.5 (9.5.5-1.pgdg16.04+1) ...
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of stop.
Removing postgresql-client-9.5 (9.5.5-1.pgdg16.04+1) ...
Processing triggers for postgresql-common (178.pgdg16.04+1) ...
Building PostgreSQL dictionaries from installed myspell/hunspell packages...
Removing obsolete dictionary files:
Errors were encountered while processing:
 postgresql-contrib-9.5

This is as far I can go and I wanted to report this before I do anything that irreversibly breaks the database. I can’t export the backup because apparently Docker can’t get PostgreSQL running in the container (to my understanding).

Any help would be greatly appreciated. The second particularity in this install is that’s it’s multisite, although I’m running a simple HTML portal along with Discourse. I don’t believe it has anything to do with these errors, which seem to be due to unsupported locales in PSQL 9.5.

Thoughts? Ideas? Prayers?

:slight_smile: lots of them

What you are going to need to do is go to make a copy of the pg 9.3 template and “muck” with it to make it able to uninstall pg 9.5 properly.

The way you would do this is:

cp templates/postgres.9.3.template.yml containers/my-hacked-template.yml

Then add

templates:
   - containers/my-hacked-template.yml

Into your container definition and remove 9.3

Then you need to start playing with

https://github.com/discourse/discourse_docker/blob/master/templates/postgres.9.3.template.yml#L23

To see what you need to do to get that apt-get remove line to work. Once you get it working, ping us back here to let us know what you did.

2 Likes

I’m afraid I’m not having much luck. How safe is it to purge PSQL, ie: apt-get purge postgresql?

I can’t get to remove 9.5 because it always triggers this error, even attempting to remove packages separately – the script always breaks here:

update-alternatives: warning: forcing reinstallation of alternative /usr/share/postgresql/9.5/man/man1/postmaster.1.gz because link group postmaster.1.gz is broken
update-alternatives: error: error creating symbolic link '/usr/share/man/man1/postmaster.1.gz.dpkg-tmp': No such file or directory

I can reproduce it myself doing this:

exec: ln -s /usr/share/postgresql/9.5/man/man1/postmaster.1.gz /usr/share/man/man1/postmaster.1.gz.dpkg-tmp

What the heck? The file exists, as confirmed by touch /usr/share/postgresql/9.5/man/man1/postmaster.1.gz. Am I even on the right track here…?

One other thing. I tried this:

- exec: update-alternatives --remove postmaster.1.gz /usr/share/postgresql/9.5/man/man1/postmaster.1.gz

Which actually did something and improved the output:

Removing postgresql-contrib-9.5 (9.5.5-1.pgdg16.04+1) ...
update-alternatives: using /usr/share/postgresql/9.5/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode
update-alternatives: error: error creating symbolic link '/usr/share/man/man1/postmaster.1.gz.dpkg-tmp': No such file or directory

But still fails when creating that darned symbolic link…

@Falco can you test out the 9.3 template to see why its not running properly on current image?

3 Likes

Thanks for the assistance, folks.

I’m wondering, I still have access to the old container. I can do ./launcher start & enter app. However --and this is what puzzles me-- the postgres command isn’t installed. The data seems to be intact in /shared/postgres_data, though.

Do I have a way to safely recover the database backup? I’m kicking myself because Amazon backups weren’t enabled. My priority right now is to backup everything – I wouldn’t mind reinstalling from scratch.

Thanks again!

Hey @AgustinCordes,

You don’t have any recent backups on /var/discourse/shared/standalone/backups/default/?

2 Likes

OMG YES I DO

OK then, let’s do this the easy way and start with a fresh install. I’m going to double check what’s going on with Spanish locales to avoid an incompatible database in the future.

1 Like

Exactly that, I have found it way easier to do this way when the automatic pg update fails.

I will take a look at our launcher script and see why it isn’t working, but figured you want the fastest way to get your forum back.

3 Likes

OK, I’ll keep the old droplet running for a few days if you want to do some tests :slight_smile: