UPGRADE OF POSTGRES FAILED - I've tried everything

jameshahnii · August 24, 2020, 2:31pm

Hello, site is suddenly stuck at Welcome to nginx! so debugging and running into these errors.

Success. You can now start the database server using:

    pg_ctlcluster 10 main start

Warning: The selected stats_temp_directory /var/run/postgresql/10-main.pg_stat_tmp
is not writable for the cluster owner. Not adding this setting in
postgresql.conf.
Ver Cluster Port Status Owner    Data directory              Log file
10  main    5433 down   postgres /var/lib/postgresql/10/main /var/log/postgresql/postgresql-10-main.log
update-alternatives: warning: forcing reinstallation of alternative /usr/share/postgresql/12/man/man1/postmaster.1.gz because link group postmaster.1.gz is broken
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Processing triggers for postgresql-common (215.pgdg100+1) ...
Building PostgreSQL dictionaries from installed myspell/hunspell packages...
Removing obsolete dictionary files:
Stopping PostgreSQL 10 database server: main.
Stopping PostgreSQL 12 database server: main.
Performing Consistency Checks
-----------------------------
Checking cluster versions                                   ok

The source cluster was not shut down cleanly.
Failure, exiting
-------------------------------------------------------------------------------------
UPGRADE OF POSTGRES FAILED

Please visit https://meta.discourse.org/t/postgresql-12-update/151236 for support

You can run ./launcher start app to restart your app in the meanwhile




FAILED
--------------------
Pups::ExecError: /root/upgrade_postgres failed with return #<Process::Status: pid 45 exit 1>
Location of failure: /pups/lib/pups/exec_command.rb:112:in `spawn'
exec failed with the params "/root/upgrade_postgres"
bfe8265213ad992fa3245d252d192977f05d902d7213b361f53d3bc2b0d16b3a
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.
root@bitkcor:/var/discourse# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            1.9G     0  1.9G   0% /dev
tmpfs           395M  648K  394M   1% /run
/dev/vda1        78G   23G   55G  30% /
tmpfs           2.0G     0  2.0G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/vda15      105M  3.4M  101M   4% /boot/efi
tmpfs           395M     0  395M   0% /run/user/0
root@bitkcor:/var/discourse# ls
bin   containers        discourse-setup  launcher  README.md  scripts  templates
cids  discourse-doctor  image            LICENSE   samples    shared
root@bitkcor:/var/discourse# cd shared
root@bitkcor:/var/discourse/shared# chown -R postgres postgres_data
chown: invalid user: ‘postgres’
root@bitkcor:/var/discourse/shared# ./launcher start app
-bash: ./launcher: No such file or directory
root@bitkcor:/var/discourse/shared# cd ..
root@bitkcor:/var/discourse# ./launcher start app

starting up existing container
+ /usr/bin/docker start app
Error response from daemon: driver failed programming external connectivity on endpoint app (10f3d3b7938496e743c73affc9ddb2d821e04419985bf8e3ebde2ec9ec704a0b): Error starting userland proxy: listen tcp 0.0.0.0:80: bind: address already in use
Error: failed to start containers: app
root@bitkcor:/var/discourse# cd containers
root@bitkcor:/var/discourse/containers# ls
app.yml  app.yml.2018-08-29-034711.bak
root@bitkcor:/var/discourse/containers# cd ..
root@bitkcor:/var/discourse# ./launcher stop app
+ /usr/bin/docker stop -t 10 app
app
root@bitkcor:/var/discourse# ./launcher stop app
+ /usr/bin/docker stop -t 10 app
app
root@bitkcor:/var/discourse# git pull
Already up to date.
root@bitkcor:/var/discourse# ./launcher start app

starting up existing container
+ /usr/bin/docker start app
Error response from daemon: driver failed programming external connectivity on endpoint app (f149fe57c125ae0c276c339a1367c83262f866227ab9317ff06bf026e8776f65): Error starting userland proxy: listen tcp 0.0.0.0:80: bind: address already in use
Error: failed to start containers: app

I’ve visited https://meta.discourse.org/t/postgresql-12-update/151236 for support and tried @Falco’s suggestions here… and any other suggestions across meta. Can’t get anything to work.

As far as I can tell I have more than enough space for this.

Can anyone tell me what I’m missing?

Falco · August 24, 2020, 2:42pm

There is a FAQ entry just for this in PostgreSQL 12 update.

jameshahnii · August 24, 2020, 2:46pm

Thanks for the quick response! Yes, I tried that as well.

root@bitkcor:/var/discourse# ./launcher start app

starting up existing container
+ /usr/bin/docker start app
Error response from daemon: driver failed programming external connectivity on endpoint app (49f1fdf896618efc824e50f782c1fba91bf81320e49ccadb5e5e80b342552e3e): Error starting userland proxy: listen tcp 0.0.0.0:80: bind: address already in use
Error: failed to start containers: app
root@bitkcor:/var/discourse# ./launcher stop app
+ /usr/bin/docker stop -t 10 app
app
root@bitkcor:/var/discourse# tail -f shared/data/log/var-log/postgres/current
tail: cannot open 'shared/data/log/var-log/postgres/current' for reading: No such file or directory
tail: no files remaining

Falco · August 24, 2020, 2:54pm

Somehow, someone installed nginx on your host. You should remove any extraneous web servers from your VPS and then follow the instructions.

jameshahnii · August 24, 2020, 3:05pm

nginx is installed because this has been a working forum for a couple years now. Was late with Digital Ocean and the Droplet got turned off. After turning it back on, I’m getting a 521 error going to the address. When I go to the IP Address I’m back to the Welcome screen.

The Cloudflare configuration has not changed.

And the nameservers are correct in NameCheap.

I haven’t changed anything in several months so I’m at a loss on where to look.

Falco · August 24, 2020, 3:23pm

So you had nginx installed but not running. Turning the droplet off and turning it back on restarted the service and now it blocks the web port.

You should really uninstall it so this doesn’t happen every time the server restarts…

jameshahnii · August 24, 2020, 3:32pm

Has something changed? When I built this the nginx welcome screen was part of the install process. I was under the impression nginx was necessary to run Discourse. How do I uninstall it?

itsbhanusharma · August 24, 2020, 3:45pm

Please change the path to
tail -f shared/standalone/log/var-log/postgres/current

The path mentioned in guide references the two container install data container whereas it seems your install is the single container one.

jameshahnii · August 24, 2020, 3:45pm

Done. Gives me this.

root@bitkcor:/var/discourse# tail -f shared/standalone/log/var-log/postgres/current
2020-07-19 03:33:56.864 UTC [19933] discourse@discourse LOG:  duration: 279.207 ms  statement: COPY public.scheduler_stats (id, name, hostname, pid, duration_ms, live_slots_start, live_slots_finish, started_at, success, error) TO stdout;
2020-07-19 03:34:09.436 UTC [19933] discourse@discourse LOG:  duration: 12555.420 ms  statement: COPY public.stylesheet_cache (id, target, digest, content, created_at, updated_at, theme_id, source_map) TO stdout;
2020-07-19 03:34:10.211 UTC [19933] discourse@discourse LOG:  duration: 727.297 ms  statement: COPY public.unsubscribe_keys (key, user_id, created_at, updated_at, unsubscribe_key_type, topic_id, post_id) TO stdout;
2020-07-21 01:56:22.105 UTC [6388] discourse@discourse LOG:  duration: 167.853 ms  execute <unnamed>: INSERT INTO "unsubscribe_keys" ("key", "user_id", "created_at", "updated_at", "unsubscribe_key_type") VALUES ('352fc5679876a1a700dfe7b45f8fa67612592421a3659e08ec5c2ccbf8f0e2d2', 2, '2020-07-21 01:56:21.932109', '2020-07-21 01:56:21.932109', 'digest') RETURNING "key"
2020-07-26 03:34:50.570 UTC [27570] discourse@discourse LOG:  duration: 147.456 ms  statement: COPY public.post_revisions (id, user_id, post_id, modifications, number, created_at, updated_at, hidden) TO stdout;
2020-07-26 03:34:50.925 UTC [27570] discourse@discourse LOG:  duration: 349.648 ms  statement: COPY public.post_search_data (post_id, search_data, raw_data, locale, version) TO stdout;
2020-07-26 03:34:51.236 UTC [27570] discourse@discourse LOG:  duration: 292.799 ms  statement: COPY public.posts (id, user_id, topic_id, post_number, raw, cooked, created_at, updated_at, reply_to_post_number, reply_count, quote_count, deleted_at, off_topic_count, like_count, incoming_link_count, bookmark_count, avg_time, score, reads, post_type, sort_order, last_editor_id, hidden, hidden_reason_id, notify_moderators_count, spam_count, illegal_count, inappropriate_count, last_version_at, user_deleted, reply_to_user_id, percent_rank, notify_user_count, like_score, deleted_by_id, edit_reason, word_count, version, cook_method, wiki, baked_at, baked_version, hidden_at, self_edits, reply_quoted, via_email, raw_email, public_version, action_code, image_url, locked_by_id) TO stdout;
2020-07-26 03:34:51.547 UTC [27570] discourse@discourse LOG:  duration: 296.400 ms  statement: COPY public.scheduler_stats (id, name, hostname, pid, duration_ms, live_slots_start, live_slots_finish, started_at, success, error) TO stdout;
2020-07-26 03:35:04.123 UTC [27570] discourse@discourse LOG:  duration: 12549.364 ms  statement: COPY public.stylesheet_cache (id, target, digest, content, created_at, updated_at, theme_id, source_map) TO stdout;
2020-07-26 03:35:04.760 UTC [27570] discourse@discourse LOG:  duration: 588.788 ms  statement: COPY public.unsubscribe_keys (key, user_id, created_at, updated_at, unsubscribe_key_type, topic_id, post_id) TO stdout;

itsbhanusharma · August 24, 2020, 3:48pm

Looks like postgres is running. Did you shut down Discourse container before checking logs?

./launcher stop app

jameshahnii · August 24, 2020, 3:49pm

I can’t seem to stop it.

root@bitkcor:/var/discourse# ./launcher stop app
+ /usr/bin/docker stop -t 10 app
app
root@bitkcor:/var/discourse# tail -f shared/standalone/log/var-log/postgres/current
2020-07-19 03:33:56.864 UTC [19933] discourse@discourse LOG:  duration: 279.207 ms  statement: COPY public.scheduler_stats (id, name, hostname, pid, duration_ms, live_slots_start, live_slots_finish, started_at, success, error) TO stdout;
2020-07-19 03:34:09.436 UTC [19933] discourse@discourse LOG:  duration: 12555.420 ms  statement: COPY public.stylesheet_cache (id, target, digest, content, created_at, updated_at, theme_id, source_map) TO stdout;
2020-07-19 03:34:10.211 UTC [19933] discourse@discourse LOG:  duration: 727.297 ms  statement: COPY public.unsubscribe_keys (key, user_id, created_at, updated_at, unsubscribe_key_type, topic_id, post_id) TO stdout;
2020-07-21 01:56:22.105 UTC [6388] discourse@discourse LOG:  duration: 167.853 ms  execute <unnamed>: INSERT INTO "unsubscribe_keys" ("key", "user_id", "created_at", "updated_at", "unsubscribe_key_type") VALUES ('352fc5679876a1a700dfe7b45f8fa67612592421a3659e08ec5c2ccbf8f0e2d2', 2, '2020-07-21 01:56:21.932109', '2020-07-21 01:56:21.932109', 'digest') RETURNING "key"
2020-07-26 03:34:50.570 UTC [27570] discourse@discourse LOG:  duration: 147.456 ms  statement: COPY public.post_revisions (id, user_id, post_id, modifications, number, created_at, updated_at, hidden) TO stdout;
2020-07-26 03:34:50.925 UTC [27570] discourse@discourse LOG:  duration: 349.648 ms  statement: COPY public.post_search_data (post_id, search_data, raw_data, locale, version) TO stdout;
2020-07-26 03:34:51.236 UTC [27570] discourse@discourse LOG:  duration: 292.799 ms  statement: COPY public.posts (id, user_id, topic_id, post_number, raw, cooked, created_at, updated_at, reply_to_post_number, reply_count, quote_count, deleted_at, off_topic_count, like_count, incoming_link_count, bookmark_count, avg_time, score, reads, post_type, sort_order, last_editor_id, hidden, hidden_reason_id, notify_moderators_count, spam_count, illegal_count, inappropriate_count, last_version_at, user_deleted, reply_to_user_id, percent_rank, notify_user_count, like_score, deleted_by_id, edit_reason, word_count, version, cook_method, wiki, baked_at, baked_version, hidden_at, self_edits, reply_quoted, via_email, raw_email, public_version, action_code, image_url, locked_by_id) TO stdout;
2020-07-26 03:34:51.547 UTC [27570] discourse@discourse LOG:  duration: 296.400 ms  statement: COPY public.scheduler_stats (id, name, hostname, pid, duration_ms, live_slots_start, live_slots_finish, started_at, success, error) TO stdout;
2020-07-26 03:35:04.123 UTC [27570] discourse@discourse LOG:  duration: 12549.364 ms  statement: COPY public.stylesheet_cache (id, target, digest, content, created_at, updated_at, theme_id, source_map) TO stdout;
2020-07-26 03:35:04.760 UTC [27570] discourse@discourse LOG:  duration: 588.788 ms  statement: COPY public.unsubscribe_keys (key, user_id, created_at, updated_at, unsubscribe_key_type, topic_id, post_id) TO stdout;c

Falco · August 25, 2020, 1:51pm

The nginx welcome screen was never part of the process. You are misremembering.

Stephen · August 25, 2020, 1:54pm

Nginx is also installed within the docker container, it has never been required outside.

You have two instances of nginx listening on the same port. If the only thing on this server is Discourse, then the duplicate nginx outside the container can be removed quite safely.

jameshahnii · August 25, 2020, 2:06pm

Okay, sorry about that.

jameshahnii · August 25, 2020, 2:06pm

Do you know where it might be located?

Stephen · August 25, 2020, 2:07pm

Assuming the server is Ubuntu run this outside the container:

sudo apt-get remove nginx nginx-common

jameshahnii · August 25, 2020, 2:13pm

Yeesh. Now the site can’t be reached. Restoring a backup and hoping for the best.

Stephen · August 25, 2020, 2:15pm

That’s actually good news, it means the nginx instance outside the container is no longer hijacking the port.

Once you uninstall it you should reboot the server so that everything can start properly.