Infinite loop during postgres upgrade

[CamilleRoux]

Hi! Each time I launch /launcher rebuild app I get the following message (even if I launch it again) :

UPGRADE OF POSTGRES COMPLETE
Old 9.5 database is stored at /shared/postgres_data_old
To complete the upgrade, rebuild again using:
./launcher rebuild app

What information do you need to help me?

PS: The first time I ran it, the command stoped because I needed more space

[CamilleRoux]

Temporarily, I used the templates/postgres.9.5.template.yml template to avoid downtime. The app is running again, but I’d like to finish the update.

When I do a cleanup I get the following message : Old PostgreSQL backup data cluster detected taking up 1.2G detected. Would you like to remove it? (Y/n):

[pfaffman (Jay Pfaffman)]

Sounds like a question for @tgxworld, but I am guessing that it failed due to lack of space but somehow didn’t catch that the upgrade had failed.

How much space do you have? And how big is your database?

[tgxworld (Alan Tan)]

Can you provide me with the full log generated during the rebuild? Thank you!

[vulkanino (Marco)]

I’ve been having the same problem for many months now and I can’t upgrade my installation.

It loops and never succeeds. One problem during the rebuild app is this:
mv: cannot move '/shared/postgres_data' to '/shared/postgres_data_old': Device or resource busy

Entering in the Docker the postgres version I have is:

/usr/lib/postgresql/10/bin/postgres --version
postgres (PostgreSQL) 10.14 (Debian 10.14-1.pgdg100+1)

These are the errors I get:

FAILED

--------------------

Pups::ExecError: /root/upgrade_postgres failed with return #<Process::Status: pid 47 exit 1>

Location of failure: /pups/lib/pups/exec_command.rb:112:in `spawn'

exec failed with the params "/root/upgrade_postgres"

0c74c9de4d4315b63c0ef9055631f38c0cf4b3dd0be6500fd83ca0a5b13e0d9d

** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.

./discourse-doctor may help diagnose the problem.

It looks like there’s a problem in the /pups/lib/pups/exec_command.rb script executing spawn.

I can start the app though
./laucher start app

These are the files I have under /shared

total 48
drwxr-xr-x 12 root      root     4096 Jan 12 13:12 .
drwxr-xr-x 57 root      root     4096 Dec 15 09:09 ..
drwxr-xr-x  3 discourse www-data 4096 Aug 20  2019 backups
drwxr-xr-x  4 root      root     4096 Aug 19  2019 log
drwxr-xr-x  2 postgres  postgres 4096 Aug 19  2019 postgres_backup
drwx------ 20 postgres  postgres 4096 Jan 12 13:14 postgres_data
drwx------ 19 postgres  postgres 4096 Jan 12 13:12 postgres_data_new
drwxrwxr-x  5 postgres  postgres 4096 Jan 12 13:14 postgres_run
drwxr-xr-x  2 redis     redis    4096 Jan 12 13:07 redis_data
drwxr-xr-x  4 root      root     4096 Aug 19  2019 state
drwxr-xr-x  4 discourse www-data 4096 Jan 12 13:14 tmp
drwxr-xr-x  4 discourse www-data 4096 Sep  8  2019 uploads

What can I do?

[Falco]

Is your shared directory some sort of network mount?

[vulkanino (Marco)]

No, but I’ve managed to do it finally!

The two mv commands in templates/postgres.template.yml are not checked for goodness. The script always says UPGRADE OF POSTGRES COMPLETE which is not true when the move is in error.

I removed the commands, then moved the directories by hand after entering in the docker, and the next rebuild app is currently working as before! I’m so happy
Thank you.

[lavamind]

I hit this problem during the latest upgrade from PotsgreSQL 13 to 15.

In our deployment, our/shared/postgres_data directory is mounted on a faster NVMe storage device, so moving it failed with a Device or resource busy error message.

We fixed this by patching the postgres template as such:

diff --git a/templates/postgres.template.yml b/templates/postgres.template.yml
index c24bfe6..03813c4 100644
--- a/templates/postgres.template.yml
+++ b/templates/postgres.template.yml
@@ -139,8 +139,10 @@ run:
            exit 1
          fi
 
-         mv /shared/postgres_data /shared/postgres_data_old
-         mv /shared/postgres_data_new /shared/postgres_data
+         mkdir /shared/postgres_data_old
+         mv /shared/postgres_data/* /shared/postgres_data_old
+         mv /shared/postgres_data_new/* /shared/postgres_data
+         rmdir /shared/postgres_data_new

Instead of attempting to manipulate the directory itself, just move the contents around.

Please consider integrating this change to improve the reliability of database upgrades.

[pfaffman (Jay Pfaffman)]

I think this might be related: FIX: improve postgres upgrade reliability by jcharaoui · Pull Request #989 · discourse/discourse_docker · GitHub

[tgxworld (Alan Tan)]

@pfaffman

FIX: improve postgres upgrade reliability by jcharaoui · Pull Request #989 · discourse/discourse_docker · GitHub

The PR has been merged