Infinite loop during postgres upgrade

Installation
1

Hi! Each time I launch /launcher rebuild app I get the following message (even if I launch it again) :

UPGRADE OF POSTGRES COMPLETE
Old 9.5 database is stored at /shared/postgres_data_old
To complete the upgrade, rebuild again using:
./launcher rebuild app

What information do you need to help me?

PS: The first time I ran it, the command stoped because I needed more space

2

Temporarily, I used the templates/postgres.9.5.template.yml template to avoid downtime. The app is running again, but I’d like to finish the update.

When I do a cleanup I get the following message : Old PostgreSQL backup data cluster detected taking up 1.2G detected. Would you like to remove it? (Y/n):

3

Sounds like a question for @tgxworld, but I am guessing that it failed due to lack of space but somehow didn’t catch that the upgrade had failed.

How much space do you have? And how big is your database?

1 Like
4

Can you provide me with the full log generated during the rebuild? Thank you!

3 Likes
5

I’ve been having the same problem for many months now and I can’t upgrade my installation.

It loops and never succeeds. One problem during the rebuild app is this:
mv: cannot move '/shared/postgres_data' to '/shared/postgres_data_old': Device or resource busy

Entering in the Docker the postgres version I have is:

/usr/lib/postgresql/10/bin/postgres --version
postgres (PostgreSQL) 10.14 (Debian 10.14-1.pgdg100+1)

These are the errors I get:

FAILED

--------------------

Pups::ExecError: /root/upgrade_postgres failed with return #<Process::Status: pid 47 exit 1>

Location of failure: /pups/lib/pups/exec_command.rb:112:in `spawn'

exec failed with the params "/root/upgrade_postgres"

0c74c9de4d4315b63c0ef9055631f38c0cf4b3dd0be6500fd83ca0a5b13e0d9d

** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.

./discourse-doctor may help diagnose the problem.

It looks like there’s a problem in the /pups/lib/pups/exec_command.rb script executing spawn.

I can start the app though
./laucher start app

These are the files I have under /shared

total 48
drwxr-xr-x 12 root      root     4096 Jan 12 13:12 .
drwxr-xr-x 57 root      root     4096 Dec 15 09:09 ..
drwxr-xr-x  3 discourse www-data 4096 Aug 20  2019 backups
drwxr-xr-x  4 root      root     4096 Aug 19  2019 log
drwxr-xr-x  2 postgres  postgres 4096 Aug 19  2019 postgres_backup
drwx------ 20 postgres  postgres 4096 Jan 12 13:14 postgres_data
drwx------ 19 postgres  postgres 4096 Jan 12 13:12 postgres_data_new
drwxrwxr-x  5 postgres  postgres 4096 Jan 12 13:14 postgres_run
drwxr-xr-x  2 redis     redis    4096 Jan 12 13:07 redis_data
drwxr-xr-x  4 root      root     4096 Aug 19  2019 state
drwxr-xr-x  4 discourse www-data 4096 Jan 12 13:14 tmp
drwxr-xr-x  4 discourse www-data 4096 Sep  8  2019 uploads

What can I do?

6

Is your shared directory some sort of network mount?

1 Like
7

No, but I’ve managed to do it finally!

The two mv commands in templates/postgres.template.yml are not checked for goodness. The script always says UPGRADE OF POSTGRES COMPLETE which is not true when the move is in error.

I removed the commands, then moved the directories by hand after entering in the docker, and the next rebuild app is currently working as before! I’m so happy :smiley:
Thank you.

3 Likes