Infinite loop during postgres upgrade

Hi! Each time I launch /launcher rebuild app I get the following message (even if I launch it again) :

UPGRADE OF POSTGRES COMPLETE
Old 9.5 database is stored at /shared/postgres_data_old
To complete the upgrade, rebuild again using:
./launcher rebuild app

What information do you need to help me?

PS: The first time I ran it, the command stoped because I needed more space

Temporarily, I used the templates/postgres.9.5.template.yml template to avoid downtime. The app is running again, but I’d like to finish the update.

When I do a cleanup I get the following message : Old PostgreSQL backup data cluster detected taking up 1.2G detected. Would you like to remove it? (Y/n):

Sounds like a question for @tgxworld, but I am guessing that it failed due to lack of space but somehow didn’t catch that the upgrade had failed.

How much space do you have? And how big is your database?

לייק 1

Can you provide me with the full log generated during the rebuild? Thank you!

3 לייקים

I’ve been having the same problem for many months now and I can’t upgrade my installation.

It loops and never succeeds. One problem during the rebuild app is this:
mv: cannot move '/shared/postgres_data' to '/shared/postgres_data_old': Device or resource busy

Entering in the Docker the postgres version I have is:

/usr/lib/postgresql/10/bin/postgres --version
postgres (PostgreSQL) 10.14 (Debian 10.14-1.pgdg100+1)

These are the errors I get:

FAILED

--------------------

Pups::ExecError: /root/upgrade_postgres failed with return #<Process::Status: pid 47 exit 1>

Location of failure: /pups/lib/pups/exec_command.rb:112:in `spawn'

exec failed with the params "/root/upgrade_postgres"

0c74c9de4d4315b63c0ef9055631f38c0cf4b3dd0be6500fd83ca0a5b13e0d9d

** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.

./discourse-doctor may help diagnose the problem.

It looks like there’s a problem in the /pups/lib/pups/exec_command.rb script executing spawn.

I can start the app though
./laucher start app

These are the files I have under /shared

total 48
drwxr-xr-x 12 root      root     4096 Jan 12 13:12 .
drwxr-xr-x 57 root      root     4096 Dec 15 09:09 ..
drwxr-xr-x  3 discourse www-data 4096 Aug 20  2019 backups
drwxr-xr-x  4 root      root     4096 Aug 19  2019 log
drwxr-xr-x  2 postgres  postgres 4096 Aug 19  2019 postgres_backup
drwx------ 20 postgres  postgres 4096 Jan 12 13:14 postgres_data
drwx------ 19 postgres  postgres 4096 Jan 12 13:12 postgres_data_new
drwxrwxr-x  5 postgres  postgres 4096 Jan 12 13:14 postgres_run
drwxr-xr-x  2 redis     redis    4096 Jan 12 13:07 redis_data
drwxr-xr-x  4 root      root     4096 Aug 19  2019 state
drwxr-xr-x  4 discourse www-data 4096 Jan 12 13:14 tmp
drwxr-xr-x  4 discourse www-data 4096 Sep  8  2019 uploads

What can I do?

Is your shared directory some sort of network mount?

לייק 1

No, but I’ve managed to do it finally!

The two mv commands in templates/postgres.template.yml are not checked for goodness. The script always says UPGRADE OF POSTGRES COMPLETE which is not true when the move is in error.

I removed the commands, then moved the directories by hand after entering in the docker, and the next rebuild app is currently working as before! I’m so happy :smiley:
Thank you.

3 לייקים

נתקלתי בבעיה זו במהלך השדרוג האחרון מ-PotsgreSQL 13 ל-15.

בפריסה שלנו, ספריית /shared/postgres_data שלנו מותקנת על התקן אחסון NVMe מהיר יותר, ולכן המעבר נכשל עם הודעת השגיאה Device or resource busy.

תיקנו זאת על ידי תיקון תבנית הפוסטגרס באופן הבא:

diff --git a/templates/postgres.template.yml b/templates/postgres.template.yml
index c24bfe6..03813c4 100644
--- a/templates/postgres.template.yml
+++ b/templates/postgres.template.yml
@@ -139,8 +139,10 @@ run:
           fi

-         mv /shared/postgres_data /shared/postgres_data_old
-         mv /shared/postgres_data_new /shared/postgres_data
+         mkdir /shared/postgres_data_old
+         mv /shared/postgres_data/* /shared/postgres_data_old
+         mv /shared/postgres_data_new/* /shared/postgres_data
+         rmdir /shared/postgres_data_new

במקום לנסות לתפעל את הספרייה עצמה, פשוט העברנו את התוכן.
אנא שקלו לשלב שינוי זה כדי לשפר את האמינות של שדרוגי מסד נתונים.

לייק 1

I think this might be related: FIX: improve postgres upgrade reliability by jcharaoui · Pull Request #989 · discourse/discourse_docker · GitHub

2 לייקים

The PR has been merged :+1:

לייק 1

נושא זה נסגר אוטומטית לאחר 6 ימים. לא ניתן להוסיף תגובות חדשות.