Infinite loop during postgres upgrade

Hi! Each time I launch /launcher rebuild app I get the following message (even if I launch it again) :

UPGRADE OF POSTGRES COMPLETE
Old 9.5 database is stored at /shared/postgres_data_old
To complete the upgrade, rebuild again using:
./launcher rebuild app

What information do you need to help me?

PS: The first time I ran it, the command stoped because I needed more space

Temporarily, I used the templates/postgres.9.5.template.yml template to avoid downtime. The app is running again, but I’d like to finish the update.

When I do a cleanup I get the following message : Old PostgreSQL backup data cluster detected taking up 1.2G detected. Would you like to remove it? (Y/n):

Sounds like a question for @tgxworld, but I am guessing that it failed due to lack of space but somehow didn’t catch that the upgrade had failed.

How much space do you have? And how big is your database?

「いいね!」 1

Can you provide me with the full log generated during the rebuild? Thank you!

「いいね!」 3

I’ve been having the same problem for many months now and I can’t upgrade my installation.

It loops and never succeeds. One problem during the rebuild app is this:
mv: cannot move '/shared/postgres_data' to '/shared/postgres_data_old': Device or resource busy

Entering in the Docker the postgres version I have is:

/usr/lib/postgresql/10/bin/postgres --version
postgres (PostgreSQL) 10.14 (Debian 10.14-1.pgdg100+1)

These are the errors I get:

FAILED

--------------------

Pups::ExecError: /root/upgrade_postgres failed with return #<Process::Status: pid 47 exit 1>

Location of failure: /pups/lib/pups/exec_command.rb:112:in `spawn'

exec failed with the params "/root/upgrade_postgres"

0c74c9de4d4315b63c0ef9055631f38c0cf4b3dd0be6500fd83ca0a5b13e0d9d

** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.

./discourse-doctor may help diagnose the problem.

It looks like there’s a problem in the /pups/lib/pups/exec_command.rb script executing spawn.

I can start the app though
./laucher start app

These are the files I have under /shared

total 48
drwxr-xr-x 12 root      root     4096 Jan 12 13:12 .
drwxr-xr-x 57 root      root     4096 Dec 15 09:09 ..
drwxr-xr-x  3 discourse www-data 4096 Aug 20  2019 backups
drwxr-xr-x  4 root      root     4096 Aug 19  2019 log
drwxr-xr-x  2 postgres  postgres 4096 Aug 19  2019 postgres_backup
drwx------ 20 postgres  postgres 4096 Jan 12 13:14 postgres_data
drwx------ 19 postgres  postgres 4096 Jan 12 13:12 postgres_data_new
drwxrwxr-x  5 postgres  postgres 4096 Jan 12 13:14 postgres_run
drwxr-xr-x  2 redis     redis    4096 Jan 12 13:07 redis_data
drwxr-xr-x  4 root      root     4096 Aug 19  2019 state
drwxr-xr-x  4 discourse www-data 4096 Jan 12 13:14 tmp
drwxr-xr-x  4 discourse www-data 4096 Sep  8  2019 uploads

What can I do?

Is your shared directory some sort of network mount?

「いいね!」 1

No, but I’ve managed to do it finally!

The two mv commands in templates/postgres.template.yml are not checked for goodness. The script always says UPGRADE OF POSTGRES COMPLETE which is not true when the move is in error.

I removed the commands, then moved the directories by hand after entering in the docker, and the next rebuild app is currently working as before! I’m so happy :smiley:
Thank you.

「いいね!」 3

PostgreSQL 13から15への最新のアップグレード中にこの問題が発生しました。

私たちのデプロイメントでは、/shared/postgres_dataディレクトリはより高速なNVMeストレージデバイスにマウントされているため、Device or resource busyというエラーメッセージが表示されて移動に失敗しました。

以下のようにpostgresテンプレートをパッチすることでこれを修正しました。

diff --git a/templates/postgres.template.yml b/templates/postgres.template.yml
index c24bfe6..03813c4 100644
--- a/templates/postgres.template.yml
+++ b/templates/postgres.template.yml
@@ -139,8 +139,10 @@ run:
            exit 1
          fi

-         mv /shared/postgres_data /shared/postgres_data_old
-         mv /shared/postgres_data_new /shared/postgres_data
+         mkdir /shared/postgres_data_old
+         mv /shared/postgres_data/* /shared/postgres_data_old
+         mv /shared/postgres_data_new/* /shared/postgres_data
+         rmdir /shared/postgres_data_new

ディレクトリ自体を操作しようとする代わりに、コンテンツを移動させます。

データベースアップグレードの信頼性を向上させるために、この変更を統合することを検討してください。

「いいね!」 1

これに関連しているかもしれません: FIX: improve postgres upgrade reliability by jcharaoui · Pull Request #989 · discourse/discourse_docker · GitHub

「いいね!」 2

PRはマージされました :+1:

「いいね!」 1

このトピックは6日後に自動的に閉じられました。返信はもう許可されていません。