yeah, so the warnings are new (and should be fixed) but those only affect the commands that are copying over the initial files to the system, and not related to updates. The best suspicion we have is the database was initialized before the LANG was applied, and only causes issues with database updates.
One thing to try is set lang temporarily to en_US.UTF-8 for upgrading and swap back after. (obviously make sure to take backups)
failed for me, something with locale too
$ cat postgres_data_new/pg_upgrade_output.d/20250131T032317.601/log/pg_upgrade_server.log
-----------------------------------------------------------------
pg_upgrade run on Fri Jan 31 03:23:17 2025
-----------------------------------------------------------------
command: "/usr/lib/postgresql/13/bin/pg_ctl" -w -l "/var/lib/postgresql/15/data/pg_upgrade_output.d/20250131T032317.601/log/pg_upgrade_server.log" -D "/var/lib/postgresql/13/data" -o "-p 50432 -b -c listen_addresses='' -c unix_socket_permissions=0700 -c unix_socket_directories='/var/lib/postgresql'" start >> "/var/lib/postgresql/15/data/pg_upgrade_output.d/20250131T032317.601/log/pg_upgrade_server.log" 2>&1
waiting for server to start....2025-01-31 03:23:17.761 UTC [268] LOG: invalid value for parameter "lc_messages": "ru_RU.UTF-8"
2025-01-31 03:23:17.761 UTC [268] LOG: invalid value for parameter "lc_monetary": "ru_RU.UTF-8"
2025-01-31 03:23:17.761 UTC [268] LOG: invalid value for parameter "lc_numeric": "ru_RU.UTF-8"
2025-01-31 03:23:17.761 UTC [268] LOG: invalid value for parameter "lc_time": "ru_RU.UTF-8"
2025-01-31 03:23:17.761 UTC [268] FATAL: configuration file "/var/lib/postgresql/13/data/postgresql.conf" contains errors
stopped waiting
pg_ctl: could not start server
Examine the log output.
Can you try to set the following in your app.yml file and rebuild?
env:
LANG: 'ru_RU.UTF-8'
...
The error you are seeing is likely related to Postgres error upgrading from 3.3 to 3.4 caused by locale issues.
I have it in yml, sorry if it’s not clear, I’m trying step with tianon/postgres-upgrade:13-to-15 container from manual update and commented on that script.
I fixed that error by enabling ru_RU.UTF-8 locale in the container, but now there is another one:
Performing Consistency Checks
-----------------------------
Checking cluster versions ok
Checking database user is the install user ok
Checking database connection settings ok
Checking for prepared transactions ok
Checking for system-defined composite types in user tables ok
Checking for reg* data types in user tables ok
Checking for contrib/isn with bigint-passing mismatch ok
Checking for user-defined encoding conversions ok
Checking for user-defined postfix operators ok
Checking for incompatible polymorphic functions ok
Creating dump of global objects ok
Creating dump of database schemas ok
lc_collate values for database "template1" do not match: old "ru_RU.UTF-8", new "en_US.utf8"
Failure, exiting
Could you confirm if you also recreated the postgres_data_new directory after updating the locale in the tianon container? It was likely initiated with the incorrect locale in the previous runs, which could explain why you saw the subsequent error stating new "en_US.utf8".
If you hadn’t done this already, you may also need to set the appropriate locale settings in the tianon container before running docker-upgrade (which also runs initdb for the new database version.) The easiest way would be copying over the locale-specific environment variables from your app container.
Yes, that was part of the problem! Thanks!
In the end i needed to do two modifications, first is to setup initdb locale with POSTGRES_INITDB_ARGS="--locale=ru_RU.UTF-8" (based on this comment), and second is to enable locale and run locale-gen, so script looked like this for me:
docker run --rm \
--entrypoint=/bin/bash \
-v /var/discourse/shared/standalone/postgres_data:/var/lib/postgresql/13/data \
-v /var/discourse/shared/standalone/postgres_data_new:/var/lib/postgresql/15/data \
-e POSTGRES_INITDB_ARGS="--locale=ru_RU.UTF-8" \
tianon/postgres-upgrade:13-to-15 \
-c "apt-get update && apt-get install -y postgresql-15-pgvector && sed -i 's/^# *\(ru_RU.UTF-8 UTF-8\)/\1/' /etc/locale.gen && locale-gen && docker-upgrade"
I also needed to fix the permissions on the postgres_data folder, otherwise there was a socket problem when rebuilding:
2025-01-31 05:38:30.430 GMT [44] LOG: skipping missing configuration file "/shared/postgres_data/postgresql.auto.conf"
2025-01-31 05:38:30.433 UTC [44] FATAL: data directory "/shared/postgres_data" has wrong ownership
2025-01-31 05:38:30.433 UTC [44] HINT: The server must be started by the user that owns the data directory.
I, [2025-01-31T05:38:35.384305 #1] INFO -- :
I, [2025-01-31T05:38:35.384481 #1] INFO -- : > /usr/local/bin/create_db
createdb: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
Is the server running locally and accepting connections on that socket?
I’m not sure and don’t remember what owner should be, so i gave it same UID/GID as postgres_backup and postgres_run folders that were nearby.
chown -R 101:104 /var/discourse/shared/standalone/postgres_data
Now everything works!
Thanks for the feedback, and sorry for the troubles. 
Ah, yes. The tianon container uses a different UID for the postgres user.
I’ve updated the instructions in OP based on your recommendations. Thank you!
Ok, I can give that a try. I assume, this means changing the settings below in app.yml? Or is there anything else I need to do? (e.g. adjust the session settings on the host system)
env:
LC_ALL: de_DE.UTF-8
LANG: de_DE.UTF-8
LANGUAGE: de_DE.UTF-8
This might be off topic but do you really need LANG and LANGUAGE, because you are using LC_ALL?
But yes, there you take locales in use.
This was really thoughtful, that is including it in the standard cleanup script, thanks!
You are right, probably not. I’ll have to check with the colleague that installed Discourse initially why he inserted all three.
I am also having the locales error.
(<unknown>): did not find expected key while parsing a block mapping at line 133 column 5 -e LANG=en_US.UTF-8
YAML syntax error. Please check your containers/*.yml config files.
It was caused by a spacing error at the bottom of the yml file. I fixed the spacing error, and the upgrade ran clean.
Swapping the locales in the app.yml, I did get a few steps further, but unfortunately not much. It seems that something still goes wrong and the postgres update is not applied correctly:
root@Ubuntu-2204-jammy-amd64-base /var/discourse # ./launcher rebuild app
x86_64 arch detected.
Ensuring launcher is up to date
Fetching origin
Launcher is up-to-date
Stopping old container
+ /usr/bin/docker stop -t 600 app
app
2.0.20250129-0720: Pulling from discourse/base
Digest: sha256:01b8516e5504c0e9bc3707773015ff4407be03a89154194ff3b5b8699291bc26
Status: Image is up to date for discourse/base:2.0.20250129-0720
docker.io/discourse/base:2.0.20250129-0720
/usr/local/lib/ruby/gems/3.3.0/gems/pups-1.2.1/lib/pups.rb
/usr/local/bin/pups --stdin
I, [2025-02-01T22:43:22.046506 #1] INFO -- : Reading from stdin
I, [2025-02-01T22:43:22.090615 #1] INFO -- : File > /etc/service/postgres/run chmod: +x chown:
I, [2025-02-01T22:43:22.095189 #1] INFO -- : File > /etc/service/postgres/log/run chmod: +x chown:
I, [2025-02-01T22:43:22.113260 #1] INFO -- : File > /etc/runit/3.d/99-postgres chmod: +x chown:
I, [2025-02-01T22:43:22.116842 #1] INFO -- : File > /root/install_postgres chmod: +x chown:
I, [2025-02-01T22:43:22.120381 #1] INFO -- : File > /root/upgrade_postgres chmod: +x chown:
I, [2025-02-01T22:43:22.165502 #1] INFO -- : Replacing data_directory = '/var/lib/postgresql/15/main' with data_directory = '/shared/postgres_data' in /etc/postgresql/15/main/postgresql.conf
I, [2025-02-01T22:43:22.192991 #1] INFO -- : Replacing (?-mix:#?listen_addresses *=.*) with listen_addresses = '*' in /etc/postgresql/15/main/postgresql.conf
I, [2025-02-01T22:43:22.193261 #1] INFO -- : Replacing (?-mix:#?synchronous_commit *=.*) with synchronous_commit = $db_synchronous_commit in /etc/postgresql/15/main/postgresql.conf
I, [2025-02-01T22:43:22.193606 #1] INFO -- : Replacing (?-mix:#?shared_buffers *=.*) with shared_buffers = $db_shared_buffers in /etc/postgresql/15/main/postgresql.conf
I, [2025-02-01T22:43:22.193923 #1] INFO -- : Replacing (?-mix:#?work_mem *=.*) with work_mem = $db_work_mem in /etc/postgresql/15/main/postgresql.conf
I, [2025-02-01T22:43:22.194221 #1] INFO -- : Replacing (?-mix:#?default_text_search_config *=.*) with default_text_search_config = '$db_default_text_search_config' in /etc/postgresql/15/main/postgresql.conf
I, [2025-02-01T22:43:22.194511 #1] INFO -- : Replacing (?-mix:#?checkpoint_segments *=.*) with checkpoint_segments = $db_checkpoint_segments in /etc/postgresql/15/main/postgresql.conf
I, [2025-02-01T22:43:22.194789 #1] INFO -- : Replacing (?-mix:#?logging_collector *=.*) with logging_collector = $db_logging_collector in /etc/postgresql/15/main/postgresql.conf
I, [2025-02-01T22:43:22.242968 #1] INFO -- : Replacing (?-mix:#?log_min_duration_statement *=.*) with log_min_duration_statement = $db_log_min_duration_statement in /etc/postgresql/15/main/postgresql.conf
I, [2025-02-01T22:43:22.256930 #1] INFO -- : Replacing (?-mix:^#local +replication +postgres +peer$) with local replication postgres peer in /etc/postgresql/15/main/pg_hba.conf
I, [2025-02-01T22:43:22.257319 #1] INFO -- : Replacing (?-mix:^host.*all.*all.*127.*$) with host all all 0.0.0.0/0 md5 in /etc/postgresql/15/main/pg_hba.conf
I, [2025-02-01T22:43:22.257663 #1] INFO -- : Replacing (?-mix:^host.*all.*all.*::1\/128.*$) with host all all ::/0 md5 in /etc/postgresql/15/main/pg_hba.conf
I, [2025-02-01T22:43:22.258234 #1] INFO -- : > if [ -f /root/install_postgres ]; then
/root/install_postgres && rm -f /root/install_postgres
elif [ -e /shared/postgres_run/.s.PGSQL.5432 ]; then
socat /dev/null UNIX-CONNECT:/shared/postgres_run/.s.PGSQL.5432 || exit 0 && echo postgres already running stop container ; exit 1
fi
2025/02/01 22:43:23 socat[33] E connect(, AF=1 "/shared/postgres_run/.s.PGSQL.5432", 36): Connection refused
I, [2025-02-01T22:43:23.856313 #1] INFO -- : Generating locales (this might take a while)...
en_US.UTF-8... done
Generation complete.
I, [2025-02-01T22:43:23.856538 #1] INFO -- : > HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/15/bin/postmaster -D /etc/postgresql/15/main
I, [2025-02-01T22:43:23.865162 #1] INFO -- : File > /usr/local/bin/create_db chmod: +x chown:
I, [2025-02-01T22:43:23.909221 #1] INFO -- : File > /var/lib/postgresql/take-database-backup chmod: +x chown: postgres:postgres
I, [2025-02-01T22:43:23.913112 #1] INFO -- : File > /var/spool/cron/crontabs/postgres chmod: chown:
I, [2025-02-01T22:43:23.913287 #1] INFO -- : > sleep 5
2025-02-01 22:43:25.004 UTC [35] FATAL: database files are incompatible with server
2025-02-01 22:43:25.004 UTC [35] DETAIL: The data directory was initialized by PostgreSQL version 13, which is not compatible with this version 15.10 (Debian 15.10-1.pgdg120+1).
I got the same error, still failing for me now too
I’ve tried
./discourse-doctor, but no luck
I, [2025-02-02T00:20:43.491510 #1] INFO -- : > if [ -f /root/install_postgres ]; then
/root/install_postgres && rm -f /root/install_postgres
elif [ -e /shared/postgres_run/.s.PGSQL.5432 ]; then
socat /dev/null UNIX-CONNECT:/shared/postgres_run/.s.PGSQL.5432 || exit 0 && echo postgres already running stop container ; exit 1
fi
2025/02/02 00:20:46 socat[33] E connect(, AF=1 "/shared/postgres_run/.s.PGSQL.5432", 36): Connection refused
I, [2025-02-02T00:20:46.466110 #1] INFO -- : Generating locales (this might take a while)...
en_US.UTF-8... done
Generation complete.
I, [2025-02-02T00:20:46.467179 #1] INFO -- : > HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/15/bin/postmaster -D /etc/postgresql/15/main
I, [2025-02-02T00:20:46.480451 #1] INFO -- : File > /usr/local/bin/create_db chmod: +x chown:
I, [2025-02-02T00:20:46.497621 #1] INFO -- : File > /var/lib/postgresql/take-database-backup chmod: +x chown: postgres:postgres
I, [2025-02-02T00:20:46.504526 #1] INFO -- : File > /var/spool/cron/crontabs/postgres chmod: chown:
I, [2025-02-02T00:20:46.505484 #1] INFO -- : > sleep 5
2025-02-02 00:20:46.647 UTC [35] FATAL: database files are incompatible with server
2025-02-02 00:20:46.647 UTC [35] DETAIL: The data directory was initialized by PostgreSQL version 13, which is not compatible with this version 15.10 (Debian 15.10-1.pgdg120+1).
I, [2025-02-02T00:20:51.511155 #1] INFO -- :
I, [2025-02-02T00:20:51.511342 #1] INFO -- : > /usr/local/bin/create_db
createdb: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: Connection refused
Is the server running locally and accepting connections on that socket?
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: Connection refused
Is the server running locally and accepting connections on that socket?
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: Connection refused
Is the server running locally and accepting connections on that socket?
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: Connection refused
Is the server running locally and accepting connections on that socket?
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: Connection refused
Is the server running locally and accepting connections on that socket?
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: Connection refused
Is the server running locally and accepting connections on that socket?
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: Connection refused
Is the server running locally and accepting connections on that socket?
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: Connection refused
Is the server running locally and accepting connections on that socket?
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: Connection refused
Is the server running locally and accepting connections on that socket?
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: Connection refused
Is the server running locally and accepting connections on that socket?
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: Connection refused
Is the server running locally and accepting connections on that socket?
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: Connection refused
Is the server running locally and accepting connections on that socket?
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: Connection refused
Is the server running locally and accepting connections on that socket?
I, [2025-02-02T00:20:52.474702 #1] INFO -- :
I, [2025-02-02T00:20:52.475371 #1] INFO -- : > echo postgres installed!
I, [2025-02-02T00:20:52.478084 #1] INFO -- : postgres installed!
and
I, [2025-02-02T00:05:22.648865 #1] INFO -- : > sleep 5
2025-02-02 00:05:22.792 UTC [35] FATAL: database files are incompatible with server
2025-02-02 00:05:22.792 UTC [35] DETAIL: The data directory was initialized by PostgreSQL version 13, which is not compatible with this version 15.10 (Debian 15.10-1.pgdg120+1).
I, [2025-02-02T00:05:27.657230 #1] INFO -- :
I, [2025-02-02T00:05:27.657387 #1] INFO -- : > /usr/local/bin/create_db
481:M 02 Feb 2025 00:07:05.174 # Failed listening on port 6379 (TCP), aborting.
I, [2025-02-02T00:13:38.491603 #1] INFO -- : > cd /var/www/discourse && su discourse -c 'bundle exec rake db:migrate'
URGENT: Failed to initialize site default: ActiveRecord::ConnectionNotEstablished connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: Connection refused
Is the server running locally and accepting connections on that socket?
rake aborted!
ActiveRecord::ConnectionNotEstablished: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: Connection refused (ActiveRecord::ConnectionNotEstablished)
Is the server running locally and accepting connections on that socket?
Caused by:
PG::ConnectionBad: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: Connection refused (PG::ConnectionBad)
Is the server running locally and accepting connections on that socket?
seems like it for me
I just tried and my server has this file, and I have rebuild failures
postgres@ubuntu-s-1vcpu-1gb-nyc3-01-app:/var/www/discourse$ ls -al /shared/postgres_run/.s.PGSQL.5432
srwxrwxrwx 1 postgres postgres 0 Feb 2 03:24 /shared/postgres_run/.s.PGSQL.5432
Hi @xFocus and @schneeland 
Is the database shutting down cleanly when you stop the app container? The logs should be similar to the output in OP.
tried to get around with this file, but failed
mv /shared/postgres_run/.s.PGSQL.5432 /shared/postgres_run/.s.PGSQL.5432.2025Feb1BackUp
root@ubuntu-s-1vcpu-1gb-nyc3-01-app:/var/www/discourse# ls -al /shared/postgres_run/.s.PGSQL.5432*
srwxrwxrwx 1 postgres postgres 0 Feb 2 03:24 /shared/postgres_run/.s.PGSQL.5432.2025Feb1BackUp
-rw------- 1 postgres postgres 62 Feb 2 03:24 /shared/postgres_run/.s.PGSQL.5432.lock
then rebuild still failing, The source cluster was not shut down cleanly.
@mwaniki
Stopping PostgreSQL 13 database server: main.
Stopping PostgreSQL 15 database server: main.
Performing Consistency Checks
-----------------------------
Checking cluster versions ok
The source cluster was not shut down cleanly.
Failure, exiting
-------------------------------------------------------------------------------------
UPGRADE OF POSTGRES FAILED
Please visit https://meta.discourse.org/t/postgresql-15-update/349515 for support.
You can run ./launcher start app to restart your app in the meanwhile
FAILED
--------------------
Pups::ExecError: if [ -f /root/install_postgres ]; then
/root/install_postgres && rm -f /root/install_postgres
elif [ -e /shared/postgres_run/.s.PGSQL.5432 ]; then
socat /dev/null UNIX-CONNECT:/shared/postgres_run/.s.PGSQL.5432 || exit 0 && echo postgres already running stop container ; exit 1
fi
failed with return #<Process::Status: pid 18 exit 1>
Location of failure: /usr/local/lib/ruby/gems/3.3.0/gems/pups-1.2.1/lib/pups/exec_command.rb:132:in `spawn'
exec failed with the params {"tag"=>"db", "cmd"=>"if [ -f /root/install_postgres ]; then\n /root/install_postgres && rm -f /root/install_postgres\nelif [ -e /shared/postgres_run/.s.PGSQL.5432 ]; then\n socat /dev/null UNIX-CONNECT:/shared/postgres_run/.s.PGSQL.5432 || exit 0 && echo postgres already running stop container ; exit 1\nfi\n"}
bootstrap failed with exit code 1
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.
931142f4cf49942fb3faf0676818c6395376c9fb6c76008d82037b0b76ae7111
I tried to run start and stop again
root@ubuntu-s-1vcpu-1gb-nyc3-01:/var/discourse# ./launcher start app
x86_64 arch detected.
starting up existing container
+ /usr/bin/docker start app
app
root@ubuntu-s-1vcpu-1gb-nyc3-01:/var/discourse# ./launcher stop app
x86_64 arch detected.
+ /usr/bin/docker stop -t 600 app
app
but I don’t have this log file… is it concerning?
thank you Mwaniki
root@ubuntu-s-1vcpu-1gb-nyc3-01:/var/discourse# tail -f shared/data/log/var-log/postgres/current
tail: cannot open 'shared/data/log/var-log/postgres/current' for reading: No such file or directory
tail: no files remaining
Yeah, that was a typo. Can you check for shared/standalone/log/var-log/postgres/current instead? I’ve just updated OP.
right I did not see the log LOG: database system is shut down
below is what I have after running ./launcher stop app , I guess that’s why during my rebuild i got error on that postgre port is in use already
2025-02-02 04:37:09.801 UTC [542] LOG: starting PostgreSQL 13.18 (Debian 13.18-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
2025-02-02 04:37:09.804 UTC [542] LOG: listening on IPv4 address "0.0.0.0", port 5432
2025-02-02 04:37:09.806 UTC [542] LOG: listening on IPv6 address "::", port 5432
2025-02-02 04:37:09.820 UTC [542] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2025-02-02 04:37:09.851 UTC [562] LOG: database system was interrupted; last known up at 2025-02-02 04:31:58 UTC
2025-02-02 04:37:10.267 UTC [562] LOG: database system was not properly shut down; automatic recovery in progress
2025-02-02 04:37:10.278 UTC [562] LOG: redo starts at 2/DB0AFFE0
2025-02-02 04:37:10.280 UTC [562] LOG: invalid record length at 2/DB0B5958: wanted 24, got 0
2025-02-02 04:37:10.280 UTC [562] LOG: redo done at 2/DB0B3990
2025-02-02 04:37:10.349 UTC [542] LOG: database system is ready to accept connections
2025-02-02 04:38:08.161 UTC [1345] discourse@discourse LOG: duration: 101.966 ms bind <unnamed>: SELECT "posts"."id", "posts"."user_id", "posts"."topic_id", "posts"."post_number", "posts"."raw", "posts"."cooked", "posts"."created_at", "posts"."updated_at", "posts"."reply_to_post_number", "posts"."reply_count", "posts"."quote_count", "posts"."deleted_at", "posts"."off_topic_count", "posts"."like_count", "posts"."incoming_link_count", "posts"."bookmark_count", "posts"."score", "posts"."reads", "posts"."post_type", "posts"."sort_order", "posts"."last_editor_id", "posts"."hidden", "posts"."hidden_reason_id", "posts"."notify_moderators_count", "posts"."spam_count", "posts"."illegal_count", "posts"."inappropriate_count", "posts"."last_version_at", "posts"."user_deleted", "posts"."reply_to_user_id", "posts"."percent_rank", "posts"."notify_user_count", "posts"."like_score", "posts"."deleted_by_id", "posts"."edit_reason", "posts"."word_count", "posts"."version", "posts"."cook_method", "posts"."wiki", "posts"."baked_at", "posts"."baked_version", "posts"."hidden_at", "posts"."self_edits", "posts"."reply_quoted", "posts"."via_email", "posts"."raw_email", "posts"."public_version", "posts"."action_code", "posts"."locked_by_id", "posts"."image_upload_id", "posts"."outbound_message_id", "posts"."qa_vote_count" FROM "posts" WHERE "posts"."deleted_at" IS NULL ORDER BY "posts"."id" ASC LIMIT 1
2025-02-02 04:38:10.917 UTC [1341] discourse@discourse LOG: duration: 238.937 ms statement: WITH tags_included_cte AS (