Update “3.4.0.beta4” failed

[anon55243134]

I received the usual email telling me to log in and click the GUI update buttons. They usually work, but this time, the entire site crashed without explanation.

Unable to use the website GUI, I SSHed into the droplet, changed into /var/discourse, and I tried to relaunch, but it freezes.

I closed the connection, SSHed again, changed into the director, and tried to rebuild, but it also freezes.

I see a few other posts with similar complaints, but no solutions. What should I do, please?

Here is the output when it freezes:

# ./launcher rebuild app
WARNING: Docker version 20.10.5 deprecated, recommend upgrade to 24.0.7 or newer.
x86_64 arch detected.
WARNING: containers/app.yml file is world-readable. You can secure this file by running: chmod o-rwx containers/app.yml
Ensuring launcher is up to date
Fetching origin
Launcher is up-to-date
2.0.20250129-0720: Pulling from discourse/base
Digest: sha256:01b8516e5504c0e9bc3707773015ff4407be03a89154194ff3b5b8699291bc26
Status: Image is up to date for discourse/base:2.0.20250129-0720
docker.io/discourse/base:2.0.20250129-0720
/usr/local/lib/ruby/gems/3.3.0/gems/pups-1.2.1/lib/pups.rb
/usr/local/bin/pups --stdin
I, [2025-02-06T23:22:07.378632 #1]  INFO -- : Reading from stdin
I, [2025-02-06T23:22:07.397972 #1]  INFO -- : File > /etc/service/postgres/run  chmod: +x  chown: 
I, [2025-02-06T23:22:07.403501 #1]  INFO -- : File > /etc/service/postgres/log/run  chmod: +x  chown: 
I, [2025-02-06T23:22:07.409307 #1]  INFO -- : File > /etc/runit/3.d/99-postgres  chmod: +x  chown: 
I, [2025-02-06T23:22:07.415306 #1]  INFO -- : File > /root/install_postgres  chmod: +x  chown: 
I, [2025-02-06T23:22:07.420321 #1]  INFO -- : File > /root/upgrade_postgres  chmod: +x  chown: 
I, [2025-02-06T23:22:07.421155 #1]  INFO -- : Replacing data_directory = '/var/lib/postgresql/15/main' with data_directory = '/shared/postgres_data' in /etc/postgresql/15/main/postgresql.conf
I, [2025-02-06T23:22:07.423599 #1]  INFO -- : Replacing (?-mix:#?listen_addresses *=.*) with listen_addresses = '*' in /etc/postgresql/15/main/postgresql.conf
I, [2025-02-06T23:22:07.428964 #1]  INFO -- : Replacing (?-mix:#?synchronous_commit *=.*) with synchronous_commit = $db_synchronous_commit in /etc/postgresql/15/main/postgresql.conf
I, [2025-02-06T23:22:07.434970 #1]  INFO -- : Replacing (?-mix:#?shared_buffers *=.*) with shared_buffers = $db_shared_buffers in /etc/postgresql/15/main/postgresql.conf
I, [2025-02-06T23:22:07.435971 #1]  INFO -- : Replacing (?-mix:#?work_mem *=.*) with work_mem = $db_work_mem in /etc/postgresql/15/main/postgresql.conf
I, [2025-02-06T23:22:07.437209 #1]  INFO -- : Replacing (?-mix:#?default_text_search_config *=.*) with default_text_search_config = '$db_default_text_search_config' in /etc/postgresql/15/main/postgresql.conf
I, [2025-02-06T23:22:07.438032 #1]  INFO -- : Replacing (?-mix:#?checkpoint_segments *=.*) with checkpoint_segments = $db_checkpoint_segments in /etc/postgresql/15/main/postgresql.conf
I, [2025-02-06T23:22:07.438766 #1]  INFO -- : Replacing (?-mix:#?logging_collector *=.*) with logging_collector = $db_logging_collector in /etc/postgresql/15/main/postgresql.conf
I, [2025-02-06T23:22:07.439506 #1]  INFO -- : Replacing (?-mix:#?log_min_duration_statement *=.*) with log_min_duration_statement = $db_log_min_duration_statement in /etc/postgresql/15/main/postgresql.conf
I, [2025-02-06T23:22:07.440192 #1]  INFO -- : Replacing (?-mix:^#local +replication +postgres +peer$) with local replication postgres  peer in /etc/postgresql/15/main/pg_hba.conf
I, [2025-02-06T23:22:07.440757 #1]  INFO -- : Replacing (?-mix:^host.*all.*all.*127.*$) with host all all 0.0.0.0/0 md5 in /etc/postgresql/15/main/pg_hba.conf
I, [2025-02-06T23:22:07.441342 #1]  INFO -- : Replacing (?-mix:^host.*all.*all.*::1\/128.*$) with host all all ::/0 md5 in /etc/postgresql/15/main/pg_hba.conf
I, [2025-02-06T23:22:07.442038 #1]  INFO -- : > if [ -f /root/install_postgres ]; then
  /root/install_postgres && rm -f /root/install_postgres
elif [ -e /shared/postgres_run/.s.PGSQL.5432 ]; then
  socat /dev/null UNIX-CONNECT:/shared/postgres_run/.s.PGSQL.5432 || exit 0 && echo postgres already running stop container ; exit 1
fi

failed to set locale!
[error] character map file `UTF-8' not found: No such file or directory
failed to set locale!
[error] default character map file `ANSI_X3.4-1968' not found: No such file or directory
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
	LANGUAGE = "en_US.UTF-8",
	LC_ALL = "en_US.UTF-8",
	LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
cat: /shared/postgres_data/PG_VERSION: No such file or directory
popen failure: Cannot allocate memory
initdb: error: program "postgres" is needed by initdb but was not found in the same directory as "/usr/lib/postgresql/15/bin/initdb"
I, [2025-02-06T23:22:09.511169 #1]  INFO -- : Generating locales (this might take a while)...
  en_US.UTF-8... done
Generation complete.
Upgrading PostgreSQL from version to 15

I, [2025-02-06T23:22:09.511908 #1]  INFO -- : > HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/15/bin/postmaster -D /etc/postgresql/15/main
I, [2025-02-06T23:22:09.514561 #1]  INFO -- : Terminating async processes
2025-02-06 23:22:09.541 GMT [51] LOG:  skipping missing configuration file "/shared/postgres_data/postgresql.auto.conf"
2025-02-06 23:22:09.542 UTC [51] FATAL:  data directory "/shared/postgres_data" has invalid permissions
2025-02-06 23:22:09.542 UTC [51] DETAIL:  Permissions should be u=rwx (0700) or u=rwx,g=rx (0750).

[pfaffman (Jay Pfaffman)]

See PostgreSQL 15 update

[pfaffman (Jay Pfaffman)]

I think you need to

@pfaffman

chown -R 101:104 /var/discourse/shared/standalone/postgres_*

[pfaffman (Jay Pfaffman)]

@anon55243134

Permissions should be u=rwx (0700) or u=rwx,g=rx (0750)

You need to change the permissions (I was wrong about the owner)

[sober]

I have the same problem

PostgreSQL update fails from China

Are you able to reach the repository below from your server? curl -v https://apt.postgresql.org/pub/repos/apt/ The repository is needed to install the old PostgreSQL packages used during the update. This is typically caused when the database was not shut down properly. We had fixed it earlier when you started and stopped the data container. The error is not seen in your latest log output.

[RGJ (Richard - Communiteq)]

@anon55243134

WARNING: Docker version 20.10.5 deprecated, recommend upgrade to 24.0.7 or newer.

@anon55243134

popen failure: Cannot allocate memory

You need to fix these issues. What are the specs of your server?

[pfaffman (Jay Pfaffman)]

OH! I hadn’t noticed the Docker version problem.

@anon55243134, you should probably make a backup (if possible) and move to a new VM (using Move a Discourse site to another VPS with rsync if you don’t have a recent backup).

[rahim123]

Related:

PSA: Stable 3.3.3 -> 3.4.0 upgrade via GUI failed, had to rebuild via command line

Just a heads-up for self-hosted stable branch users. For me the upgrade via the GUI failed, I’m not sure if this would happen to everyone, as I saw some errors about unicorn processes, which tends to be a random issue in my experience. But at any rate that left my forum in a state where it claimed to be up-to-date in /admin/update while still being stuck on the old version. So I had to SSH into the server and /var/discourse/launcher rebuild app two times, the first time it just upgrades Postgre…

[sober]

PostgreSQL update fails from China

I got the upgrade PostgreSQL15 Templates need to be used in sequence postgres.13.template.yml postgres.15.template.yml postgres.template.yml

[pfaffman (Jay Pfaffman)]

@anon55243134

What’s wrong with our current VM?

My guess is that it’s beyond end of life.

Spinning up a new vm and switching to it requires zero downtime and if something goes wrong you just stay on the existing server. It’s a bit complicated though.

Sorry that you had no choice but to go to an old backup. I’d switch to daily backups and take a backup before you update so you won’t need to lose anything.

You can switch to the beta or stable branches before you upgrsde. You can read about Configure a supported tracking branch to get Discourse software updates

[sober]

I am using data.yml. If you are using app.yml, take note of the referenced configuration for postgres.template.yml within app.yml. During the database upgrade process, hierarchical references must be utilized to achieve the upgrade from version 13 to 15.