Hey posting here after trying everything and got stuck with my upgrade.
- I was updating the forum after many months. I ran a backup as usual and downloaded it.
- Apparently, in this update process, there is a migration from PG 13 to 15.
Everything was running with no issues until the update failed I was interrupted due to lack of disk space. - I went to my Digitalocean droplet, stoped it, expanded the disk, and turn it on again
- I ran some regular commants to release disk space (removing old logs, ./launcher cleanup, etc)
So I started the rebuild, and then then hit the first wall:
root@discourse-ubuntu-s-2vcpu-4gb-fra1-01:/var/discourse# ./launcher rebuild app
WARNING: Docker version 20.10.1 deprecated, recommend upgrade to 24.0.7 or newer.
x86_64 arch detected.
WARNING: containers/app.yml file is world-readable. You can secure this file by running: chmod o-rwx containers/app.yml
Ensuring launcher is up to date
Fetching origin
Launcher is up-to-date
2.0.20250226-0128: Pulling from discourse/base
Digest: sha256:6f18aa2cd22bba0deb91d69194e577d4f96130ad555ae8ec646a8792cbfe37db
Status: Image is up to date for discourse/base:2.0.20250226-0128
docker.io/discourse/base:2.0.20250226-0128
/usr/local/lib/ruby/gems/3.3.0/gems/pups-1.2.1/lib/pups.rb
/usr/local/bin/pups --stdin
I, [2025-06-11T08:57:06.040908 #1] INFO -- : Reading from stdin
I, [2025-06-11T08:57:06.055466 #1] INFO -- : File > /etc/service/postgres/run chmod: +x chown:
I, [2025-06-11T08:57:06.059785 #1] INFO -- : File > /etc/service/postgres/log/run chmod: +x chown:
I, [2025-06-11T08:57:06.064063 #1] INFO -- : File > /etc/runit/3.d/99-postgres chmod: +x chown:
I, [2025-06-11T08:57:06.068330 #1] INFO -- : File > /root/install_postgres chmod: +x chown:
I, [2025-06-11T08:57:06.072788 #1] INFO -- : File > /root/upgrade_postgres chmod: +x chown:
I, [2025-06-11T08:57:06.073111 #1] INFO -- : Replacing data_directory = '/var/lib/postgresql/15/main' with data_directory = '/shared/postgres_data' in /etc/postgresql/15/main/postgresql.conf
I, [2025-06-11T08:57:06.074045 #1] INFO -- : Replacing (?-mix:#?listen_addresses *=.*) with listen_addresses = '*' in /etc/postgresql/15/main/postgresql.conf
I, [2025-06-11T08:57:06.074682 #1] INFO -- : Replacing (?-mix:#?synchronous_commit *=.*) with synchronous_commit = $db_synchronous_commit in /etc/postgresql/15/main/postgresql.conf
I, [2025-06-11T08:57:06.075269 #1] INFO -- : Replacing (?-mix:#?shared_buffers *=.*) with shared_buffers = $db_shared_buffers in /etc/postgresql/15/main/postgresql.conf
I, [2025-06-11T08:57:06.080274 #1] INFO -- : Replacing (?-mix:#?work_mem *=.*) with work_mem = $db_work_mem in /etc/postgresql/15/main/postgresql.conf
I, [2025-06-11T08:57:06.081190 #1] INFO -- : Replacing (?-mix:#?default_text_search_config *=.*) with default_text_search_config = '$db_default_text_search_config' in /etc/postgresql/15/main/postgresql.conf
I, [2025-06-11T08:57:06.081687 #1] INFO -- : Replacing (?-mix:#?checkpoint_segments *=.*) with checkpoint_segments = $db_checkpoint_segments in /etc/postgresql/15/main/postgresql.conf
I, [2025-06-11T08:57:06.082397 #1] INFO -- : Replacing (?-mix:#?logging_collector *=.*) with logging_collector = $db_logging_collector in /etc/postgresql/15/main/postgresql.conf
I, [2025-06-11T08:57:06.083025 #1] INFO -- : Replacing (?-mix:#?log_min_duration_statement *=.*) with log_min_duration_statement = $db_log_min_duration_statement in /etc/postgresql/15/main/postgresql.conf
I, [2025-06-11T08:57:06.083657 #1] INFO -- : Replacing (?-mix:^#local +replication +postgres +peer$) with local replication postgres peer in /etc/postgresql/15/main/pg_hba.conf
I, [2025-06-11T08:57:06.083968 #1] INFO -- : Replacing (?-mix:^host.*all.*all.*127.*$) with host all all 0.0.0.0/0 md5 in /etc/postgresql/15/main/pg_hba.conf
I, [2025-06-11T08:57:06.084391 #1] INFO -- : Replacing (?-mix:^host.*all.*all.*::1\/128.*$) with host all all ::/0 md5 in /etc/postgresql/15/main/pg_hba.conf
I, [2025-06-11T08:57:06.084877 #1] INFO -- : > if [ -f /root/install_postgres ]; then
/root/install_postgres && rm -f /root/install_postgres
elif [ -e /shared/postgres_run/.s.PGSQL.5432 ]; then
socat /dev/null UNIX-CONNECT:/shared/postgres_run/.s.PGSQL.5432 || exit 0 && echo postgres already running stop container ; exit 1
fi
failed to set locale!
[error] character map file `UTF-8' not found: No such file or directory
failed to set locale!
[error] default character map file `ANSI_X3.4-1968' not found: No such file or directory
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = "en_US.UTF-8",
LC_ALL = "en_US.UTF-8",
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
popen failure: Cannot allocate memory
initdb: error: program "postgres" is needed by initdb but was not found in the same directory as "/usr/lib/postgresql/15/bin/initdb"
I, [2025-06-11T08:57:07.265267 #1] INFO -- : Generating locales (this might take a while)...
en_US.UTF-8... done
Generation complete.
Upgrading PostgreSQL from version 13 to 15
I, [2025-06-11T08:57:07.265539 #1] INFO -- : > HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/15/bin/postmaster -D /etc/postgresql/15/main
I, [2025-06-11T08:57:07.267705 #1] INFO -- : Terminating async processes
2025-06-11 08:57:07.283 UTC [50] FATAL: database files are incompatible with server
2025-06-11 08:57:07.283 UTC [50] DETAIL: The data directory was initialized by PostgreSQL version 13, which is not compatible with this version 15.12 (Debian 15.12-1.pgdg120+1).
Searching on Google and a bit of chatGPT to understand the log, I decided to go back to PG 13 and focus on restore the forum, editing the template in the yml to force âtemplates/postgres.13.template.yml
â
And the hit the second wall:
root@discourse-ubuntu-s-2vcpu-4gb-fra1-01:/var/discourse# ./launcher rebuild app
WARNING: Docker version 20.10.1 deprecated, recommend upgrade to 24.0.7 or newer.
x86_64 arch detected.
WARNING: containers/app.yml file is world-readable. You can secure this file by running: chmod o-rwx containers/app.yml
Ensuring launcher is up to date
Fetching origin
Launcher is up-to-date
2.0.20250226-0128: Pulling from discourse/base
Digest: sha256:6f18aa2cd22bba0deb91d69194e577d4f96130ad555ae8ec646a8792cbfe37db
Status: Image is up to date for discourse/base:2.0.20250226-0128
docker.io/discourse/base:2.0.20250226-0128
/usr/local/lib/ruby/gems/3.3.0/gems/pups-1.2.1/lib/pups.rb
/usr/local/bin/pups --stdin
I, [2025-06-11T09:07:40.418731 #1] INFO -- : Reading from stdin
I, [2025-06-11T09:07:40.428644 #1] INFO -- : > DEBIAN_FRONTEND=noninteractive apt-get purge -y postgresql-15 postgresql-client-15 postgresql-contrib-15 postgresql-15-pgvector
E: Problem executing scripts DPkg::Post-Invoke 'rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin || true'
E: Sub-process returned an error code
I, [2025-06-11T09:07:41.892381 #1] INFO -- : Reading package lists...
Building dependency tree...
Reading state information...
The following packages were automatically installed and are no longer required:
libcommon-sense-perl libio-pty-perl libipc-run-perl libjson-perl
libjson-xs-perl libllvm19 libtypes-serialiser-perl libz3-4
postgresql-client-common postgresql-common postgresql-common-dev ssl-cert
Use 'apt autoremove' to remove them.
The following packages will be REMOVED:
postgresql-15* postgresql-15-pgvector* postgresql-client-15*
0 upgraded, 0 newly installed, 3 to remove and 0 not upgraded.
After this operation, 60.9 MB disk space will be freed.
(Reading database ... 33951 files and directories currently installed.)
Removing postgresql-15-pgvector (0.8.0-1.pgdg120+1) ...
Removing postgresql-15 (15.12-1.pgdg120+1) ...
invoke-rc.d: could not determine current runlevel
update-alternatives: warning: forcing reinstallation of alternative /usr/share/postgresql/15/man/man1/psql.1.gz because link group psql.1.gz is broken
Removing postgresql-client-15 (15.12-1.pgdg120+1) ...
Processing triggers for postgresql-common (274.pgdg120+1) ...
Building PostgreSQL dictionaries from installed myspell/hunspell packages...
Removing obsolete dictionary files:
(Reading database ... 31913 files and directories currently installed.)
Purging configuration files for postgresql-15 (15.12-1.pgdg120+1) ...
Dropping cluster main...
FAILED
--------------------
Pups::ExecError: DEBIAN_FRONTEND=noninteractive apt-get purge -y postgresql-15 postgresql-client-15 postgresql-contrib-15 postgresql-15-pgvector failed with return #<Process::Status: pid 7 exit 100>
Location of failure: /usr/local/lib/ruby/gems/3.3.0/gems/pups-1.2.1/lib/pups/exec_command.rb:132:in `spawn'
exec failed with the params "DEBIAN_FRONTEND=noninteractive apt-get purge -y postgresql-15 postgresql-client-15 postgresql-contrib-15 postgresql-15-pgvector"
bootstrap failed with exit code 100
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.
59b3543b15880ef0391688f71c9cb827ef791dbc37f1e4f896d9d7a31841c83b
So now Iâm stuck and not sure how to process, any recomendation would be great, it is a huge forum and I have just a few hours to put it back online before getting fire