Upgrade failed spectacularily

Support
1

I have updated my several times over the years and mostly without issue. I got a message that my discourse needed to be updated. Initially i tried to use the web update but it said I needed to login and run ./launcher app rebuild. Which I did.

Below is the failure which appears to be related to the postgres upgrade. I searched the forum and see that there seems to be special instructions to upgrade postgres but not knowing that this was part of this update I did not follow them,. After the failure I tried to follow the steps but those fail as well.

this is the ouotput of the failed update:

wordsmith@digitalwind-ubuntu-20:/var/discourse$ sudo ./launcher rebuild app
[sudo] password for wordsmith:
Sorry, try again.
[sudo] password for wordsmith:
Sorry, try again.
[sudo] password for wordsmith:
x86_64 arch detected.
WARNING: containers/app.yml file is world-readable. You can secure this file by running: chmod o-rwx containers/app.yml
Ensuring launcher is up to date
Fetching origin
remote: Enumerating objects: 765, done.
remote: Counting objects: 100% (299/299), done.
remote: Compressing objects: 100% (67/67), done.
remote: Total 765 (delta 252), reused 244 (delta 226), pack-reused 466 (from 4)
Receiving objects: 100% (765/765), 292.42 KiB | 9.75 MiB/s, done.
Resolving deltas: 100% (430/430), completed with 44 local objects.
From https://github.com/discourse/discourse_docker
   ffa8264..efd6bc8  main                               -> origin/main
 * [new branch]      consolidate_layers                 -> origin/consolidate_layers
 * [new branch]      dev-nginx-conf                     -> origin/dev-nginx-conf
 * [new branch]      docker-compose-generate            -> origin/docker-compose-generate
 * [new branch]      gh-action-upload-to-release        -> origin/gh-action-upload-to-release
 * [new branch]      nginx-conf-updates                 -> origin/nginx-conf-updates
 * [new branch]      parameterize-config-filename       -> origin/parameterize-config-filename
 * [new branch]      pento/dev-test-pnpm-10             -> origin/pento/dev-test-pnpm-10
 * [new branch]      revert_only_shipping_en_us_locales -> origin/revert_only_shipping_en_us_locales
 * [new branch]      slim-image-3                       -> origin/slim-image-3
 * [new branch]      update-build-compile-steps         -> origin/update-build-compile-steps
 * [new branch]      update-launcher                    -> origin/update-launcher
 * [new branch]      update_tag                         -> origin/update_tag
Updating Launcher...
Updating ffa8264..efd6bc8
Fast-forward
 .github/workflows/build.yml                   | 257 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-----------------------------------
 .gitignore                                    |   2 +
 discourse-setup                               |  23 ++++++----
 image/README.md                               |  12 +----
 image/auto_build.rb                           | 103 +++++++++++++++++++++++++++++++++---------
 image/base/Dockerfile                         | 178 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 image/base/install-imagemagick                |  26 +++++------
 image/base/install-jemalloc                   |   6 +--
 image/base/install-nginx                      |  13 +++---
 image/base/install-oxipng                     |  35 ++++++++-------
 image/base/install-redis                      |   4 +-
 image/base/install-ruby                       |  16 -------
 image/base/install-rust                       |   2 +-
 image/base/nginx_public_keys.key              | 374 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 image/base/release.Dockerfile                 |  16 -------
 image/base/slim.Dockerfile                    | 128 ----------------------------------------------------
 image/discourse_bench/Dockerfile              |  34 --------------
 image/discourse_bench/run_bench.sh            |  19 --------
 image/discourse_dev/Dockerfile                |  38 ++++++++++++++--
 image/discourse_dev/postgres_dev.template.yml |  41 +++--------------
 image/discourse_fast_switch/Dockerfile        |  23 ----------
 image/discourse_fast_switch/create_switch.rb  |  24 ----------
 image/discourse_test/Dockerfile               |  10 ++---
 image/discourse_test/install-chrome           |   2 +-
 image/monitor/Dockerfile                      |  13 ------
 image/monitor/src/monitor.rb                  |  98 ----------------------------------------
 launcher                                      |   6 +--
 scripts/launcher-completion.bash              |  44 ++++++++++++++++++
 templates/import/mbox.template.yml            |   2 +-
 templates/import/mssql-dep.template.yml       |   2 +-
 templates/import/mysql-dep.template.yml       |   2 +-
 templates/import/phpbb3.template.yml          |   2 +-
 templates/import/vanilla.template.yml         |  30 +++++++------
 templates/postgres.10.template.yml            |   3 +-
 templates/postgres.12.template.yml            |   3 +-
 templates/postgres.13.template.yml            | 175 +++++++++++++++++++++++++++--------------------------------------------
 templates/postgres.15.template.yml            |  62 ++++++++++++++------------
 templates/postgres.9.5.template.yml           |   3 +-
 templates/postgres.template.yml               | 105 ++++++++++++++++++++++++++-----------------
 templates/redis.template.yml                  |   6 ++-
 templates/web.china.template.yml              |   3 +-
 templates/web.socketed.template.yml           |   4 +-
 templates/web.ssl.template.yml                |  32 ++++++-------
 templates/web.template.yml                    |  23 +++++-----
 44 files changed, 1186 insertions(+), 818 deletions(-)
 create mode 100644 image/base/Dockerfile
 delete mode 100755 image/base/install-ruby
 create mode 100644 image/base/nginx_public_keys.key
 delete mode 100644 image/base/release.Dockerfile
 delete mode 100644 image/base/slim.Dockerfile
 delete mode 100644 image/discourse_bench/Dockerfile
 delete mode 100644 image/discourse_bench/run_bench.sh
 delete mode 100644 image/discourse_fast_switch/Dockerfile
 delete mode 100644 image/discourse_fast_switch/create_switch.rb
 delete mode 100644 image/monitor/Dockerfile
 delete mode 100644 image/monitor/src/monitor.rb
 create mode 100644 scripts/launcher-completion.bash
Launcher updated, restarting...
WARNING: Docker version 20.10.7 deprecated, recommend upgrade to 24.0.7 or newer.
x86_64 arch detected.

WARNING: We are about to start downloading the Discourse base image
This process may take anywhere between a few minutes to an hour, depending on your network speed

Please be patient

2.0.20250129-0720: Pulling from discourse/base
af302e5c37e9: Pull complete
ed49ab5c933d: Pull complete
7cb9e730d898: Pull complete
7c28a6b187b2: Pull complete
23222af7238f: Pull complete
59b87d9ed5c1: Pull complete
90d60b432298: Pull complete
d3b3135aa82b: Pull complete
94e7b594395d: Pull complete
cf15e2c89d00: Pull complete
2c0cccf9ec27: Pull complete
5d8ea2a8d564: Pull complete
020f04b42275: Pull complete
06e22604f387: Pull complete
8b0ecc579aad: Pull complete
4d577f7424c0: Pull complete
37ef4781925f: Pull complete
30d67d1af761: Pull complete
43eaf8c26af4: Pull complete
02f4cc50eabd: Pull complete
93896ee36e56: Pull complete
fadd5d615e0e: Pull complete
85f9459cb30e: Pull complete
eaddca2afacb: Pull complete
f54443d29a2d: Pull complete
ed6f0f133683: Pull complete
3d1894dc563f: Pull complete
51ca3f63c909: Pull complete
b87374db2274: Pull complete
Digest: sha256:01b8516e5504c0e9bc3707773015ff4407be03a89154194ff3b5b8699291bc26
Status: Downloaded newer image for discourse/base:2.0.20250129-0720
docker.io/discourse/base:2.0.20250129-0720
WARNING: containers/app.yml file is world-readable. You can secure this file by running: chmod o-rwx containers/app.yml
Ensuring launcher is up to date
Fetching origin
Launcher is up-to-date
Stopping old container
+ /usr/bin/docker stop -t 600 app
app
2.0.20250129-0720: Pulling from discourse/base
Digest: sha256:01b8516e5504c0e9bc3707773015ff4407be03a89154194ff3b5b8699291bc26
Status: Image is up to date for discourse/base:2.0.20250129-0720
docker.io/discourse/base:2.0.20250129-0720
/usr/local/lib/ruby/gems/3.3.0/gems/pups-1.2.1/lib/pups.rb
/usr/local/bin/pups --stdin
I, [2025-02-06T00:07:56.208669 #1]  INFO -- : Reading from stdin
I, [2025-02-06T00:07:56.232158 #1]  INFO -- : File > /etc/service/postgres/run  chmod: +x  chown:
I, [2025-02-06T00:07:56.237056 #1]  INFO -- : File > /etc/service/postgres/log/run  chmod: +x  chown:
I, [2025-02-06T00:07:56.242252 #1]  INFO -- : File > /etc/runit/3.d/99-postgres  chmod: +x  chown:
I, [2025-02-06T00:07:56.246861 #1]  INFO -- : File > /root/install_postgres  chmod: +x  chown:
I, [2025-02-06T00:07:56.251666 #1]  INFO -- : File > /root/upgrade_postgres  chmod: +x  chown:
I, [2025-02-06T00:07:56.253133 #1]  INFO -- : Replacing data_directory = '/var/lib/postgresql/15/main' with data_directory = '/shared/postgres_data' in /etc/postgresql/15/main/postgresql.conf
I, [2025-02-06T00:07:56.254477 #1]  INFO -- : Replacing (?-mix:#?listen_addresses *=.*) with listen_addresses = '*' in /etc/postgresql/15/main/postgresql.conf
I, [2025-02-06T00:07:56.255380 #1]  INFO -- : Replacing (?-mix:#?synchronous_commit *=.*) with synchronous_commit = $db_synchronous_commit in /etc/postgresql/15/main/postgresql.conf
I, [2025-02-06T00:07:56.256084 #1]  INFO -- : Replacing (?-mix:#?shared_buffers *=.*) with shared_buffers = $db_shared_buffers in /etc/postgresql/15/main/postgresql.conf
I, [2025-02-06T00:07:56.256905 #1]  INFO -- : Replacing (?-mix:#?work_mem *=.*) with work_mem = $db_work_mem in /etc/postgresql/15/main/postgresql.conf
I, [2025-02-06T00:07:56.257717 #1]  INFO -- : Replacing (?-mix:#?default_text_search_config *=.*) with default_text_search_config = '$db_default_text_search_config' in /etc/postgresql/15/main/postgresql.conf
I, [2025-02-06T00:07:56.258521 #1]  INFO -- : Replacing (?-mix:#?checkpoint_segments *=.*) with checkpoint_segments = $db_checkpoint_segments in /etc/postgresql/15/main/postgresql.conf
I, [2025-02-06T00:07:56.259186 #1]  INFO -- : Replacing (?-mix:#?logging_collector *=.*) with logging_collector = $db_logging_collector in /etc/postgresql/15/main/postgresql.conf
I, [2025-02-06T00:07:56.265147 #1]  INFO -- : Replacing (?-mix:#?log_min_duration_statement *=.*) with log_min_duration_statement = $db_log_min_duration_statement in /etc/postgresql/15/main/postgresql.conf
I, [2025-02-06T00:07:56.266734 #1]  INFO -- : Replacing (?-mix:^#local +replication +postgres +peer$) with local replication postgres  peer in /etc/postgresql/15/main/pg_hba.conf
I, [2025-02-06T00:07:56.267608 #1]  INFO -- : Replacing (?-mix:^host.*all.*all.*127.*$) with host all all 0.0.0.0/0 md5 in /etc/postgresql/15/main/pg_hba.conf
I, [2025-02-06T00:07:56.268347 #1]  INFO -- : Replacing (?-mix:^host.*all.*all.*::1\/128.*$) with host all all ::/0 md5 in /etc/postgresql/15/main/pg_hba.conf
I, [2025-02-06T00:07:56.268965 #1]  INFO -- : > if [ -f /root/install_postgres ]; then
  /root/install_postgres && rm -f /root/install_postgres
elif [ -e /shared/postgres_run/.s.PGSQL.5432 ]; then
  socat /dev/null UNIX-CONNECT:/shared/postgres_run/.s.PGSQL.5432 || exit 0 && echo postgres already running stop container ; exit 1
fi

failed to set locale!
[error] character map file `UTF-8' not found: No such file or directory
failed to set locale!
[error] default character map file `ANSI_X3.4-1968' not found: No such file or directory
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
	LANGUAGE = "en_US.UTF-8",
	LC_ALL = "en_US.UTF-8",
	LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
popen failure: Cannot allocate memory
initdb: error: program "postgres" is needed by initdb but was not found in the same directory as "/usr/lib/postgresql/15/bin/initdb"
I, [2025-02-06T00:07:59.495629 #1]  INFO -- : Generating locales (this might take a while)...
  en_US.UTF-8... done
Generation complete.
Upgrading PostgreSQL from version 13 to 15

I, [2025-02-06T00:07:59.496900 #1]  INFO -- : > HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/15/bin/postmaster -D /etc/postgresql/15/main
I, [2025-02-06T00:07:59.500783 #1]  INFO -- : Terminating async processes
2025-02-06 00:07:59.629 UTC [51] FATAL:  database files are incompatible with server
2025-02-06 00:07:59.629 UTC [51] DETAIL:  The data directory was initialized by PostgreSQL version 13, which is not compatible with this version 15.10 (Debian 15.10-1.pgdg120+1).

Help!!!

4 Likes
2

there is an update to Postgre SQL database version in this update.
My install is new (6 months) but I was still nervous after watching this forum for the last few days. Several reports of upgrade not working perfectly but mostly with older and/or migrated databases

what is history of your install? for more hints

Here is more info about db upgrade and troubleshooting info

Good news, my upgrade went perfect so it the upgrade works.

good luck yeah

3

I so wish I had emailed myself a backup before I started!
AND tested my linode backups work… I just restored one and it doesn’t contain what I thought.

7

Could you share contents of your app.yml file? Redact any sensitive information before posting. @Neptune

Also, confirm your Docker version. See the topic below for a similar issue which was resolved by updating Docker.

1 Like
9

Docker version 20.10.7, build f0df350

here is my app.yml

## this is the all-in-one, standalone Discourse Docker container template
##
## After making changes to this file, you MUST rebuild
## /var/discourse/launcher rebuild app
##
## BE *VERY* CAREFUL WHEN EDITING!
## YAML FILES ARE SUPER SUPER SENSITIVE TO MISTAKES IN WHITESPACE OR ALIGNMENT!
## visit http://www.yamllint.com/ to validate this file as needed
templates:
  - "templates/postgres.template.yml"
  - "templates/redis.template.yml"
  - "templates/web.template.yml"
  - "templates/web.ratelimited.template.yml"
  - "templates/web.socketed.template.yml"
## Uncomment these two lines if you wish to add Lets Encrypt (https)
## - "templates/web.ssl.template.yml"
## - "templates/web.letsencrypt.ssl.template.yml"

## which TCP/IP ports should this container expose?
## If you want Discourse to share a port with another webserver like Apache or nginx,
## see https://meta.discourse.org/t/17247 for details
## expose:
##   - "8080:80"   # http
##   - "8443:443" # https

params:
  db_default_text_search_config: "pg_catalog.english"

  ## Set db_shared_buffers to a max of 25% of the total memory.
  ## will be set automatically by bootstrap based on detected RAM, or you can override
  db_shared_buffers: "256MB"

  ## can improve sorting performance, but adds memory usage per-connection
  #db_work_mem: "40MB"

  ## Which Git revision should this container use? (default: tests-passed)
  #version: tests-passed

env:
  LC_ALL: en_US.UTF-8
  LANG: en_US.UTF-8
  LANGUAGE: en_US.UTF-8
  # DISCOURSE_DEFAULT_LOCALE: en

  ## How many concurrent web requests are supported? Depends on memory and CPU cores.
  ## will be set automatically by bootstrap based on detected CPUs, or you can override
  UNICORN_WORKERS: 4

  ## TODO: The domain name this Discourse instance will respond to
  ## Required. Discourse will not work with a bare IP number.
  DISCOURSE_HOSTNAME: wordsmiths.digitalwind.net

  ## Uncomment if you want the container to be started with the same
  ## hostname (-h option) as specified above (default "$hostname-$config")
  #DOCKER_USE_HOSTNAME: true

  ## TODO: List of comma delimited emails that will be made admin and developer
  ## on initial signup example 'user1@example.com,user2@example.com'
  DISCOURSE_DEVELOPER_EMAILS: 'xxx@dev.null'

  ## TODO: The SMTP mail server used to validate new accounts and send notifications
  # SMTP ADDRESS, username, and password are required
  # WARNING the char '#' in SMTP password can cause problems!
  DISCOURSE_SMTP_ADDRESS: dev.null
  DISCOURSE_SMTP_PORT: 587
  DISCOURSE_SMTP_USER_NAME: xxx@dev.null
  DISCOURSE_SMTP_PASSWORD: "not-here"
  DISCOURSE_SMTP_OPENSSL_VERIFY_MODE: nope
  #DISCOURSE_SMTP_ENABLE_START_TLS: true           # (optional, default true)
  DISCOURSE_SMTP_DOMAIN: dev.null
  DISCOURSE_NOTIFICATION_EMAIL: xxx@dev.null

  ## If you added the Lets Encrypt template, uncomment below to get a free SSL certificate
  LETSENCRYPT_ACCOUNT_EMAIL: xxx@dev.null

  ## The http or https CDN address for this Discourse instance (configured to pull)
  ## see https://meta.discourse.org/t/14857 for details
  #DISCOURSE_CDN_URL: https://discourse-cdn.example.com
  
  ## The maxmind geolocation IP address key for IP address lookup
  ## see https://meta.discourse.org/t/-/137387/23 for details
  #DISCOURSE_MAXMIND_LICENSE_KEY: 1234567890123456

## The Docker container is stateless; all data is stored in /shared
volumes:
  - volume:
      host: /var/discourse/shared/standalone
      guest: /shared
  - volume:
      host: /var/discourse/shared/standalone/log/var-log
      guest: /var/log

## Plugins go here
## see https://meta.discourse.org/t/19157 for details
hooks:
  after_code:
    - exec:
        cd: $home/plugins
        cmd:
          - git clone https://github.com/discourse/docker_manager.git
          - git clone https://github.com/discourse/discourse-data-explorer.git
          - git clone https://github.com/discourse/discourse-voting.git
          - git clone https://github.com/discourse/wp-discourse.git
          - git clone https://github.com/discourse/discourse-spoiler-alert.git
          - git clone https://github.com/discourse/discourse-bbcode-color.git 
          - git clone https://github.com/discourse/discourse-calendar
          - git clone https://github.com/discourse/discourse-checklist
          - git clone https://github.com/discourse/discourse-reactions
          - git clone https://github.com/discourse/discourse-policy

## Any custom commands to run after building
run:
  - exec: echo "Beginning of custom commands"
  ## If you want to set the 'From' email address for your first registration, uncomment and change:
  ## After getting the first signup email, re-comment the line. It only needs to run once.
  #- exec: rails r "SiteSetting.notification_email='info@unconfigured.discourse.org'"
  - exec: echo "End of custom commands"
10

I updgraded docker and tried to re-run:

wordsmith@digitalwind-ubuntu-20:/var/discourse/containers$ docker --version
Docker version 27.5.1, build 9f9e405
wordsmith@digitalwind-ubuntu-20:/var/discourse/containers$ cd …
wordsmith@digitalwind-ubuntu-20:/var/discourse$ sudo ./launcher app rebuild
x86_64 arch detected.
ERROR: containers/rebuild.yml does not exist or is not readable.

Available configs ( app )
wordsmith@digitalwind-ubuntu-20:/var/discourse$
12

You must finish upgraiding in a console doing ./launcher rebuild app.

Out there is new stable release

13

That is what I did above?

1 Like
14

Can anyone help me here? My installation is still down and I am at a loss.

15

I’m a newbie but I’ll try my best to help you. I read in an earlier post you said

Are you in the right directory?
Is the above the most recent attempt and current error message?

16

Did you restart your container as suggested?

I would restart it, make a backup, and move to a new vm with an up to date os.

1 Like
17

I think that should be sudo ./launcher rebuild app

4 Likes
19

A GUI update won’t upgrade your database, or to put it a different way: you must, on this occasion upgrade from the command line to pick up further updates.

2 Likes
20

That’s not going to happen.
I may have misread or am misremembering, but I’m pretty sure that your docker and os are no longer supported, so just stop upgrading everything.

Just don’t upgrade. I helped someone recently who hadn’t upgraded in six years. They were still running Ubuntu 14.04.

1 Like
23

And yet it is an upgrade. And you can’t do it via GUI because of database upgrading :man_shrugging:

1 Like
24

It will be exactly the same. You will still need to upgrade docker. You will still need to upgrade the database. There is a stable release available now if I remember your version correctly.

1 Like
26

That is maybe too bold of a statement. With the skills some have here could likely fix the issue. Though always remember the first commandment with upgrading. Always make a backup

27

They do. In your App yml you can set Discourse to Stable Branch. However you still need to once in a while login to the server and upgrade.

This is the benefit of paying for a hosted plan is that you don’t need to worry about crashes as the Host plan provider looks after it

Self+Hosted has benefits but you may run into trouble and if you are unable to fix it. May need to pay someone to correct the issue(s). This is why making backups is important. Especially with a major update. Your VPs provider also has an option to make a snapshot of your server that you can restore if an update via say command line fails “Spectacularly”.

In my journey now if just over 7 years I have had to learn many new things. Most recently spun up a new server and used Rsync to move my forum to a fully updated current Ubuntu server. Old one was iirc was Ubuntu 20.x. Now running the latest iirc is Ubuntu 24.x. had I not taken the time to learn and complete this U would have eventually had a “Spectacular” failed upgrade.

1 Like
30

Why do you think so?
Beta does not mean “not ready for use in production.” Stable means fewer changes, which results in fewer new bugs. However, it also means fewer bug fixes and new features, as only security bug fixes are backported.

1 Like
31

You have been lucky for several years. As Jay mentioned your Docker is outdated and requires a hard update vs a GUI soft update.

The creators of Discourse even state explicitly that you need to once and awhile so command line updates. I have in my over 7 years of use in the early days have observed the web GUI specifically tell me that I need to do a command line update. And you also need to at times perform an OS upgrade(or create a new server to refresh OS and move Discourse to it)

This is one of the cons of Self Hosted is you as the Self Hosted have to maintain the backend and fix things when needed or goto Marketplace and pay someone who has the skills needed

1 Like