Postgres error upgrading from 3.3 to 3.4 caused by locale issues

Hi

Happy New Year to all. I am trying to update from 3.3 b10 to 3.4 b3. I did the following:

cd /var/discourse/git
git remote update
git pull --rebase origin main

Then I did the standard ./launcher rebuild app.

I am running Ubuntu 20.04 with Docker 20.10.7. I understand the Docker version is deprecated but the issues seem to come from the Postgre container.

I’m getting the following locale error:

Stopping old container
+ /usr/bin/docker stop -t 600 app
app
2.0.20241203-0251: Pulling from discourse/base
Digest: sha256:d5d4e6d361d56745e37f2609db58a6970cc9b3fcd26d0b2dcba23d0cb581f067
Status: Image is up to date for discourse/base:2.0.20241203-0251
docker.io/discourse/base:2.0.20241203-0251
/usr/local/lib/ruby/gems/3.3.0/gems/pups-1.2.1/lib/pups.rb
/usr/local/bin/pups --stdin
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_GB.UTF-8)
I, [2025-01-02T00:29:38.387006 #1]  INFO -- : Reading from stdin
I, [2025-01-02T00:29:38.394543 #1]  INFO -- : File > /etc/service/postgres/run  chmod: +x  chown: 
I, [2025-01-02T00:29:38.397263 #1]  INFO -- : File > /etc/service/postgres/log/run  chmod: +x  chown: 
I, [2025-01-02T00:29:38.400022 #1]  INFO -- : File > /etc/runit/3.d/99-postgres  chmod: +x  chown: 
I, [2025-01-02T00:29:38.402670 #1]  INFO -- : File > /root/install_postgres  chmod: +x  chown: 
I, [2025-01-02T00:29:38.405305 #1]  INFO -- : File > /root/upgrade_postgres  chmod: +x  chown: 
I, [2025-01-02T00:29:38.405637 #1]  INFO -- : Replacing data_directory = '/var/lib/postgresql/13/main' with data_directory = '/shared/postgres_data' in /etc/postgresql/13/main/postgresql.conf
I, [2025-01-02T00:29:38.406337 #1]  INFO -- : Replacing (?-mix:#?listen_addresses *=.*) with listen_addresses = '*' in /etc/postgresql/13/main/postgresql.conf
I, [2025-01-02T00:29:38.470425 #1]  INFO -- : Replacing (?-mix:#?synchronous_commit *=.*) with synchronous_commit = $db_synchronous_commit in /etc/postgresql/13/main/postgresql.conf
I, [2025-01-02T00:29:38.501384 #1]  INFO -- : Replacing (?-mix:#?shared_buffers *=.*) with shared_buffers = $db_shared_buffers in /etc/postgresql/13/main/postgresql.conf
I, [2025-01-02T00:29:38.501924 #1]  INFO -- : Replacing (?-mix:#?work_mem *=.*) with work_mem = $db_work_mem in /etc/postgresql/13/main/postgresql.conf
I, [2025-01-02T00:29:38.502432 #1]  INFO -- : Replacing (?-mix:#?default_text_search_config *=.*) with default_text_search_config = '$db_default_text_search_config' in /etc/postgresql/13/main/postgresql.conf
I, [2025-01-02T00:29:38.502928 #1]  INFO -- : Replacing (?-mix:#?checkpoint_segments *=.*) with checkpoint_segments = $db_checkpoint_segments in /etc/postgresql/13/main/postgresql.conf
I, [2025-01-02T00:29:38.503401 #1]  INFO -- : Replacing (?-mix:#?logging_collector *=.*) with logging_collector = $db_logging_collector in /etc/postgresql/13/main/postgresql.conf
I, [2025-01-02T00:29:38.503882 #1]  INFO -- : Replacing (?-mix:#?log_min_duration_statement *=.*) with log_min_duration_statement = $db_log_min_duration_statement in /etc/postgresql/13/main/postgresql.conf
I, [2025-01-02T00:29:38.504384 #1]  INFO -- : Replacing (?-mix:^#local +replication +postgres +peer$) with local replication postgres  peer in /etc/postgresql/13/main/pg_hba.conf
I, [2025-01-02T00:29:38.504883 #1]  INFO -- : Replacing (?-mix:^host.*all.*all.*127.*$) with host all all 0.0.0.0/0 md5 in /etc/postgresql/13/main/pg_hba.conf
I, [2025-01-02T00:29:38.505325 #1]  INFO -- : Replacing (?-mix:^host.*all.*all.*::1\/128.*$) with host all all ::/0 md5 in /etc/postgresql/13/main/pg_hba.conf
I, [2025-01-02T00:29:38.505810 #1]  INFO -- : > if [ -f /root/install_postgres ]; then
  /root/install_postgres && rm -f /root/install_postgres
elif [ -e /shared/postgres_run/.s.PGSQL.5432 ]; then
  socat /dev/null UNIX-CONNECT:/shared/postgres_run/.s.PGSQL.5432 || exit 0 && echo postgres already running stop container ; exit 1
fi

/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_GB.UTF-8)
failed to set locale!
[error] character map file `UTF-8' not found: No such file or directory
failed to set locale!
[error] default character map file `ANSI_X3.4-1968' not found: No such file or directory
failed to set locale!
[error] character map file `UTF-8' not found: No such file or directory
failed to set locale!
[error] default character map file `ANSI_X3.4-1968' not found: No such file or directory
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
	LANGUAGE = "en_GB.UTF-8",
	LC_ALL = "en_GB.UTF-8",
	LANG = "en_GB.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
I, [2025-01-02T00:29:41.001456 #1]  INFO -- : Generating locales (this might take a while)...
  en_GB.UTF-8... done
  en_US.UTF-8... done
Generation complete.

I, [2025-01-02T00:29:41.002017 #1]  INFO -- : > HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main
I, [2025-01-02T00:29:41.003269 #1]  INFO -- : Terminating async processes
2025-01-02 00:29:41.050 UTC [38] LOG:  starting PostgreSQL 13.18 (Debian 13.18-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
2025-01-02 00:29:41.051 UTC [38] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2025-01-02 00:29:41.051 UTC [38] LOG:  listening on IPv6 address "::", port 5432
2025-01-02 00:29:41.053 UTC [38] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2025-01-02 00:29:41.058 UTC [39] LOG:  database system was shut down at 2025-01-02 00:29:20 UTC
2025-01-02 00:29:41.297 UTC [38] LOG:  database system is ready to accept connections
2025-01-02 00:30:01.915 UTC [46] FATAL:  database locale is incompatible with operating system
2025-01-02 00:30:01.915 UTC [46] DETAIL:  The database was initialized with LC_COLLATE "en_GB.UTF-8",  which is not recognized by setlocale().
2025-01-02 00:30:01.915 UTC [46] HINT:  Recreate the database with another locale or install the missing locale.
2025-01-02 00:30:21.343 UTC [47] FATAL:  database locale is incompatible with operating system
2025-01-02 00:30:21.343 UTC [47] DETAIL:  The database was initialized with LC_COLLATE "en_GB.UTF-8",  which is not recognized by setlocale().
2025-01-02 00:30:21.343 UTC [47] HINT:  Recreate the database with another locale or install the missing locale.
2025-01-02 00:30:41.344 UTC [48] FATAL:  database locale is incompatible with operating system
2025-01-02 00:30:41.344 UTC [48] DETAIL:  The database was initialized with LC_COLLATE "en_GB.UTF-8",  which is not recognized by setlocale().
2025-01-02 00:30:41.344 UTC [48] HINT:  Recreate the database with another locale or install the missing locale.
2025-01-02 00:31:01.341 UTC [49] FATAL:  database locale is incompatible with operating system
2025-01-02 00:31:01.341 UTC [49] DETAIL:  The database was initialized with LC_COLLATE "en_GB.UTF-8",  which is not recognized by setlocale().
2025-01-02 00:31:01.341 UTC [49] HINT:  Recreate the database with another locale or install the missing locale.
2025-01-02 00:31:21.418 UTC [50] FATAL:  database locale is incompatible with operating system
2025-01-02 00:31:21.418 UTC [50] DETAIL:  The database was initialized with LC_COLLATE "en_GB.UTF-8",  which is not recognized by setlocale().
2025-01-02 00:31:21.418 UTC [50] HINT:  Recreate the database with another locale or install the missing locale.
2025-01-02 00:31:41.360 UTC [51] FATAL:  database locale is incompatible with operating system
2025-01-02 00:31:41.360 UTC [51] DETAIL:  The database was initialized with LC_COLLATE "en_GB.UTF-8",  which is not recognized by setlocale().
2025-01-02 00:31:41.360 UTC [51] HINT:  Recreate the database with another locale or install the missing locale.
2025-01-02 00:32:01.358 UTC [52] FATAL:  database locale is incompatible with operating system
2025-01-02 00:32:01.358 UTC [52] DETAIL:  The database was initialized with LC_COLLATE "en_GB.UTF-8",  which is not recognized by setlocale().
2025-01-02 00:32:01.358 UTC [52] HINT:  Recreate the database with another locale or install the missing locale.
2025-01-02 00:32:21.379 UTC [53] FATAL:  database locale is incompatible with operating system
2025-01-02 00:32:21.379 UTC [53] DETAIL:  The database was initialized with LC_COLLATE "en_GB.UTF-8",  which is not recognized by setlocale().
2025-01-02 00:32:21.379 UTC [53] HINT:  Recreate the database with another locale or install the missing locale.
2025-01-02 00:32:41.380 UTC [54] FATAL:  database locale is incompatible with operating system
2025-01-02 00:32:41.380 UTC [54] DETAIL:  The database was initialized with LC_COLLATE "en_GB.UTF-8",  which is not recognized by setlocale().
2025-01-02 00:32:41.380 UTC [54] HINT:  Recreate the database with another locale or install the missing locale.
2025-01-02 00:33:01.373 UTC [55] FATAL:  database locale is incompatible with operating system
2025-01-02 00:33:01.373 UTC [55] DETAIL:  The database was initialized with LC_COLLATE "en_GB.UTF-8",  which is not recognized by setlocale().
2025-01-02 00:33:01.373 UTC [55] HINT:  Recreate the database with another locale or install the missing locale.
2025-01-02 00:33:21.391 UTC [56] FATAL:  database locale is incompatible with operating system
2025-01-02 00:33:21.391 UTC [56] DETAIL:  The database was initialized with LC_COLLATE "en_GB.UTF-8",  which is not recognized by setlocale().
2025-01-02 00:33:21.391 UTC [56] HINT:  Recreate the database with another locale or install the missing locale.
2025-01-02 00:33:41.391 UTC [57] FATAL:  database locale is incompatible with operating system
2025-01-02 00:33:41.391 UTC [57] DETAIL:  The database was initialized with LC_COLLATE "en_GB.UTF-8",  which is not recognized by setlocale().
2025-01-02 00:33:41.391 UTC [57] HINT:  Recreate the database with another locale or install the missing locale.
2025-01-02 00:34:01.391 UTC [58] FATAL:  database locale is incompatible with operating system
2025-01-02 00:34:01.391 UTC [58] DETAIL:  The database was initialized with LC_COLLATE "en_GB.UTF-8",  which is not recognized by setlocale().
2025-01-02 00:34:01.391 UTC [58] HINT:  Recreate the database with another locale or install the missing locale.
2025-01-02 00:34:21.391 UTC [59] FATAL:  database locale is incompatible with operating system
2025-01-02 00:34:21.391 UTC [59] DETAIL:  The database was initialized with LC_COLLATE "en_GB.UTF-8",  which is not recognized by setlocale().
2025-01-02 00:34:21.391 UTC [59] HINT:  Recreate the database with another locale or install the missing locale.
2025-01-02 00:34:41.409 UTC [60] FATAL:  database locale is incompatible with operating system
2025-01-02 00:34:41.409 UTC [60] DETAIL:  The database was initialized with LC_COLLATE "en_GB.UTF-8",  which is not recognized by setlocale().
2025-01-02 00:34:41.409 UTC [60] HINT:  Recreate the database with another locale or install the missing locale.
2025-01-02 00:35:01.405 UTC [61] FATAL:  database locale is incompatible with operating system
2025-01-02 00:35:01.405 UTC [61] DETAIL:  The database was initialized with LC_COLLATE "en_GB.UTF-8",  which is not recognized by setlocale().
2025-01-02 00:35:01.405 UTC [61] HINT:  Recreate the database with another locale or install the missing locale.
2025-01-02 00:35:21.392 UTC [62] FATAL:  database locale is incompatible with operating system
2025-01-02 00:35:21.392 UTC [62] DETAIL:  The database was initialized with LC_COLLATE "en_GB.UTF-8",  which is not recognized by setlocale().
2025-01-02 00:35:21.392 UTC [62] HINT:  Recreate the database with another locale or install the missing locale.
2025-01-02 00:35:41.423 UTC [63] FATAL:  database locale is incompatible with operating system
2025-01-02 00:35:41.423 UTC [63] DETAIL:  The database was initialized with LC_COLLATE "en_GB.UTF-8",  which is not recognized by setlocale().
2025-01-02 00:35:41.423 UTC [63] HINT:  Recreate the database with another locale or install the missing locale.
2025-01-02 00:36:01.421 UTC [64] FATAL:  database locale is incompatible with operating system
2025-01-02 00:36:01.421 UTC [64] DETAIL:  The database was initialized with LC_COLLATE "en_GB.UTF-8",  which is not recognized by setlocale().
2025-01-02 00:36:01.421 UTC [64] HINT:  Recreate the database with another locale or install the missing locale.
2025-01-02 00:36:21.392 UTC [65] FATAL:  database locale is incompatible with operating system
2025-01-02 00:36:21.392 UTC [65] DETAIL:  The database was initialized with LC_COLLATE "en_GB.UTF-8",  which is not recognized by setlocale().
2025-01-02 00:36:21.392 UTC [65] HINT:  Recreate the database with another locale or install the missing locale.
2025-01-02 00:36:41.434 UTC [66] FATAL:  database locale is incompatible with operating system
2025-01-02 00:36:41.434 UTC [66] DETAIL:  The database was initialized with LC_COLLATE "en_GB.UTF-8",  which is not recognized by setlocale().
2025-01-02 00:36:41.434 UTC [66] HINT:  Recreate the database with another locale or install the missing locale.
2025-01-02 00:37:01.437 UTC [67] FATAL:  database locale is incompatible with operating system
2025-01-02 00:37:01.437 UTC [67] DETAIL:  The database was initialized with LC_COLLATE "en_GB.UTF-8",  which is not recognized by setlocale().
2025-01-02 00:37:01.437 UTC [67] HINT:  Recreate the database with another locale or install the missing locale.
2025-01-02 00:37:21.391 UTC [68] FATAL:  database locale is incompatible with operating system
2025-01-02 00:37:21.391 UTC [68] DETAIL:  The database was initialized with LC_COLLATE "en_GB.UTF-8",  which is not recognized by setlocale().
2025-01-02 00:37:21.391 UTC [68] HINT:  Recreate the database with another locale or install the missing locale.
2025-01-02 00:37:41.533 UTC [69] FATAL:  database locale is incompatible with operating system
2025-01-02 00:37:41.533 UTC [69] DETAIL:  The database was initialized with LC_COLLATE "en_GB.UTF-8",  which is not recognized by setlocale().
2025-01-02 00:37:41.533 UTC [69] HINT:  Recreate the database with another locale or install the missing locale.
2025-01-02 00:38:01.454 UTC [70] FATAL:  database locale is incompatible with operating system
2025-01-02 00:38:01.454 UTC [70] DETAIL:  The database was initialized with LC_COLLATE "en_GB.UTF-8",  which is not recognized by setlocale().
2025-01-02 00:38:01.454 UTC [70] HINT:  Recreate the database with another locale or install the missing locale.
2025-01-02 00:38:21.491 UTC [71] FATAL:  database locale is incompatible with operating system
2025-01-02 00:38:21.491 UTC [71] DETAIL:  The database was initialized with LC_COLLATE "en_GB.UTF-8",  which is not recognized by setlocale().
2025-01-02 00:38:21.491 UTC [71] HINT:  Recreate the database with another locale or install the missing locale.
lo^C^C^C^C^C^C2025-01-02 00:38:41.553 UTC [72] FATAL:  database locale is incompatible with operating system
2025-01-02 00:38:41.553 UTC [72] DETAIL:  The database was initialized with LC_COLLATE "en_GB.UTF-8",  which is not recognized by setlocale().
2025-01-02 00:38:41.553 UTC [72] HINT:  Recreate the database with another locale or install the missing locale.
2025-01-02 00:39:01.470 UTC [73] FATAL:  database locale is incompatible with operating system
2025-01-02 00:39:01.470 UTC [73] DETAIL:  The database was initialized with LC_COLLATE "en_GB.UTF-8",  which is not recognized by setlocale().
2025-01-02 00:39:01.470 UTC [73] HINT:  Recreate the database with another locale or install the missing locale.
2025-01-02 00:39:21.511 UTC [74] FATAL:  database locale is incompatible with operating system
2025-01-02 00:39:21.511 UTC [74] DETAIL:  The database was initialized with LC_COLLATE "en_GB.UTF-8",  which is not recognized by setlocale().
2025-01-02 00:39:21.511 UTC [74] HINT:  Recreate the database with another locale or install the missing locale.

I can’t see how this can be a host issue, but to rule out such an issue, I verified the locales are good on the host system:

root@discourse:/var/discourse# locale -a
C
C.UTF-8
en_AG
en_AG.utf8
en_AU.utf8
en_BW.utf8
en_CA.utf8
en_DK.utf8
en_GB.utf8
en_HK.utf8
en_IE.utf8
en_IL
en_IL.utf8
en_IN
en_IN.utf8
en_NG
en_NG.utf8
en_NZ.utf8
en_PH.utf8
en_SG.utf8
en_US.utf8
en_ZA.utf8
en_ZM
en_ZM.utf8
en_ZW.utf8
POSIX

I can see they are present.
Re-running ./launcher rebuild app does not help and I get the same issue. It’s odd because I can see that the locales are being generated in the container, but are clearly not enough to satisfy Postgre.

Any clues?

Maybe related to your change @featheredtoast ?

2 Likes

If you add the following to your container definition file it should resolve the issue.

env:
  LANG: en_GB.UTF-8
5 Likes

Thanks for the prompt responses.

But I do see:

I, [2025-01-02T00:29:41.001456 #1]  INFO -- : Generating locales (this might take a while)...
  en_GB.UTF-8... done
  en_US.UTF-8... done

FWIW, I have dist-upgraded the Ubuntu 20.04 system fully so that Docker version is now:

Docker version 24.0.7, build 24.0.7-0ubuntu2~20.04.1

My discourse_docker repo sits at:

commit 0d0e1eb1de3dbffc432dff6bf03ebcf64d84150b (HEAD -> main, origin/main, origin/HEAD)
Author: Alan Guo Xiang Tan <gxtan1990@gmail.com>
Date:   Fri Dec 27 12:51:33 2024 +0800

    Clean up firefox tar file in `discourse/discourse_test` image (#904)

I performed a rebuild after updating Docker. This worked. As a result: would you still recommend I set this variable @tgxworld?

Cheers

Sam

1 Like

Yes please continue to set the env.

2 Likes

I think I have an issue in similar manner :confused:

I have updated the forum via admin interface and it did not start. So I tried to rebuild from console.

my env is set to Polish:

env:
  LANG: pl_PL.UTF-8

Changing this to anything does not help.

The locales are there:

ubuntu@vps:/var/discourse/containers$ locale -a
C
C.UTF-8
POSIX
en_US.utf8
pl_PL.utf8

This is what has been there for the past 2 years…

But I get this fail on rebuild:

failed to set locale!
[error] character map file `UTF-8' not found: No such file or directory
failed to set locale!
[error] default character map file `ANSI_X3.4-1968' not found: No such file or directory
failed to set locale!
[error] character map file `UTF-8' not found: No such file or directory
failed to set locale!
[error] default character map file `ANSI_X3.4-1968' not found: No such file or directory
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
        LANGUAGE = "en_US.UTF-8",
        LC_ALL = "en_US.UTF-8",
        LANG = "pl_PL.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
I, [2025-01-06T10:08:48.737950 #1]  INFO -- : Generating locales (this might take a while)...
  en_US.UTF-8... done
  pl_PL.UTF-8... done
Generation complete.

And then it starts looping with these errors:

2025-01-06 10:09:48.880 UTC [47] FATAL:  database locale is incompatible with operating system
2025-01-06 10:09:48.880 UTC [47] DETAIL:  The database was initialized with LC_COLLATE "en_US.UTF-8",  which is not recognized by setlocale().
2025-01-06 10:09:48.880 UTC [47] HINT:  Recreate the database with another locale or install the missing locale.

What’s your Docker version? For me that seemed to solve things.

2 Likes

WARNING: Docker version 20.10.8 deprecated, recommend upgrade to 24.0.7 or newer.
so I guess thats that ;]

ubuntu@vps-d6490d26:/var/discourse/containers$ docker --version
Docker version 20.10.8, build 3967b7d

apt-get reports that this is the newest wersion. I’m running Ubuntu 20.10

3 Likes

Yeah, upgrading Docker to 27.4.1 solved this, although I did have to first upgrade my Ubuntu from 20.10 to 22.04, but thats on me ;]

6 Likes