Upgrade failed. Database stopped. (multisite install)

Installatie
1

I logged in. Haven’t updated in 11 months. Started with trying to upgrade the Docker Manager. It never seemed to complete. Next day the admin page was pretty much borked. I tried to do the ./launcher rebuild app. I get the

2025-10-30 20:04:52.144 UTC [1892] discourse@c_discourse ERROR:  must be owner of extension vector
2025-10-30 20:04:52.144 UTC [1892] discourse@c_discourse STATEMENT:  ALTER EXTENSION vector UPDATE TO ‘0.7.0’;

But the database is stopped so the fixes I saw don’t work.

2025-10-31 00:27:40.021 UTC [662] FATAL:  database files are incompatible with server
2025-10-31 00:27:40.021 UTC [662] DETAIL:  The data directory was initialized by PostgreSQL version 15, which is not compatible with this version 13.16 (Debian 13.16-1.pgdg120+1).

But

root@main-app:/var/lib/postgresql# pg_config --version
PostgreSQL 16.4 (Debian 16.4-1.pgdg120+1)

EDIT: It looks like the DB is still in 13 format?

root@main-app:/var/lib/postgresql# ls
13  take-database-backup
2

If you haven’t seen the topic PostgreSQL 15 update - Announcements you might start there. It discusses a number of issues and recommendations.

2 likes
3

Thanks. Too much to really grok it all but nothing seems like my problem. :frowning: The database seems to run during the rebuild process which is the strange part. At least from what I can tell in the stuff that scrolls by.

4

That’s mostly never going to work after 11 months.

Your definitely need to do the postgres 15 upgrade, and that’s part of what’s wrong. It’s possible that that can switch to the postgres 13 and get things going again before doing the upgrade.

1 like
5

Is this a standard install ?

6

Not if multisite is still considered unsupported.

The issue seems to be when I installed it there was no user/password connected to the individual databases. That seems to change. That’s where the problem lies. I’ve tried to tinker a little with it by adding a user and then a user with a blank password but that didn’t help

My config versus the new sample config:

secondsite:
adapter: postgresql
database: b_discourse

pool: 25
timeout: 5000
db_id: 2
host_names:

mlp:
adapter: postgresql
database: discourse_mlp
username: discourse_mlp
password: applejack
host: dbhost
pool: 5
timeout: 5000

host_names:

7

Does this mean that the database actually starts during the install process?

2025-10-31 15:14:30.984 UTC [45] LOG:  starting PostgreSQL 15.14 (Debian 15.14-1.pgdg12+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0
-14+deb12u1) 12.2.0, 64-bit
2025-10-31 15:14:30.984 UTC [45] LOG:  listening on IPv4 address “0.0.0.0”, port 5432
2025-10-31 15:14:30.984 UTC [45] LOG:  listening on IPv6 address “::”, port 5432
2025-10-31 15:14:31.055 UTC [45] LOG:  listening on Unix socket “/var/run/postgresql/.s.PGSQL.5432”
2025-10-31 15:14:31.135 UTC [56] LOG:  database system was shut down at 2025-10-31 02:08:49 UTC
2025-10-31 15:14:31.199 UTC [45] LOG:  database system is ready to accept connections
I, [2025-10-31T15:14:35.760150 #1]  INFO – :
I, [2025-10-31T15:14:35.760557 #1]  INFO – : > /usr/local/bin/create_db
2025-10-31 15:14:35.912 UTC [63] postgres@postgres ERROR:  database “discourse” already exists
2025-10-31 15:14:35.912 UTC [63] postgres@postgres STATEMENT:  CREATE DATABASE discourse;
createdb: error: database creation failed: ERROR:  database “discourse” already exists
2025-10-31 15:14:36.060 UTC [66] postgres@discourse ERROR:  role “discourse” already exists
2025-10-31 15:14:36.060 UTC [66] postgres@discourse STATEMENT:  create user discourse;
ERROR:  role “discourse” already exists
NOTICE:  extension “hstore” already exists, skipping
NOTICE:  extension “pg_trgm” already exists, skipping
NOTICE:  extension “vector” already exists, skipping
NOTICE:  version “0.8.1” of extension “vector” is already installed
NOTICE:  extension “hstore” already exists, skipping
NOTICE:  extension “pg_trgm” already exists, skipping
NOTICE:  extension “vector” already exists, skipping
NOTICE:  version “0.8.1” of extension “vector” is already installed
I, [2025-10-31T15:14:37.265035 #1]  INFO – : GRANT
ALTER SCHEMA
CREATE EXTENSION
CREATE EXTENSION
CREATE EXTENSION
ALTER EXTENSION
CREATE EXTENSION
CREATE EXTENSION
CREATE EXTENSION
ALTER EXTENSION
UPDATE 0

I, [2025-10-31T15:14:37.265325 #1]  INFO – : > echo postgres installed!
I, [2025-10-31T15:14:37.270967 #1]  INFO – : postgres installed!

I, [2025-10-31T15:14:37.271578 #1]  INFO – : > sudo -u postgres createdb b_discourse || exit 0
2025-10-31 15:14:37.380 UTC [104] postgres@postgres ERROR:  database “b_discourse” already exists
2025-10-31 15:14:37.380 UTC [104] postgres@postgres STATEMENT:  CREATE DATABASE b_discourse;
createdb: error: database creation failed: ERROR:  database “b_discourse” already exists
I, [2025-10-31T15:14:37.385149 #1]  INFO – :
I, [2025-10-31T15:14:37.385968 #1]  INFO – : > sudo -u postgres psql b_discourse
I, [2025-10-31T15:14:37.390476 #1]  INFO – : grant all privileges on database b_discourse to discourse;

I, [2025-10-31T15:14:37.515870 #1]  INFO – : > sudo -u postgres createdb c_discourse || exit 0
2025-10-31 15:14:37.625 UTC [111] postgres@postgres ERROR:  database “c_discourse” already exists
2025-10-31 15:14:37.625 UTC [111] postgres@postgres STATEMENT:  CREATE DATABASE c_discourse;
createdb: error: database creation failed: ERROR:  database “c_discourse” already exists
I, [2025-10-31T15:14:37.629387 #1]  INFO – :
I, [2025-10-31T15:14:37.630145 #1]  INFO – : > sudo -u postgres psql c_discourse
I, [2025-10-31T15:14:37.634672 #1]  INFO – : grant all privileges on database c_discourse to discourse;

I, [2025-10-31T15:14:37.758171 #1]  INFO – : > /bin/bash -c ‘sudo -u postgres psql b_discourse <<< “alter schema public owner to discourse;”’
I, [2025-10-31T15:14:37.876557 #1]  INFO – : ALTER SCHEMA

I, [2025-10-31T15:14:37.877104 #1]  INFO – : > /bin/bash -c ‘sudo -u postgres psql b_discourse <<< “create extension if not exists hstore;”’
NOTICE:  extension “hstore” already exists, skipping
I, [2025-10-31T15:14:37.997465 #1]  INFO – : CREATE EXTENSION

I, [2025-10-31T15:14:37.998029 #1]  INFO – : > /bin/bash -c ‘sudo -u postgres psql b_discourse <<< “create extension if not exists pg_trgm;”’
NOTICE:  extension “pg_trgm” already exists, skipping
I, [2025-10-31T15:14:38.117879 #1]  INFO – : CREATE EXTENSION

2025-10-31 15:14:30.984 UTC [45] LOG:  starting PostgreSQL 15.14 (Debian 15.14-1.pgdg12+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0
-14+deb12u1) 12.2.0, 64-bit
2025-10-31 15:14:30.984 UTC [45] LOG:  listening on IPv4 address “0.0.0.0”, port 5432
2025-10-31 15:14:30.984 UTC [45] LOG:  listening on IPv6 address “::”, port 5432
2025-10-31 15:14:31.055 UTC [45] LOG:  listening on Unix socket “/var/run/postgresql/.s.PGSQL.5432”
2025-10-31 15:14:31.135 UTC [56] LOG:  database system was shut down at 2025-10-31 02:08:49 UTC
2025-10-31 15:14:31.199 UTC [45] LOG:  database system is ready to accept connections
I, [2025-10-31T15:14:35.760150 #1]  INFO – :
I, [2025-10-31T15:14:35.760557 #1]  INFO – : > /usr/local/bin/create_db
2025-10-31 15:14:35.912 UTC [63] postgres@postgres ERROR:  database “discourse” already exists
2025-10-31 15:14:35.912 UTC [63] postgres@postgres STATEMENT:  CREATE DATABASE discourse;
createdb: error: database creation failed: ERROR:  database “discourse” already exists
2025-10-31 15:14:36.060 UTC [66] postgres@discourse ERROR:  role “discourse” already exists
2025-10-31 15:14:36.060 UTC [66] postgres@discourse STATEMENT:  create user discourse;
ERROR:  role “discourse” already exists
NOTICE:  extension “hstore” already exists, skipping
NOTICE:  extension “pg_trgm” already exists, skipping
NOTICE:  extension “vector” already exists, skipping
NOTICE:  version “0.8.1” of extension “vector” is already installed
NOTICE:  extension “hstore” already exists, skipping
NOTICE:  extension “pg_trgm” already exists, skipping
NOTICE:  extension “vector” already exists, skipping
NOTICE:  version “0.8.1” of extension “vector” is already installed
I, [2025-10-31T15:14:37.265035 #1]  INFO – : GRANT
ALTER SCHEMA
CREATE EXTENSION
CREATE EXTENSION
CREATE EXTENSION
ALTER EXTENSION
CREATE EXTENSION
CREATE EXTENSION
CREATE EXTENSION
ALTER EXTENSION
UPDATE 0

I, [2025-10-31T15:14:37.265325 #1]  INFO – : > echo postgres installed!
I, [2025-10-31T15:14:37.270967 #1]  INFO – : postgres installed!

I, [2025-10-31T15:14:37.271578 #1]  INFO – : > sudo -u postgres createdb b_discourse || exit 0
2025-10-31 15:14:37.380 UTC [104] postgres@postgres ERROR:  database “b_discourse” already exists
2025-10-31 15:14:37.380 UTC [104] postgres@postgres STATEMENT:  CREATE DATABASE b_discourse;
createdb: error: database creation failed: ERROR:  database “b_discourse” already exists
I, [2025-10-31T15:14:37.385149 #1]  INFO – :
I, [2025-10-31T15:14:37.385968 #1]  INFO – : > sudo -u postgres psql b_discourse
I, [2025-10-31T15:14:37.390476 #1]  INFO – : grant all privileges on database b_discourse to discourse;

I, [2025-10-31T15:14:37.515870 #1]  INFO – : > sudo -u postgres createdb c_discourse || exit 0
2025-10-31 15:14:37.625 UTC [111] postgres@postgres ERROR:  database “c_discourse” already exists
2025-10-31 15:14:37.625 UTC [111] postgres@postgres STATEMENT:  CREATE DATABASE c_discourse;
createdb: error: database creation failed: ERROR:  database “c_discourse” already exists
I, [2025-10-31T15:14:37.629387 #1]  INFO – :
I, [2025-10-31T15:14:37.630145 #1]  INFO – : > sudo -u postgres psql c_discourse
I, [2025-10-31T15:14:37.634672 #1]  INFO – : grant all privileges on database c_discourse to discourse;

I, [2025-10-31T15:14:37.758171 #1]  INFO – : > /bin/bash -c ‘sudo -u postgres psql b_discourse <<< “alter schema public owner to discourse;”’
I, [2025-10-31T15:14:37.876557 #1]  INFO – : ALTER SCHEMA

I, [2025-10-31T15:14:37.877104 #1]  INFO – : > /bin/bash -c ‘sudo -u postgres psql b_discourse <<< “create extension if not exists hstore;”’
NOTICE:  extension “hstore” already exists, skipping
I, [2025-10-31T15:14:37.997465 #1]  INFO – : CREATE EXTENSION

I, [2025-10-31T15:14:37.998029 #1]  INFO – : > /bin/bash -c ‘sudo -u postgres psql b_discourse <<< “create extension if not exists pg_trgm;”’
NOTICE:  extension “pg_trgm” already exists, skipping
I, [2025-10-31T15:14:38.117879 #1]  INFO – : CREATE EXTENSION
8

Pgvector extension is either out of date or not set up on one of your databases. I have encountered this on a few installs a few months ago. It is very simple to resolve, You just need to update it.

enter the db container, psql as discourse (or whatever is your db owner) and then choose each database and run

ALTER EXTENSION vector UPDATE;

Ps: it seems as if your setup is a bit different, the above will only work if you have a separate data container.

1 like
9

Looks like the extension owner for the c_discourse database isn’t the discourse user - you’ll need to check to see who is the owner, e.g.:

discourse_development=# select extname, extowner, usename, extversion from pg_extension JOIN pg_user on pg_extension.extowner = pg_user.usesysid;
 extname  | extowner | usename  | extversion 
----------+----------+----------+------------
 vector   |    16385 | michael  | 0.8.0
 unaccent |       10 | postgres | 1.1
 pg_trgm  |       10 | postgres | 1.6
 hstore   |       10 | postgres | 1.8
 plpgsql  |       10 | postgres | 1.0

and run the extension update command as that user or a database superuser.

2 likes
10

Any help on how I start the database to do this? That’s really the sticking point.

EDIT: The only postgres files I find are of the /13. So I’m kind of in a state of limbo. Seems the database program upgraded but the database files didn’t. I’ve done some searches but nothing I’m comfortable with short of trying to copy the files out and playing with them.

2025-10-31 00:27:40.021 UTC [662] FATAL: database files are incompatible with server
2025-10-31 00:27:40.021 UTC [662] DETAIL: The data directory was initialized by PostgreSQL version 15, which is not compatible with this version 13.16 (Debian 13.16-1.pgdg120+1).

11

You should be able to edit the app.yml and use the postgres 13 template and rebuild, as described in the PostgreSQL 15 update topic.

Then you can rebuild. Then you can start the container. Then you can enter the container. Then you can do the pgvectkr stuff.

12

Well, that purged 15 and installed 13, but strangely, same error. It aborted much closer to the beginning of the process. :frowning:

2025-11-01 15:30:26.522 UTC [2547] FATAL: database files are incompatible with server
2025-11-01 15:30:26.522 UTC [2547] DETAIL: The data directory was initialized by PostgreSQL version 15, which is not compatible with this versio
n 13.22 (Debian 13.22-1.pgdg12+1).

This is the point of failure:
Failed to migrate secondsite
#<StandardError:"An error has occurred, this and all later migrations canceled:\n\nERROR: must be owner of extension vector\n">
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rack-mini-profiler-4.0.1/lib/patches/db/pg/alias_method.rb:109:in `exec'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rack-mini-profiler-4.0.1/lib/patches/db/pg/alias_method.rb:109:in `async_exec'

I tried the 13 template. The problem is the database is already altered by 15 but the migration of the 2 added multi sites were somehow added with a user that seems to be unrecognized by the installer. Like I said above, there was no requirement for a user/password combo when I added these initially so I don’t know exactly how they got added to the data base.

Is there some value I can add to the user/password options that would satisfy them? Remember, I didn’t add them so what did the initial script put them in under if not postgres which seems to be the default user indicated by the output I see.

Would it be possible to just comment out the two added sites to get the database up and THEN maybe use whatever import method that is being used now?

This is a personal site mostly and losing it will be annoying but not devastating. I ‘stew_pid’ly thought my regular backups were getting this. :frowning:

13

If you’ll look at the pg 15 topic it’ll tell you how to rename the backup directory that still has the pg 13 files.

14

That’s a long thread so let me make sure I’m looking at the correct thing.

  1. Use the 13 template
  2. mv /shared/postgres_data to something like /shared/postgres_data.bak
  3. mv /shared/postgres_data_old to /shared/postgres_data

And this will get the database running with the old?

root@main-app:/shared/postgres_data_old# cat PG_VERSION
13
root@main-app:/shared/postgres_data# cat PG_VERSION
15

Then worry about the

Database has always been my weak spot. :frowning: It seems this is structured different than what I thought which was that all this stuff was in /var/lib/posgresql under one of those strange, opaquely numbered directories.

I do appreciate all the help. There are a few forums where I’m the one shaking my head at some of the users. :wink:

15

If you are building a version of discourse that includes bundled plugins, the build will fail with missing pgvector extension irrespective of what postgres version you are on. if you are an older version of postgres, you will have even more issues than you started with.

I hope you took a backup before this upgrade journey,

I would suggest setting up a fresh discourse instance elsewhere and restoring your backup. That’s the only reliable way to get out of this situation.

You can do it by hand too, but it will be a lot more work and won’t guarantee results.

16

Did a new version get pushed last night? I think I saw some git pulls. Anyway, using the 13 template and moving data worked. Kinda. :wink:
The data base seemed to migrate fine. I can’t find the ‘grant error’ in the pages and pages of output. Doesn’t mean it isn’t there. I might have to run discourse_doctor so I have a saved output I can search for errors. The DB is up and running. I now have a page error instead of bad gateway on my browser. I’ll put the last of the output here for completeness, but, if I don’t find the same grant error listed previously, I’ll start a new thread.

Thanks to all that have helped so far. At minimum I can get into the database now. A big thank you to all that have helped me get this far.

GREATLY APPRECIATED!

== 20240611170904 UpgradePgvector070: migrating ===============================

I, [2025-11-02T15:17:02.281908 #1] INFO -- : Terminating async processes
I, [2025-11-02T15:17:02.282204 #1] INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U p
ostgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main pid: 2547
I, [2025-11-02T15:17:02.282450 #1] INFO -- : Sending TERM to exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf pid: 2644
2025-11-02 15:17:02.289 UTC [2547] LOG: received fast shutdown request
2644:signal-handler (1762096622) Received SIGTERM scheduling shutdown...
2644:M 02 Nov 2025 15:17:02.299 # User requested shutdown...
2644:M 02 Nov 2025 15:17:02.299 * Saving the final RDB snapshot before exiting.
2025-11-02 15:17:02.384 UTC [2547] LOG: aborting any active transactions
2025-11-02 15:17:02.391 UTC [2547] LOG: background worker "logical replication launcher" (PID 2562) exited with exit code 1
2025-11-02 15:17:02.452 UTC [2557] LOG: shutting down
2644:M 02 Nov 2025 15:17:02.549 * DB saved on disk
2644:M 02 Nov 2025 15:17:02.549 # Redis is now ready to exit, bye bye...
2025-11-02 15:17:06.304 UTC [2547] LOG: database system is shut down

FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && sudo -E -u discourse bundle exec rake multisite:migrate failed with return #<Process::Status: pid 4308
exit 1>
Location of failure: /usr/local/lib/ruby/gems/3.3.0/gems/pups-1.3.0/lib/pups/exec_command.rb:131:in `spawn'
exec failed with the params "cd /var/www/discourse && sudo -E -u discourse bundle exec rake multisite:migrate"
bootstrap failed with exit code 1
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.

EDIT: Found this in the logs. I’m not going to do anything until I run doctor and search for errors though.
2025-11-02 16:10:09.680 UTC [6244] postgres@c_discourse LOG: provided user name (postgres) and authenticated user name (discourse) do not match
2025-11-02 16:10:09.680 UTC [6244] postgres@c_discourse FATAL: Peer authentication failed for user "postgres"
2025-11-02 16:10:09.680 UTC [6244] postgres@c_discourse DETAIL: Connection matched pg_hba.conf line 89: "local all postgres
peer"

17

Just to close out this thread, I did run doctor and found the alter error. Since the database was running I ran the alter command on the databases involved in multisite, reran the build and they are now migrated. The user access to the database persists but I’ll start another thread for that.

Again, thanks to all for getting me to this point.

18

Since I moved them as root I had to chown and then chmod the files. I didn’t document these but the error output give it.

19

Link to current problem.

20

A post was merged into an existing topic: Database access issues after upgrade v3.5.2 → v3.6.0.beta2