Forum crashes after upgrading from 2.4.0

skozz · July 3, 2020, 5:43pm

Hi! today I upgraded the forum in a regular way (admin > dashboard > upgrade) from 2.4.0.
After that, back to the upgrades view to keep upgrading the rest of the plunging this message came up

You are running an old version of the Discourse image.
Upgrades via the web UI are disabled until you run the latest image.

To do so log in to your server using SSH and run:
    cd /var/discourse
    git pull
    ./launcher rebuild app

Nothing new until now. Opened the term, joined the virtual machine and followed those steps as always, but now I'm receiving an error I've never seen before. Here are the logs 

root@foro:/var/discourse# ./launcher rebuild app
Ensuring launcher is up to date
Fetching origin
Launcher is up-to-date
cd /pups && git pull && /pups/bin/pups --stdin
Already up to date.
I, [2020-07-03T17:40:38.599175 #1]  INFO -- : Loading --stdin
I, [2020-07-03T17:40:38.608573 #1]  INFO -- : > apt-get remove -y postgresql-10 postgresql-client-10 postgresql-contrib-10
I, [2020-07-03T17:40:40.109309 #1]  INFO -- : Reading package lists...
Building dependency tree...
Reading state information...
Package 'postgresql-10' is not installed, so not removed
Package 'postgresql-client-10' is not installed, so not removed
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

I, [2020-07-03T17:40:40.110000 #1]  INFO -- : > apt-get update && apt-get install -y postgresql-9.5 postgresql-client-9.5 postgresql-contrib-9.5
debconf: delaying package configuration, since apt-utils is not installed
I, [2020-07-03T17:40:49.742789 #1]  INFO -- : Get:1 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
Get:2 http://apt.postgresql.org/pub/repos/apt buster-pgdg InRelease [84.6 kB]
Hit:3 http://deb.debian.org/debian buster InRelease
Get:4 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]
Get:5 https://deb.nodesource.com/node_10.x buster InRelease [4,584 B]
Get:6 http://security.debian.org/debian-security buster/updates/main amd64 Packages [208 kB]
Get:7 http://apt.postgresql.org/pub/repos/apt buster-pgdg/main amd64 Packages [177 kB]
Get:8 http://deb.debian.org/debian buster-updates/main amd64 Packages.diff/Index [3,688 B]
Get:9 http://deb.debian.org/debian buster-updates/main amd64 Packages 2020-06-04-2016.16.pdiff [1,101 B]
Get:10 http://deb.debian.org/debian buster-updates/main amd64 Packages 2020-06-07-1403.53.pdiff [439 B]
Get:11 http://deb.debian.org/debian buster-updates/main amd64 Packages 2020-06-13-2000.26.pdiff [552 B]
Get:11 http://deb.debian.org/debian buster-updates/main amd64 Packages 2020-06-13-2000.26.pdiff [552 B]
Get:12 https://deb.nodesource.com/node_10.x buster/main amd64 Packages [768 B]
Fetched 599 kB in 1s (686 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
Suggested packages:
  postgresql-doc-9.5 libdbd-pg-perl
The following NEW packages will be installed:
  postgresql-9.5 postgresql-client-9.5 postgresql-contrib-9.5
0 upgraded, 3 newly installed, 0 to remove and 13 not upgraded.
Need to get 5,700 kB of archives.
After this operation, 27.2 MB of additional disk space will be used.
Get:1 http://apt.postgresql.org/pub/repos/apt buster-pgdg/main amd64 postgresql-client-9.5 amd64 9.5.22-1.pgdg100+1 [1,252 kB]
Get:2 http://apt.postgresql.org/pub/repos/apt buster-pgdg/main amd64 postgresql-9.5 amd64 9.5.22-1.pgdg100+1 [3,985 kB]
Get:3 http://apt.postgresql.org/pub/repos/apt buster-pgdg/main amd64 postgresql-contrib-9.5 amd64 9.5.22-1.pgdg100+1 [463 kB]
Fetched 5,700 kB in 0s (27.1 MB/s)
Selecting previously unselected package postgresql-client-9.5.
(Reading database ... 43929 files and directories currently installed.)
Preparing to unpack .../postgresql-client-9.5_9.5.22-1.pgdg100+1_amd64.deb ...
Unpacking postgresql-client-9.5 (9.5.22-1.pgdg100+1) ...
Selecting previously unselected package postgresql-9.5.
Preparing to unpack .../postgresql-9.5_9.5.22-1.pgdg100+1_amd64.deb ...
Unpacking postgresql-9.5 (9.5.22-1.pgdg100+1) ...
Selecting previously unselected package postgresql-contrib-9.5.
Preparing to unpack .../postgresql-contrib-9.5_9.5.22-1.pgdg100+1_amd64.deb ...
Unpacking postgresql-contrib-9.5 (9.5.22-1.pgdg100+1) ...
Setting up postgresql-client-9.5 (9.5.22-1.pgdg100+1) ...
update-alternatives: warning: forcing reinstallation of alternative /usr/share/postgresql/12/man/man1/psql.1.gz because link group psql.1.gz is broken
Setting up postgresql-9.5 (9.5.22-1.pgdg100+1) ...
Creating new PostgreSQL cluster 9.5/main ...
/usr/lib/postgresql/9.5/bin/initdb -D /var/lib/postgresql/9.5/main --auth-local peer --auth-host md5
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "C.UTF-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/9.5/main ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default timezone ... Etc/UTC
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
creating template1 database in /var/lib/postgresql/9.5/main/base/1 ... ok
initializing pg_authid ... ok
initializing dependencies ... ok
creating system views ... ok
loading system objects' descriptions ... ok
creating collations ... ok
creating conversions ... ok
creating dictionaries ... ok
setting privileges on built-in objects ... ok
creating information schema ... ok
loading PL/pgSQL server-side language ... ok
vacuuming database template1 ... ok
copying template1 to template0 ... ok
copying template1 to postgres ... ok
syncing data to disk ... ok

Success. You can now start the database server using:

    /usr/lib/postgresql/9.5/bin/pg_ctl -D /var/lib/postgresql/9.5/main -l logfile start

Ver Cluster Port Status Owner    Data directory               Log file
9.5 main    5433 down   postgres /var/lib/postgresql/9.5/main /var/log/postgresql/postgresql-9.5-main.log
update-alternatives: warning: forcing reinstallation of alternative /usr/share/postgresql/12/man/man1/postmaster.1.gz because link group postmaster.1.gz is broken
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Setting up postgresql-contrib-9.5 (9.5.22-1.pgdg100+1) ...
Processing triggers for postgresql-common (213.pgdg100+1) ...
Building PostgreSQL dictionaries from installed myspell/hunspell packages...
Removing obsolete dictionary files:

I, [2020-07-03T17:40:49.743707 #1]  INFO -- : > mkdir -p /shared/postgres_run
I, [2020-07-03T17:40:49.747572 #1]  INFO -- :
I, [2020-07-03T17:40:49.748032 #1]  INFO -- : > chown postgres:postgres /shared/postgres_run
I, [2020-07-03T17:40:49.750621 #1]  INFO -- :
I, [2020-07-03T17:40:49.751119 #1]  INFO -- : > chmod 775 /shared/postgres_run
I, [2020-07-03T17:40:49.753527 #1]  INFO -- :
I, [2020-07-03T17:40:49.753870 #1]  INFO -- : > rm -fr /var/run/postgresql
I, [2020-07-03T17:40:49.756586 #1]  INFO -- :
I, [2020-07-03T17:40:49.756966 #1]  INFO -- : > ln -s /shared/postgres_run /var/run/postgresql
I, [2020-07-03T17:40:49.759345 #1]  INFO -- :
I, [2020-07-03T17:40:49.759525 #1]  INFO -- : > socat /dev/null UNIX-CONNECT:/shared/postgres_run/.s.PGSQL.5432 || exit 0 && echo postgres already running stop container ; exit 1
2020/07/03 17:40:49 socat[1570] E connect(6, AF=1 "/shared/postgres_run/.s.PGSQL.5432", 36): No such file or directory
I, [2020-07-03T17:40:49.764619 #1]  INFO -- :
I, [2020-07-03T17:40:49.764790 #1]  INFO -- : > rm -fr /shared/postgres_run/.s*
I, [2020-07-03T17:40:49.768012 #1]  INFO -- :
I, [2020-07-03T17:40:49.768201 #1]  INFO -- : > rm -fr /shared/postgres_run/*.pid
I, [2020-07-03T17:40:49.771186 #1]  INFO -- :
I, [2020-07-03T17:40:49.771329 #1]  INFO -- : > mkdir -p /shared/postgres_run/9.5-main.pg_stat_tmp
I, [2020-07-03T17:40:49.773782 #1]  INFO -- :
I, [2020-07-03T17:40:49.774066 #1]  INFO -- : > chown postgres:postgres /shared/postgres_run/9.5-main.pg_stat_tmp
I, [2020-07-03T17:40:49.776490 #1]  INFO -- :
I, [2020-07-03T17:40:49.782041 #1]  INFO -- : File > /etc/service/postgres/run  chmod: +x  chown:
I, [2020-07-03T17:40:49.788039 #1]  INFO -- : File > /etc/runit/3.d/99-postgres  chmod: +x  chown:
I, [2020-07-03T17:40:49.788568 #1]  INFO -- : > chown -R root /var/lib/postgresql/9.5/main
I, [2020-07-03T17:40:49.799608 #1]  INFO -- :
I, [2020-07-03T17:40:49.799833 #1]  INFO -- : > [ ! -e /shared/postgres_data ] && install -d -m 0755 -o postgres -g postgres /shared/postgres_data && sudo -E -u postgres /usr/lib/postgresql/9.5/bin/initdb -D /shared/postgres_data || exit 0
I, [2020-07-03T17:40:49.802712 #1]  INFO -- :
I, [2020-07-03T17:40:49.802859 #1]  INFO -- : > chown -R postgres:postgres /shared/postgres_data
I, [2020-07-03T17:40:49.823259 #1]  INFO -- :
I, [2020-07-03T17:40:49.823552 #1]  INFO -- : > chown -R postgres:postgres /var/run/postgresql
I, [2020-07-03T17:40:49.827629 #1]  INFO -- :
I, [2020-07-03T17:40:49.828112 #1]  INFO -- : Replacing data_directory = '/var/lib/postgresql/9.5/main' with data_directory = '/shared/postgres_data' in /etc/postgresql/9.5/main/postgresql.conf
I, [2020-07-03T17:40:49.828954 #1]  INFO -- : Replacing (?-mix:#?listen_addresses *=.*) with listen_addresses = '*' in /etc/postgresql/9.5/main/postgresql.conf
I, [2020-07-03T17:40:49.829559 #1]  INFO -- : Replacing (?-mix:#?synchronous_commit *=.*) with synchronous_commit = $db_synchronous_commit in /etc/postgresql/9.5/main/postgresql.conf
I, [2020-07-03T17:40:49.830168 #1]  INFO -- : Replacing (?-mix:#?shared_buffers *=.*) with shared_buffers = $db_shared_buffers in /etc/postgresql/9.5/main/postgresql.conf
I, [2020-07-03T17:40:49.830753 #1]  INFO -- : Replacing (?-mix:#?work_mem *=.*) with work_mem = $db_work_mem in /etc/postgresql/9.5/main/postgresql.conf
I, [2020-07-03T17:40:49.831385 #1]  INFO -- : Replacing (?-mix:#?default_text_search_config *=.*) with default_text_search_config = '$db_default_text_search_config' in /etc/postgresql/9.5/main/postgresql.conf
I, [2020-07-03T17:40:49.831981 #1]  INFO -- : > install -d -m 0755 -o postgres -g postgres /shared/postgres_backup
I, [2020-07-03T17:40:49.835928 #1]  INFO -- :
I, [2020-07-03T17:40:49.836456 #1]  INFO -- : Replacing (?-mix:#?max_wal_senders *=.*) with max_wal_senders = $db_max_wal_senders in /etc/postgresql/9.5/main/postgresql.conf
I, [2020-07-03T17:40:49.837120 #1]  INFO -- : Replacing (?-mix:#?wal_level *=.*) with wal_level = $db_wal_level in /etc/postgresql/9.5/main/postgresql.conf
I, [2020-07-03T17:40:49.837800 #1]  INFO -- : Replacing (?-mix:#?checkpoint_segments *=.*) with checkpoint_segments = $db_checkpoint_segments in /etc/postgresql/9.5/main/postgresql.conf
I, [2020-07-03T17:40:49.838367 #1]  INFO -- : Replacing (?-mix:#?logging_collector *=.*) with logging_collector = $db_logging_collector in /etc/postgresql/9.5/main/postgresql.conf
I, [2020-07-03T17:40:49.838967 #1]  INFO -- : Replacing (?-mix:#?log_min_duration_statement *=.*) with log_min_duration_statement = $db_log_min_duration_statement in /etc/postgresql/9.5/main/postgresql.conf
I, [2020-07-03T17:40:49.839599 #1]  INFO -- : Replacing (?-mix:^#local +replication +postgres +peer$) with local replication postgres  peer in /etc/postgresql/9.5/main/pg_hba.conf
I, [2020-07-03T17:40:49.840164 #1]  INFO -- : Replacing (?-mix:^host.*all.*all.*127.*$) with host all all 0.0.0.0/0 md5 in /etc/postgresql/9.5/main/pg_hba.conf
I, [2020-07-03T17:40:49.840674 #1]  INFO -- : > HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/9.5/bin/postmaster -D /etc/postgresql/9.5/main
I, [2020-07-03T17:40:49.843014 #1]  INFO -- : > sleep 5
2020-07-03 17:40:49.929 UTC [1589] LOG:  database system was shut down at 2020-07-03 17:35:44 UTC
2020-07-03 17:40:49.935 UTC [1589] LOG:  MultiXact member wraparound protections are now enabled
2020-07-03 17:40:49.939 UTC [1593] LOG:  autovacuum launcher started
2020-07-03 17:40:49.939 UTC [1586] LOG:  database system is ready to accept connections
I, [2020-07-03T17:40:54.846567 #1]  INFO -- :
I, [2020-07-03T17:40:54.847806 #1]  INFO -- : > su postgres -c 'createdb discourse' || true
createdb: error: could not connect to database template1: could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
I, [2020-07-03T17:40:54.926943 #1]  INFO -- :
I, [2020-07-03T17:40:54.927231 #1]  INFO -- : > su postgres -c 'psql discourse -c "create user discourse;"' || true
psql: error: could not connect to server: could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
I, [2020-07-03T17:40:55.015051 #1]  INFO -- :
I, [2020-07-03T17:40:55.015594 #1]  INFO -- : > su postgres -c 'psql discourse -c "grant all privileges on database discourse to discourse;"' || true
psql: error: could not connect to server: could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
I, [2020-07-03T17:40:55.091507 #1]  INFO -- :
I, [2020-07-03T17:40:55.091839 #1]  INFO -- : > su postgres -c 'psql discourse -c "alter schema public owner to discourse;"'
psql: error: could not connect to server: could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
I, [2020-07-03T17:40:55.167810 #1]  INFO -- :
I, [2020-07-03T17:40:55.168235 #1]  INFO -- : Terminating async processes
I, [2020-07-03T17:40:55.168298 #1]  INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/9.5/bin/postmaster -D /etc/postgresql/9.5/main pid: 1586
2020-07-03 17:40:55.168 UTC [1586] LOG:  received fast shutdown request
2020-07-03 17:40:55.168 UTC [1586] LOG:  aborting any active transactions
2020-07-03 17:40:55.170 UTC [1593] LOG:  autovacuum launcher shutting down
2020-07-03 17:40:55.172 UTC [1590] LOG:  shutting down
2020-07-03 17:40:55.189 UTC [1590] LOG:  database system is shut down


FAILED
--------------------
Pups::ExecError: su postgres -c 'psql discourse -c "alter schema public owner to discourse;"' failed with return #<Process::Status: pid 1618 exit 2>
Location of failure: /pups/lib/pups/exec_command.rb:112:in `spawn'
exec failed with the params "su postgres -c 'psql $db_name -c \"alter schema public owner to $db_user;\"'"
a10b0d1db8256d70432e08508f453835a4bd7dce8f4ac601b6be1031bb3ce01b
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.
root@foro:/var/discourse#

After trying all the classic tricks (including “turn it off and on again”), I’m a little bit lost here. I don’t understand why it can access to the DB

Falco · July 3, 2020, 5:46pm

PostgreSQL 9.5!? Damn that is old.

Can you please copy and paste here the first lines of the /var/discourse/containers/app.yml file?

skozz · July 3, 2020, 5:50pm

Hola! thanks for answering!

root@foro:/var/discourse# cat /var/discourse/containers/app.yml
##
## After making changes to this file, you MUST rebuild for any changes
## to take effect in your live Discourse instance:
##
## /var/discourse/launcher rebuild app
##
## Make sure to obey YAML syntax! You can use this site to help check:
## http://www.yamllint.com/

## this is the all-in-one, standalone Discourse Docker container template

# You may add rate limiting by uncommenting the web.ratelimited template.
# Out of the box it allows 12 reqs a second per ip, and 100 per minute per ip
# This is configurable by amending the params in this file

templates:
  # - "templates/postgres.template.yml"
  - "templates/postgres.9.5.template.yml"
  - "templates/redis.template.yml"
  - "templates/web.template.yml"
  - "templates/web.ratelimited.template.yml"

 ## Comment out the following lines if you wish to add Lets Encrypt for your Discourse install
  - "templates/web.ssl.template.yml"
  - "templates/web.letsencrypt.ssl.template.yml"

I just tried with “templates/postgres.10.template.yml” (as I just read here) but still throwing errors:

Launcher is up-to-date
cd /pups && git pull && /pups/bin/pups --stdin
Already up to date.
I, [2020-07-03T17:50:57.037855 #1]  INFO -- : Loading --stdin
I, [2020-07-03T17:50:57.050344 #1]  INFO -- : > DEBIAN_FRONTEND=noninteractive apt-get purge -y postgresql-12 postgresql-client-12 postgresql-contrib-12
I, [2020-07-03T17:50:59.618428 #1]  INFO -- : Reading package lists...
Building dependency tree...
Reading state information...
The following package was automatically installed and is no longer required:
  libllvm7
Use 'apt autoremove' to remove it.
The following packages will be REMOVED:
  postgresql-12* postgresql-client-12*
0 upgraded, 0 newly installed, 2 to remove and 0 not upgraded.
After this operation, 52.8 MB disk space will be freed.
(Reading database ... 43929 files and directories currently installed.)
Removing postgresql-12 (12.2-2.pgdg100+1) ...
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of stop.
Removing postgresql-client-12 (12.2-2.pgdg100+1) ...
Processing triggers for postgresql-common (213.pgdg100+1) ...
Building PostgreSQL dictionaries from installed myspell/hunspell packages...
Removing obsolete dictionary files:
(Reading database ... 42106 files and directories currently installed.)
Purging configuration files for postgresql-12 (12.2-2.pgdg100+1) ...
Dropping cluster main...

I, [2020-07-03T17:50:59.618701 #1]  INFO -- : > apt-get update && apt-get install -y postgresql-10 postgresql-client-10 postgresql-contrib-10
debconf: delaying package configuration, since apt-utils is not installed
I, [2020-07-03T17:51:08.574737 #1]  INFO -- : Hit:1 http://deb.debian.org/debian buster InRelease
Get:2 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]
Get:3 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
Get:4 https://deb.nodesource.com/node_10.x buster InRelease [4,584 B]
Get:5 http://apt.postgresql.org/pub/repos/apt buster-pgdg InRelease [84.6 kB]
Get:6 http://security.debian.org/debian-security buster/updates/main amd64 Packages [208 kB]
Get:7 http://deb.debian.org/debian buster-updates/main amd64 Packages.diff/Index [3,688 B]
Get:8 http://deb.debian.org/debian buster-updates/main amd64 Packages 2020-06-04-2016.16.pdiff [1,101 B]
Get:9 http://deb.debian.org/debian buster-updates/main amd64 Packages 2020-06-07-1403.53.pdiff [439 B]
Get:10 http://deb.debian.org/debian buster-updates/main amd64 Packages 2020-06-13-2000.26.pdiff [552 B]
Get:10 http://deb.debian.org/debian buster-updates/main amd64 Packages 2020-06-13-2000.26.pdiff [552 B]
Get:11 https://deb.nodesource.com/node_10.x buster/main amd64 Packages [768 B]
Get:12 http://apt.postgresql.org/pub/repos/apt buster-pgdg/main amd64 Packages [177 kB]
Fetched 599 kB in 1s (616 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following package was automatically installed and is no longer required:
  libllvm7
Use 'apt autoremove' to remove it.
Suggested packages:
  postgresql-doc-10
The following NEW packages will be installed:
  postgresql-10 postgresql-client-10
0 upgraded, 2 newly installed, 0 to remove and 11 not upgraded.
Need to get 6,390 kB of archives.
After this operation, 30.6 MB of additional disk space will be used.
Get:1 http://apt.postgresql.org/pub/repos/apt buster-pgdg/main amd64 postgresql-client-10 amd64 10.13-1.pgdg100+1 [1,428 kB]
Get:2 http://apt.postgresql.org/pub/repos/apt buster-pgdg/main amd64 postgresql-10 amd64 10.13-1.pgdg100+1 [4,961 kB]
Fetched 6,390 kB in 0s (28.3 MB/s)
Selecting previously unselected package postgresql-client-10.
(Reading database ... 42106 files and directories currently installed.)
Preparing to unpack .../postgresql-client-10_10.13-1.pgdg100+1_amd64.deb ...
Unpacking postgresql-client-10 (10.13-1.pgdg100+1) ...
Selecting previously unselected package postgresql-10.
Preparing to unpack .../postgresql-10_10.13-1.pgdg100+1_amd64.deb ...
Unpacking postgresql-10 (10.13-1.pgdg100+1) ...
Setting up postgresql-client-10 (10.13-1.pgdg100+1) ...
update-alternatives: using /usr/share/postgresql/10/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode
Setting up postgresql-10 (10.13-1.pgdg100+1) ...
Creating new PostgreSQL cluster 10/main ...
/usr/lib/postgresql/10/bin/initdb -D /var/lib/postgresql/10/main --auth-local peer --auth-host md5
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "C.UTF-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/10/main ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default timezone ... Etc/UTC
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

Success. You can now start the database server using:

    pg_ctlcluster 10 main start

Ver Cluster Port Status Owner    Data directory              Log file
10  main    5432 down   postgres /var/lib/postgresql/10/main /var/log/postgresql/postgresql-10-main.log
update-alternatives: using /usr/share/postgresql/10/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Processing triggers for postgresql-common (213.pgdg100+1) ...
Building PostgreSQL dictionaries from installed myspell/hunspell packages...
Removing obsolete dictionary files:

I, [2020-07-03T17:51:08.576309 #1]  INFO -- : > mkdir -p /shared/postgres_run
I, [2020-07-03T17:51:08.579856 #1]  INFO -- :
I, [2020-07-03T17:51:08.580190 #1]  INFO -- : > chown postgres:postgres /shared/postgres_run
I, [2020-07-03T17:51:08.582830 #1]  INFO -- :
I, [2020-07-03T17:51:08.583011 #1]  INFO -- : > chmod 775 /shared/postgres_run
I, [2020-07-03T17:51:08.585062 #1]  INFO -- :
I, [2020-07-03T17:51:08.585264 #1]  INFO -- : > rm -fr /var/run/postgresql
I, [2020-07-03T17:51:08.587809 #1]  INFO -- :
I, [2020-07-03T17:51:08.588006 #1]  INFO -- : > ln -s /shared/postgres_run /var/run/postgresql
I, [2020-07-03T17:51:08.590249 #1]  INFO -- :
I, [2020-07-03T17:51:08.590442 #1]  INFO -- : > socat /dev/null UNIX-CONNECT:/shared/postgres_run/.s.PGSQL.5432 || exit 0 && echo postgres already running stop container ; exit 1
2020/07/03 17:51:08 socat[1560] E connect(6, AF=1 "/shared/postgres_run/.s.PGSQL.5432", 36): No such file or directory
I, [2020-07-03T17:51:08.596137 #1]  INFO -- :
I, [2020-07-03T17:51:08.596439 #1]  INFO -- : > rm -fr /shared/postgres_run/.s*
I, [2020-07-03T17:51:08.600217 #1]  INFO -- :
I, [2020-07-03T17:51:08.600879 #1]  INFO -- : > rm -fr /shared/postgres_run/*.pid
I, [2020-07-03T17:51:08.604747 #1]  INFO -- :
I, [2020-07-03T17:51:08.604982 #1]  INFO -- : > mkdir -p /shared/postgres_run/10-main.pg_stat_tmp
I, [2020-07-03T17:51:08.608774 #1]  INFO -- :
I, [2020-07-03T17:51:08.609388 #1]  INFO -- : > chown postgres:postgres /shared/postgres_run/10-main.pg_stat_tmp
I, [2020-07-03T17:51:08.613026 #1]  INFO -- :
I, [2020-07-03T17:51:08.621472 #1]  INFO -- : File > /etc/service/postgres/run  chmod: +x  chown:
I, [2020-07-03T17:51:08.628392 #1]  INFO -- : File > /etc/runit/3.d/99-postgres  chmod: +x  chown:
I, [2020-07-03T17:51:08.628991 #1]  INFO -- : > chown -R root /var/lib/postgresql/10/main
I, [2020-07-03T17:51:08.639852 #1]  INFO -- :
I, [2020-07-03T17:51:08.640419 #1]  INFO -- : > [ ! -e /shared/postgres_data ] && install -d -m 0755 -o postgres -g postgres /shared/postgres_data && sudo -E -u postgres /usr/lib/postgresql/10/bin/initdb -D /shared/postgres_data || exit 0
I, [2020-07-03T17:51:08.642774 #1]  INFO -- :
I, [2020-07-03T17:51:08.642909 #1]  INFO -- : > chown -R postgres:postgres /shared/postgres_data
I, [2020-07-03T17:51:08.662272 #1]  INFO -- :
I, [2020-07-03T17:51:08.662970 #1]  INFO -- : > chown -R postgres:postgres /var/run/postgresql
I, [2020-07-03T17:51:08.666112 #1]  INFO -- :
I, [2020-07-03T17:51:08.666551 #1]  INFO -- : Replacing data_directory = '/var/lib/postgresql/10/main' with data_directory = '/shared/postgres_data' in /etc/postgresql/10/main/postgresql.conf
I, [2020-07-03T17:51:08.667306 #1]  INFO -- : Replacing (?-mix:#?listen_addresses *=.*) with listen_addresses = '*' in /etc/postgresql/10/main/postgresql.conf
I, [2020-07-03T17:51:08.667901 #1]  INFO -- : Replacing (?-mix:#?synchronous_commit *=.*) with synchronous_commit = $db_synchronous_commit in /etc/postgresql/10/main/postgresql.conf
I, [2020-07-03T17:51:08.668472 #1]  INFO -- : Replacing (?-mix:#?shared_buffers *=.*) with shared_buffers = $db_shared_buffers in /etc/postgresql/10/main/postgresql.conf
I, [2020-07-03T17:51:08.668961 #1]  INFO -- : Replacing (?-mix:#?work_mem *=.*) with work_mem = $db_work_mem in /etc/postgresql/10/main/postgresql.conf
I, [2020-07-03T17:51:08.669431 #1]  INFO -- : Replacing (?-mix:#?default_text_search_config *=.*) with default_text_search_config = '$db_default_text_search_config' in /etc/postgresql/10/main/postgresql.conf
I, [2020-07-03T17:51:08.669849 #1]  INFO -- : > install -d -m 0755 -o postgres -g postgres /shared/postgres_backup
I, [2020-07-03T17:51:08.674201 #1]  INFO -- :
I, [2020-07-03T17:51:08.674855 #1]  INFO -- : Replacing (?-mix:#?max_wal_senders *=.*) with max_wal_senders = $db_max_wal_senders in /etc/postgresql/10/main/postgresql.conf
I, [2020-07-03T17:51:08.675531 #1]  INFO -- : Replacing (?-mix:#?wal_level *=.*) with wal_level = $db_wal_level in /etc/postgresql/10/main/postgresql.conf
I, [2020-07-03T17:51:08.676111 #1]  INFO -- : Replacing (?-mix:#?checkpoint_segments *=.*) with checkpoint_segments = $db_checkpoint_segments in /etc/postgresql/10/main/postgresql.conf
I, [2020-07-03T17:51:08.676715 #1]  INFO -- : Replacing (?-mix:#?logging_collector *=.*) with logging_collector = $db_logging_collector in /etc/postgresql/10/main/postgresql.conf
I, [2020-07-03T17:51:08.677378 #1]  INFO -- : Replacing (?-mix:#?log_min_duration_statement *=.*) with log_min_duration_statement = $db_log_min_duration_statement in /etc/postgresql/10/main/postgresql.conf
I, [2020-07-03T17:51:08.678065 #1]  INFO -- : Replacing (?-mix:^#local +replication +postgres +peer$) with local replication postgres  peer in /etc/postgresql/10/main/pg_hba.conf
I, [2020-07-03T17:51:08.678642 #1]  INFO -- : Replacing (?-mix:^host.*all.*all.*127.*$) with host all all 0.0.0.0/0 md5 in /etc/postgresql/10/main/pg_hba.conf
I, [2020-07-03T17:51:08.679176 #1]  INFO -- : > HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/10/bin/postmaster -D /etc/postgresql/10/main
I, [2020-07-03T17:51:08.681526 #1]  INFO -- : > sleep 5
2020-07-03 17:51:08.696 UTC [1576] FATAL:  database files are incompatible with server
2020-07-03 17:51:08.696 UTC [1576] DETAIL:  The data directory was initialized by PostgreSQL version 9.5, which is not compatible with this version 10.13 (Debian 10.13-1.pgdg100+1).
I, [2020-07-03T17:51:13.684752 #1]  INFO -- :
I, [2020-07-03T17:51:13.685864 #1]  INFO -- : > su postgres -c 'createdb discourse' || true
createdb: could not connect to database template1: could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
I, [2020-07-03T17:51:13.767307 #1]  INFO -- :
I, [2020-07-03T17:51:13.767716 #1]  INFO -- : > su postgres -c 'psql discourse -c "create user discourse;"' || true
psql: could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
I, [2020-07-03T17:51:13.851212 #1]  INFO -- :
I, [2020-07-03T17:51:13.851495 #1]  INFO -- : > su postgres -c 'psql discourse -c "grant all privileges on database discourse to discourse;"' || true
psql: could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
I, [2020-07-03T17:51:13.928923 #1]  INFO -- :
I, [2020-07-03T17:51:13.929167 #1]  INFO -- : > su postgres -c 'psql discourse -c "alter schema public owner to discourse;"'
psql: could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
I, [2020-07-03T17:51:14.007486 #1]  INFO -- :
I, [2020-07-03T17:51:14.007988 #1]  INFO -- : Terminating async processes


FAILED
--------------------
Pups::ExecError: su postgres -c 'psql discourse -c "alter schema public owner to discourse;"' failed with return #<Process::Status: pid 1602 exit 2>
Location of failure: /pups/lib/pups/exec_command.rb:112:in `spawn'
exec failed with the params "su postgres -c 'psql $db_name -c \"alter schema public owner to $db_user;\"'"
2941a508509ae0342bb77af4711f0e0a38436095cd1468d01cbe16aba7c3fc41
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.
root@foro:/var/discourse#

Falco · July 3, 2020, 5:52pm

Can you try with templates/postgres.template.yml instead?

skozz · July 3, 2020, 5:53pm

Sure, here are the logs for that one

root@foro:/var/discourse# ./launcher rebuild app
Ensuring launcher is up to date
Fetching origin
Launcher is up-to-date
cd /pups && git pull && /pups/bin/pups --stdin
Already up to date.
I, [2020-07-03T17:53:22.625663 #1]  INFO -- : Loading --stdin
I, [2020-07-03T17:53:22.635149 #1]  INFO -- : > locale-gen $LANG && update-locale
I, [2020-07-03T17:53:22.671728 #1]  INFO -- : Generating locales (this might take a while)...
Generation complete.

I, [2020-07-03T17:53:22.672429 #1]  INFO -- : > mkdir -p /shared/postgres_run
I, [2020-07-03T17:53:22.677925 #1]  INFO -- :
I, [2020-07-03T17:53:22.678274 #1]  INFO -- : > chown postgres:postgres /shared/postgres_run
I, [2020-07-03T17:53:22.681699 #1]  INFO -- :
I, [2020-07-03T17:53:22.681920 #1]  INFO -- : > chmod 775 /shared/postgres_run
I, [2020-07-03T17:53:22.684258 #1]  INFO -- :
I, [2020-07-03T17:53:22.684487 #1]  INFO -- : > rm -fr /var/run/postgresql
I, [2020-07-03T17:53:22.687048 #1]  INFO -- :
I, [2020-07-03T17:53:22.687247 #1]  INFO -- : > ln -s /shared/postgres_run /var/run/postgresql
I, [2020-07-03T17:53:22.689561 #1]  INFO -- :
I, [2020-07-03T17:53:22.689885 #1]  INFO -- : > socat /dev/null UNIX-CONNECT:/shared/postgres_run/.s.PGSQL.5432 || exit 0 && echo postgres already running stop container ; exit 1
2020/07/03 17:53:22 socat[26] E connect(6, AF=1 "/shared/postgres_run/.s.PGSQL.5432", 36): No such file or directory
I, [2020-07-03T17:53:22.694785 #1]  INFO -- :
I, [2020-07-03T17:53:22.695006 #1]  INFO -- : > rm -fr /shared/postgres_run/.s*
I, [2020-07-03T17:53:22.698248 #1]  INFO -- :
I, [2020-07-03T17:53:22.698408 #1]  INFO -- : > rm -fr /shared/postgres_run/*.pid
I, [2020-07-03T17:53:22.701558 #1]  INFO -- :
I, [2020-07-03T17:53:22.701719 #1]  INFO -- : > mkdir -p /shared/postgres_run/12-main.pg_stat_tmp
I, [2020-07-03T17:53:22.704625 #1]  INFO -- :
I, [2020-07-03T17:53:22.704807 #1]  INFO -- : > chown postgres:postgres /shared/postgres_run/12-main.pg_stat_tmp
I, [2020-07-03T17:53:22.707685 #1]  INFO -- :
I, [2020-07-03T17:53:22.713135 #1]  INFO -- : File > /etc/service/postgres/run  chmod: +x  chown:
I, [2020-07-03T17:53:22.718045 #1]  INFO -- : File > /etc/service/postgres/log/run  chmod: +x  chown:
I, [2020-07-03T17:53:22.723816 #1]  INFO -- : File > /etc/runit/3.d/99-postgres  chmod: +x  chown:
I, [2020-07-03T17:53:22.730016 #1]  INFO -- : File > /root/upgrade_postgres  chmod: +x  chown:
I, [2020-07-03T17:53:22.730328 #1]  INFO -- : > chown -R root /var/lib/postgresql/12/main
I, [2020-07-03T17:53:22.862955 #1]  INFO -- :
I, [2020-07-03T17:53:22.863371 #1]  INFO -- : > [ ! -e /shared/postgres_data ] && install -d -m 0755 -o postgres -g postgres /shared/postgres_data && sudo -E -u postgres /usr/lib/postgresql/12/bin/initdb -D /shared/postgres_data || exit 0
I, [2020-07-03T17:53:22.866954 #1]  INFO -- :
I, [2020-07-03T17:53:22.867304 #1]  INFO -- : > chown -R postgres:postgres /shared/postgres_data
I, [2020-07-03T17:53:22.887517 #1]  INFO -- :
I, [2020-07-03T17:53:22.887828 #1]  INFO -- : > chown -R postgres:postgres /var/run/postgresql
I, [2020-07-03T17:53:22.891051 #1]  INFO -- :
I, [2020-07-03T17:53:22.891375 #1]  INFO -- : > /root/upgrade_postgres
I, [2020-07-03T17:53:22.919256 #1]  INFO -- : Upgrading PostgreSQL from version 9.5 to 12
WARNING: Upgrading PostgresSQL would require an addtional 53G of disk space
Please free up some space, or expand your disk, before continuing.

To avoid upgrading change "templates/postgres.template.yml" TO "templates/postgres.10.template.yml" in containers/app.yml



FAILED
--------------------
Pups::ExecError: /root/upgrade_postgres failed with return #<Process::Status: pid 45 exit 1>
Location of failure: /pups/lib/pups/exec_command.rb:112:in `spawn'
exec failed with the params "/root/upgrade_postgres"
584f999e4af2664cac606c4f56491e458a4f40913dc2f0f4c595792854c33fae
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.
root@foro:/var/discourse#

skozz · July 3, 2020, 5:54pm

@Falco, hold on

WARNING: Upgrading PostgresSQL would require an addtional 53G of disk space

maybe all the problems are related to the disk space!?

Falco · July 3, 2020, 6:02pm

Now, that you are using our latest and supported postgres template, they indeed are.

Our script asks for 2x the current DB size in free space. Looks like you have a pretty big instance!

skozz · July 3, 2020, 6:11pm

A 3-years forum with a lot of activity I’m expanding the Digitalocean space (it takes some time), I will back later to update my first post if finally it’s the solution.

Thank a lot @Falco, I was losing my mind the problem and talking to someone else helps a lot

pfaffman · July 3, 2020, 6:26pm

You might be better off creating a new droplet with an updated OS and restoring the data to the new instance.

That’ll give you a new OS, not require so much disk space, and have less downtime–you’d just put the old site in read-only while you backup/restore to the new droplet, then switch over DNS. You could use a static-ip (I forget what they call it) to make the switch at the network rather than DNS level (after switching dns to the static IP).

skozz · July 4, 2020, 1:45pm

@Falco confirmed: changing the template and expanding the disk space works like a charm. Thanks!

@pfaffman thats a great idea, I will try to do it during the weekend. Thanks for the suggestions

system · August 3, 2020, 1:45pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Failed upgrade from 2.5.0beta4 to 2.5.0beta5 Installation	6	1007	June 29, 2020
Upgrade Failed from 2.7.0.beta1 to 2.7.0.beta3 Installation	13	1864	February 15, 2021
Discourse Rebuild Installation	14	1362	January 28, 2021
When rebuild, Discourse removes PostGreSQL 10 and reinstalls version 9.5 Installation	6	945	May 8, 2020
Updated to latest version: ./analyze_new_cluster.sh message Installation	14	3610	June 25, 2020

Forum crashes after upgrading from 2.4.0

Related topics