Failed web upgrade

Installation
1

Hi

I am having trouble rescuing a failed upgrade on a Ubuntu 20.04.5 LTS (Focal Fossa)" local machine. Following the failed update, from the command line I ran first

sudo apt-get update
sudo apt-get upgrade
cd /var/discourse
git pull
sudo ./launcher rebuild app

the last part of the output was

- dist/javascripts/workbox/workbox-expiration.dev.js: 20.83 KB (5.48 KB gzipped)
 - dist/javascripts/workbox/workbox-expiration.prod.js: 2.82 KB (1.22 KB gzipped)
 - dist/javascripts/workbox/workbox-routing.dev.js: 31.54 KB (7.58 KB gzipped)
 - dist/javascripts/workbox/workbox-routing.prod.js: 3.32 KB (1.44 KB gzipped)
 - dist/javascripts/workbox/workbox-strategies.dev.js: 34.65 KB (4.79 KB gzipped)
 - dist/javascripts/workbox/workbox-strategies.prod.js: 4.75 KB (1.16 KB gzipped)
 - dist/javascripts/workbox/workbox-sw.js: 1.3 KB (741 B gzipped)
Done in 90.53s.

I, [2022-11-24T15:11:00.523692 #1]  INFO -- : Terminating async processes
I, [2022-11-24T15:11:00.523748 #1]  INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main pid: 42
I, [2022-11-24T15:11:00.523808 #1]  INFO -- : Sending TERM to exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf pid: 103
103:signal-handler (1669302660) Received SIGTERM scheduling shutdown...
2022-11-24 15:11:00.524 UTC [42] LOG:  received fast shutdown request
2022-11-24 15:11:00.540 UTC [42] LOG:  aborting any active transactions
2022-11-24 15:11:00.542 UTC [42] LOG:  background worker "logical replication launcher" (PID 51) exited with exit code 1
2022-11-24 15:11:00.544 UTC [46] LOG:  shutting down
103:M 24 Nov 2022 15:11:00.546 # User requested shutdown...
103:M 24 Nov 2022 15:11:00.546 * Saving the final RDB snapshot before exiting.
103:M 24 Nov 2022 15:11:00.597 * DB saved on disk
103:M 24 Nov 2022 15:11:00.597 # Redis is now ready to exit, bye bye...
2022-11-24 15:11:00.680 UTC [42] LOG:  database system is shut down


FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && su discourse -c 'bundle exec rake themes:update assets:precompile' failed with return #<Process::Status: pid 1415 exit 1>
Location of failure: /usr/local/lib/ruby/gems/2.7.0/gems/pups-1.1.1/lib/pups/exec_command.rb:117:in `spawn'
exec failed with the params {"cd"=>"$home", "hook"=>"assets_precompile", "cmd"=>["su discourse -c 'bundle exec rake themes:update assets:precompile'"]}
bootstrap failed with exit code 1
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.
abfc5da15ae62310ad82ccb495992f76ef7b9a87b7bb024ca4bf327ea993bc54

I then tried

sudo apt-get dist-upgrade 
cd /var/discourse
git pull
sudo ./launcher rebuild app

and the output is now

sudo ./launcher rebuild app
WARNING: Docker version 17.05.0-ce deprecated, recommend upgrade to 17.06.2 or newer.
x86_64 arch detected.
WARNING: containers/app.yml file is world-readable. You can secure this file by running: chmod o-rwx containers/app.yml
Ensuring launcher is up to date
Fetching origin
Launcher is up-to-date
Stopping old container
+ /usr/bin/docker stop -t 60 app
app
2.0.20220818-0047: Pulling from discourse/base
Digest: sha256:7734701087766821ffb2ddcef423754798bd345c2ac0d550131c6e6905c268e8
Status: Image is up to date for discourse/base:2.0.20220818-0047
/usr/local/lib/ruby/gems/2.7.0/gems/pups-1.1.1/lib/pups.rb
/usr/local/bin/pups --stdin
I, [2022-11-24T17:05:57.385657 #1]  INFO -- : Reading from stdin
I, [2022-11-24T17:05:57.389797 #1]  INFO -- : > locale-gen $LANG && update-locale
I, [2022-11-24T17:05:57.422186 #1]  INFO -- : Generating locales (this might take a while)...
Generation complete.

I, [2022-11-24T17:05:57.422436 #1]  INFO -- : > mkdir -p /shared/postgres_run
I, [2022-11-24T17:05:57.425262 #1]  INFO -- : 
I, [2022-11-24T17:05:57.425450 #1]  INFO -- : > chown postgres:postgres /shared/postgres_run
I, [2022-11-24T17:05:57.427977 #1]  INFO -- : 
I, [2022-11-24T17:05:57.428106 #1]  INFO -- : > chmod 775 /shared/postgres_run
I, [2022-11-24T17:05:57.429878 #1]  INFO -- : 
I, [2022-11-24T17:05:57.429989 #1]  INFO -- : > rm -fr /var/run/postgresql
I, [2022-11-24T17:05:57.431844 #1]  INFO -- : 
I, [2022-11-24T17:05:57.431957 #1]  INFO -- : > ln -s /shared/postgres_run /var/run/postgresql
I, [2022-11-24T17:05:57.433756 #1]  INFO -- : 
I, [2022-11-24T17:05:57.433876 #1]  INFO -- : > socat /dev/null UNIX-CONNECT:/shared/postgres_run/.s.PGSQL.5432 || exit 0 && echo postgres already running stop container ; exit 1
2022/11/24 17:05:57 socat[19] E connect(6, AF=1 "/shared/postgres_run/.s.PGSQL.5432", 36): No such file or directory
I, [2022-11-24T17:05:57.439214 #1]  INFO -- : 
I, [2022-11-24T17:05:57.439321 #1]  INFO -- : > rm -fr /shared/postgres_run/.s*
I, [2022-11-24T17:05:57.441654 #1]  INFO -- : 
I, [2022-11-24T17:05:57.441752 #1]  INFO -- : > rm -fr /shared/postgres_run/*.pid
I, [2022-11-24T17:05:57.444021 #1]  INFO -- : 
I, [2022-11-24T17:05:57.444120 #1]  INFO -- : > mkdir -p /shared/postgres_run/13-main.pg_stat_tmp
I, [2022-11-24T17:05:57.446040 #1]  INFO -- : 
I, [2022-11-24T17:05:57.446171 #1]  INFO -- : > chown postgres:postgres /shared/postgres_run/13-main.pg_stat_tmp
I, [2022-11-24T17:05:57.448063 #1]  INFO -- : 
I, [2022-11-24T17:05:57.452270 #1]  INFO -- : File > /etc/service/postgres/run  chmod: +x  chown: 
I, [2022-11-24T17:05:57.456198 #1]  INFO -- : File > /etc/service/postgres/log/run  chmod: +x  chown: 
I, [2022-11-24T17:05:57.460141 #1]  INFO -- : File > /etc/runit/3.d/99-postgres  chmod: +x  chown: 
I, [2022-11-24T17:05:57.464080 #1]  INFO -- : File > /root/upgrade_postgres  chmod: +x  chown: 
I, [2022-11-24T17:05:57.464267 #1]  INFO -- : > chown -R root /var/lib/postgresql/13/main
I, [2022-11-24T17:05:57.537077 #1]  INFO -- : 
I, [2022-11-24T17:05:57.537178 #1]  INFO -- : > [ ! -e /shared/postgres_data ] && install -d -m 0755 -o postgres -g postgres /shared/postgres_data && sudo -E -u postgres /usr/lib/postgresql/13/bin/initdb -D /shared/postgres_data || exit 0
I, [2022-11-24T17:05:57.538742 #1]  INFO -- : 
I, [2022-11-24T17:05:57.538799 #1]  INFO -- : > chown -R postgres:postgres /shared/postgres_data
I, [2022-11-24T17:05:57.550302 #1]  INFO -- : 
I, [2022-11-24T17:05:57.550394 #1]  INFO -- : > chown -R postgres:postgres /var/run/postgresql
I, [2022-11-24T17:05:57.552387 #1]  INFO -- : 
I, [2022-11-24T17:05:57.552528 #1]  INFO -- : > /root/upgrade_postgres
I, [2022-11-24T17:05:57.556422 #1]  INFO -- : 
I, [2022-11-24T17:05:57.556533 #1]  INFO -- : > rm /root/upgrade_postgres
I, [2022-11-24T17:05:57.558395 #1]  INFO -- : 
I, [2022-11-24T17:05:57.558734 #1]  INFO -- : Replacing data_directory = '/var/lib/postgresql/13/main' with data_directory = '/shared/postgres_data' in /etc/postgresql/13/main/postgresql.conf
I, [2022-11-24T17:05:57.559138 #1]  INFO -- : Replacing (?-mix:#?listen_addresses *=.*) with listen_addresses = '*' in /etc/postgresql/13/main/postgresql.conf
I, [2022-11-24T17:05:57.559350 #1]  INFO -- : Replacing (?-mix:#?synchronous_commit *=.*) with synchronous_commit = $db_synchronous_commit in /etc/postgresql/13/main/postgresql.conf
I, [2022-11-24T17:05:57.559550 #1]  INFO -- : Replacing (?-mix:#?shared_buffers *=.*) with shared_buffers = $db_shared_buffers in /etc/postgresql/13/main/postgresql.conf
I, [2022-11-24T17:05:57.559748 #1]  INFO -- : Replacing (?-mix:#?work_mem *=.*) with work_mem = $db_work_mem in /etc/postgresql/13/main/postgresql.conf
I, [2022-11-24T17:05:57.559945 #1]  INFO -- : Replacing (?-mix:#?default_text_search_config *=.*) with default_text_search_config = '$db_default_text_search_config' in /etc/postgresql/13/main/postgresql.conf
I, [2022-11-24T17:05:57.560122 #1]  INFO -- : > install -d -m 0755 -o postgres -g postgres /shared/postgres_backup
I, [2022-11-24T17:05:57.562571 #1]  INFO -- : 
I, [2022-11-24T17:05:57.562855 #1]  INFO -- : Replacing (?-mix:#?checkpoint_segments *=.*) with checkpoint_segments = $db_checkpoint_segments in /etc/postgresql/13/main/postgresql.conf
I, [2022-11-24T17:05:57.563086 #1]  INFO -- : Replacing (?-mix:#?logging_collector *=.*) with logging_collector = $db_logging_collector in /etc/postgresql/13/main/postgresql.conf
I, [2022-11-24T17:05:57.563295 #1]  INFO -- : Replacing (?-mix:#?log_min_duration_statement *=.*) with log_min_duration_statement = $db_log_min_duration_statement in /etc/postgresql/13/main/postgresql.conf
I, [2022-11-24T17:05:57.563506 #1]  INFO -- : Replacing (?-mix:^#local +replication +postgres +peer$) with local replication postgres  peer in /etc/postgresql/13/main/pg_hba.conf
I, [2022-11-24T17:05:57.563702 #1]  INFO -- : Replacing (?-mix:^host.*all.*all.*127.*$) with host all all 0.0.0.0/0 md5 in /etc/postgresql/13/main/pg_hba.conf
I, [2022-11-24T17:05:57.563874 #1]  INFO -- : Replacing (?-mix:^host.*all.*all.*::1\/128.*$) with host all all ::/0 md5 in /etc/postgresql/13/main/pg_hba.conf
I, [2022-11-24T17:05:57.564025 #1]  INFO -- : > HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main
I, [2022-11-24T17:05:57.565221 #1]  INFO -- : > sleep 5
2022-11-24 17:05:57.591 UTC [42] FATAL:  private key file "/etc/ssl/private/ssl-cert-snakeoil.key" must be owned by the database user or root
2022-11-24 17:05:57.591 UTC [42] LOG:  database system is shut down
I, [2022-11-24T17:06:02.567410 #1]  INFO -- : 
I, [2022-11-24T17:06:02.567949 #1]  INFO -- : > su postgres -c 'createdb discourse' || true
createdb: error: could not connect to database template1: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
	Is the server running locally and accepting connections on that socket?
I, [2022-11-24T17:06:02.612636 #1]  INFO -- : 
I, [2022-11-24T17:06:02.612780 #1]  INFO -- : > su postgres -c 'psql discourse -c "create user discourse;"' || true
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
	Is the server running locally and accepting connections on that socket?
I, [2022-11-24T17:06:02.649900 #1]  INFO -- : 
I, [2022-11-24T17:06:02.650042 #1]  INFO -- : > su postgres -c 'psql discourse -c "grant all privileges on database discourse to discourse;"' || true
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
	Is the server running locally and accepting connections on that socket?
I, [2022-11-24T17:06:02.688244 #1]  INFO -- : 
I, [2022-11-24T17:06:02.688412 #1]  INFO -- : > su postgres -c 'psql discourse -c "alter schema public owner to discourse;"'
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
	Is the server running locally and accepting connections on that socket?
I, [2022-11-24T17:06:02.725728 #1]  INFO -- : 
I, [2022-11-24T17:06:02.725969 #1]  INFO -- : Terminating async processes


FAILED
--------------------
Pups::ExecError: su postgres -c 'psql discourse -c "alter schema public owner to discourse;"' failed with return #<Process::Status: pid 54 exit 2>
Location of failure: /usr/local/lib/ruby/gems/2.7.0/gems/pups-1.1.1/lib/pups/exec_command.rb:117:in `spawn'
exec failed with the params "su postgres -c 'psql $db_name -c \"alter schema public owner to $db_user;\"'"
bootstrap failed with exit code 2
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.
8c252b2f3146df27193b8a6966b36a75230460f92397cd206aa5540082090660

Can anyone help?

1 Like
2

Hello and welcome @Nanovitruvius :slight_smile:

There have been a couple of reports of failed upgrades in the past hour or so, and a couple of fixes have now been put in place for those which seemed to have worked for most. :+1:

The error "alter schema public owner to discourse;" in your second codeblock cropped up the other day too, and the advice was:

4 Likes
3

It looks to me like this is the important part - the ownership of a file is wrong, somehow:

In my case (a working installation not updated for a week or so) this file is owned by root:

# cd /var/discourse/
# ./launcher enter app
x86_64 arch detected.
/var/www/discourse# ls -l /etc/ssl/private/ssl-cert-snakeoil.key
-rw-r----- 1 root ssl-cert 1704 Jul 20 00:14 /etc/ssl/private/ssl-cert-snakeoil.key
# exit
2 Likes
4

That version of docker is more than five years old. I would strongly recommend you look to upgrade it while you’re taking care of the other points raised above.

3 Likes
5

Thank you so much for your swift replies!
@JammyDodger - I have increased the relevant line to 10 s, but still get the same issue now as before (it always returns with the Fatal SSL certificate.
@Ed_S I have checked the ownership of the file as you suggested, but it seems identical to yours (except the dates):

nano@nano:/var/discourse$ sudo ls -l /etc/ssl/private/ssl-cert-snakeoil.key
-rw-r----- 1 root ssl-cert 1704 jan.  11  2017 /etc/ssl/private/ssl-cert-snakeoil.key

However I had to sudo to get the command to work, so maybe I need to grant some more privileges when running the update? I’m running it as sudo ./launcher rebuild app so not sure what else I could do here.

Stephen - I will upgrade docker now and rerun everything, then report back here

Thanks for all your help!

1 Like
6

I’ve now upgraded docker to the latest version, the error still looks the same. During the upgrade I got some permission errors (it still seems to work ok):

sudo apt-get install docker-ce docker-ce-cli containerd.io docker-compose-plugin
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  docker-ce-rootless-extras docker-scan-plugin pigz slirp4netns
Suggested packages:
  aufs-tools cgroupfs-mount | cgroup-lite
The following NEW packages will be installed:
  containerd.io docker-ce docker-ce-cli docker-ce-rootless-extras docker-compose-plugin docker-scan-plugin pigz slirp4netns
0 upgraded, 8 newly installed, 0 to remove and 0 not upgraded.
Need to get 111 MB of archives.
After this operation, 428 MB of additional disk space will be used.
Do you want to continue? [Y/n] 
Get:1 https://download.docker.com/linux/ubuntu focal/stable amd64 containerd.io amd64 1.6.10-1 [27,7 MB]
Get:2 https://download.docker.com/linux/ubuntu focal/stable amd64 docker-ce-cli amd64 5:20.10.21~3-0~ubuntu-focal [41,5 MB]
Get:3 http://archive.ubuntu.com/ubuntu focal/universe amd64 pigz amd64 2.4-1 [57,4 kB]
Get:4 https://download.docker.com/linux/ubuntu focal/stable amd64 docker-ce amd64 5:20.10.21~3-0~ubuntu-focal [20,5 MB]
Get:5 http://archive.ubuntu.com/ubuntu focal/universe amd64 slirp4netns amd64 0.4.3-1 [74,3 kB]
Get:6 https://download.docker.com/linux/ubuntu focal/stable amd64 docker-ce-rootless-extras amd64 5:20.10.21~3-0~ubuntu-focal [8 394 kB]
Get:7 https://download.docker.com/linux/ubuntu focal/stable amd64 docker-compose-plugin amd64 2.12.2~ubuntu-focal [9 567 kB]
Get:8 https://download.docker.com/linux/ubuntu focal/stable amd64 docker-scan-plugin amd64 0.21.0~ubuntu-focal [3 622 kB]
Fetched 111 MB in 2s (73,6 MB/s)                
Selecting previously unselected package pigz.
(Reading database ... 308915 files and directories currently installed.)
Preparing to unpack .../0-pigz_2.4-1_amd64.deb ...
Unpacking pigz (2.4-1) ...
Selecting previously unselected package containerd.io.
Preparing to unpack .../1-containerd.io_1.6.10-1_amd64.deb ...
Unpacking containerd.io (1.6.10-1) ...
Selecting previously unselected package docker-ce-cli.
Preparing to unpack .../2-docker-ce-cli_5%3a20.10.21~3-0~ubuntu-focal_amd64.deb ...
Unpacking docker-ce-cli (5:20.10.21~3-0~ubuntu-focal) ...
Selecting previously unselected package docker-ce.
Preparing to unpack .../3-docker-ce_5%3a20.10.21~3-0~ubuntu-focal_amd64.deb ...
Unpacking docker-ce (5:20.10.21~3-0~ubuntu-focal) ...
Selecting previously unselected package docker-ce-rootless-extras.
Preparing to unpack .../4-docker-ce-rootless-extras_5%3a20.10.21~3-0~ubuntu-focal_amd64.deb ...
Unpacking docker-ce-rootless-extras (5:20.10.21~3-0~ubuntu-focal) ...
Selecting previously unselected package docker-compose-plugin.
Preparing to unpack .../5-docker-compose-plugin_2.12.2~ubuntu-focal_amd64.deb ...
Unpacking docker-compose-plugin (2.12.2~ubuntu-focal) ...
Selecting previously unselected package docker-scan-plugin.
Preparing to unpack .../6-docker-scan-plugin_0.21.0~ubuntu-focal_amd64.deb ...
Unpacking docker-scan-plugin (0.21.0~ubuntu-focal) ...
Selecting previously unselected package slirp4netns.
Preparing to unpack .../7-slirp4netns_0.4.3-1_amd64.deb ...
Unpacking slirp4netns (0.4.3-1) ...
Setting up slirp4netns (0.4.3-1) ...
Setting up docker-scan-plugin (0.21.0~ubuntu-focal) ...
Setting up containerd.io (1.6.10-1) ...
Created symlink /etc/systemd/system/multi-user.target.wants/containerd.service → /lib/systemd/system/containerd.service.
Setting up docker-compose-plugin (2.12.2~ubuntu-focal) ...
Setting up docker-ce-cli (5:20.10.21~3-0~ubuntu-focal) ...
Setting up pigz (2.4-1) ...
Setting up docker-ce-rootless-extras (5:20.10.21~3-0~ubuntu-focal) ...
Setting up docker-ce (5:20.10.21~3-0~ubuntu-focal) ...
Processing triggers for man-db (2.9.1-1) ...
/usr/bin/mandb: can't chmod /var/cache/man/hu/CACHEDIR.TAG: Operation not permitted
/usr/bin/mandb: can't remove /var/cache/man/hu/CACHEDIR.TAG: Permission denied
/usr/bin/mandb: fopen /var/cache/man/hu/50779: Permission denied
/usr/bin/mandb: can't chmod /var/cache/man/fi/CACHEDIR.TAG: Operation not permitted
/usr/bin/mandb: can't remove /var/cache/man/fi/CACHEDIR.TAG: Permission denied
/usr/bin/mandb: fopen /var/cache/man/fi/50779: Permission denied
/usr/bin/mandb: can't chmod /var/cache/man/ja/CACHEDIR.TAG: Operation not permitted
/usr/bin/mandb: can't remove /var/cache/man/ja/CACHEDIR.TAG: Permission denied
/usr/bin/mandb: fopen /var/cache/man/ja/50779: Permission denied
/usr/bin/mandb: can't chmod /var/cache/man/ko/CACHEDIR.TAG: Operation not permitted
/usr/bin/mandb: can't remove /var/cache/man/ko/CACHEDIR.TAG: Permission denied
/usr/bin/mandb: fopen /var/cache/man/ko/50779: Permission denied
/usr/bin/mandb: can't chmod /var/cache/man/fr.UTF-8/CACHEDIR.TAG: Operation not permitted
/usr/bin/mandb: can't remove /var/cache/man/fr.UTF-8/CACHEDIR.TAG: Permission denied
/usr/bin/mandb: fopen /var/cache/man/fr.UTF-8/50779: Permission denied
/usr/bin/mandb: can't chmod /var/cache/man/tr/CACHEDIR.TAG: Operation not permitted
/usr/bin/mandb: can't remove /var/cache/man/tr/CACHEDIR.TAG: Permission denied
/usr/bin/mandb: fopen /var/cache/man/tr/50779: Permission denied
/usr/bin/mandb: can't chmod /var/cache/man/pt_BR/CACHEDIR.TAG: Operation not permitted
/usr/bin/mandb: can't remove /var/cache/man/pt_BR/CACHEDIR.TAG: Permission denied
/usr/bin/mandb: fopen /var/cache/man/pt_BR/50779: Permission denied
/usr/bin/mandb: can't chmod /var/cache/man/id/CACHEDIR.TAG: Operation not permitted
/usr/bin/mandb: can't remove /var/cache/man/id/CACHEDIR.TAG: Permission denied
/usr/bin/mandb: fopen /var/cache/man/id/50779: Permission denied
/usr/bin/mandb: can't chmod /var/cache/man/it/CACHEDIR.TAG: Operation not permitted
/usr/bin/mandb: can't remove /var/cache/man/it/CACHEDIR.TAG: Permission denied
/usr/bin/mandb: fopen /var/cache/man/it/50779: Permission denied
/usr/bin/mandb: can't chmod /var/cache/man/de/CACHEDIR.TAG: Operation not permitted
/usr/bin/mandb: can't remove /var/cache/man/de/CACHEDIR.TAG: Permission denied
/usr/bin/mandb: fopen /var/cache/man/de/50779: Permission denied
/usr/bin/mandb: can't chmod /var/cache/man/da/CACHEDIR.TAG: Operation not permitted
/usr/bin/mandb: can't remove /var/cache/man/da/CACHEDIR.TAG: Permission denied
/usr/bin/mandb: fopen /var/cache/man/da/50779: Permission denied
/usr/bin/mandb: can't chmod /var/cache/man/fr.ISO8859-1/CACHEDIR.TAG: Operation not permitted
/usr/bin/mandb: can't remove /var/cache/man/fr.ISO8859-1/CACHEDIR.TAG: Permission denied
/usr/bin/mandb: fopen /var/cache/man/fr.ISO8859-1/50779: Permission denied
/usr/bin/mandb: can't chmod /var/cache/man/pl/CACHEDIR.TAG: Operation not permitted
/usr/bin/mandb: can't remove /var/cache/man/pl/CACHEDIR.TAG: Permission denied
/usr/bin/mandb: fopen /var/cache/man/pl/50779: Permission denied
/usr/bin/mandb: can't chmod /var/cache/man/zh_TW/CACHEDIR.TAG: Operation not permitted
/usr/bin/mandb: can't remove /var/cache/man/zh_TW/CACHEDIR.TAG: Permission denied
/usr/bin/mandb: fopen /var/cache/man/zh_TW/50779: Permission denied
/usr/bin/mandb: can't chmod /var/cache/man/gl/CACHEDIR.TAG: Operation not permitted
/usr/bin/mandb: can't remove /var/cache/man/gl/CACHEDIR.TAG: Permission denied
/usr/bin/mandb: fopen /var/cache/man/gl/50779: Permission denied
/usr/bin/mandb: can't chmod /var/cache/man/sv/CACHEDIR.TAG: Operation not permitted
/usr/bin/mandb: can't remove /var/cache/man/sv/CACHEDIR.TAG: Permission denied
/usr/bin/mandb: fopen /var/cache/man/sv/50779: Permission denied
/usr/bin/mandb: can't chmod /var/cache/man/nl/CACHEDIR.TAG: Operation not permitted
/usr/bin/mandb: can't remove /var/cache/man/nl/CACHEDIR.TAG: Permission denied
/usr/bin/mandb: fopen /var/cache/man/nl/50779: Permission denied
/usr/bin/mandb: can't chmod /var/cache/man/sl/CACHEDIR.TAG: Operation not permitted
/usr/bin/mandb: can't remove /var/cache/man/sl/CACHEDIR.TAG: Permission denied
/usr/bin/mandb: fopen /var/cache/man/sl/50779: Permission denied
/usr/bin/mandb: can't chmod /var/cache/man/pt/CACHEDIR.TAG: Operation not permitted
/usr/bin/mandb: can't remove /var/cache/man/pt/CACHEDIR.TAG: Permission denied
/usr/bin/mandb: fopen /var/cache/man/pt/50779: Permission denied
/usr/bin/mandb: can't chmod /var/cache/man/ru/CACHEDIR.TAG: Operation not permitted
/usr/bin/mandb: can't remove /var/cache/man/ru/CACHEDIR.TAG: Permission denied
/usr/bin/mandb: fopen /var/cache/man/ru/50779: Permission denied
/usr/bin/mandb: can't chmod /var/cache/man/sr/CACHEDIR.TAG: Operation not permitted
/usr/bin/mandb: can't remove /var/cache/man/sr/CACHEDIR.TAG: Permission denied
/usr/bin/mandb: fopen /var/cache/man/sr/50779: Permission denied
/usr/bin/mandb: can't chmod /var/cache/man/es/CACHEDIR.TAG: Operation not permitted
/usr/bin/mandb: can't remove /var/cache/man/es/CACHEDIR.TAG: Permission denied
/usr/bin/mandb: fopen /var/cache/man/es/50779: Permission denied
/usr/bin/mandb: can't chmod /var/cache/man/cs/CACHEDIR.TAG: Operation not permitted
/usr/bin/mandb: can't remove /var/cache/man/cs/CACHEDIR.TAG: Permission denied
/usr/bin/mandb: fopen /var/cache/man/cs/50779: Permission denied
/usr/bin/mandb: can't chmod /var/cache/man/fr/CACHEDIR.TAG: Operation not permitted
/usr/bin/mandb: can't remove /var/cache/man/fr/CACHEDIR.TAG: Permission denied
/usr/bin/mandb: fopen /var/cache/man/fr/50779: Permission denied
/usr/bin/mandb: can't chmod /var/cache/man/zh_CN/CACHEDIR.TAG: Operation not permitted
/usr/bin/mandb: can't remove /var/cache/man/zh_CN/CACHEDIR.TAG: Permission denied
/usr/bin/mandb: fopen /var/cache/man/zh_CN/50779: Permission denied
Processing triggers for systemd (245.4-4ubuntu3.19)

Its strange that when using sudo I should get permissions denied?

the output of rebuilding the app is now


nano@nano:/var/discourse$ sudo ./launcher rebuild app
x86_64 arch detected.
WARNING: containers/app.yml file is world-readable. You can secure this file by running: chmod o-rwx containers/app.yml
Ensuring launcher is up to date
Fetching origin
Launcher is up-to-date
Stopping old container
+ /usr/bin/docker stop -t 60 app
app
2.0.20220818-0047: Pulling from discourse/base
Digest: sha256:7734701087766821ffb2ddcef423754798bd345c2ac0d550131c6e6905c268e8
Status: Image is up to date for discourse/base:2.0.20220818-0047
docker.io/discourse/base:2.0.20220818-0047
/usr/local/lib/ruby/gems/2.7.0/gems/pups-1.1.1/lib/pups.rb
/usr/local/bin/pups --stdin
I, [2022-11-25T09:00:51.131964 #1]  INFO -- : Reading from stdin
I, [2022-11-25T09:00:51.135935 #1]  INFO -- : > locale-gen $LANG && update-locale
I, [2022-11-25T09:00:51.171482 #1]  INFO -- : Generating locales (this might take a while)...
Generation complete.

I, [2022-11-25T09:00:51.171677 #1]  INFO -- : > mkdir -p /shared/postgres_run
I, [2022-11-25T09:00:51.176135 #1]  INFO -- : 
I, [2022-11-25T09:00:51.176692 #1]  INFO -- : > chown postgres:postgres /shared/postgres_run
I, [2022-11-25T09:00:51.179060 #1]  INFO -- : 
I, [2022-11-25T09:00:51.179185 #1]  INFO -- : > chmod 775 /shared/postgres_run
I, [2022-11-25T09:00:51.181507 #1]  INFO -- : 
I, [2022-11-25T09:00:51.181637 #1]  INFO -- : > rm -fr /var/run/postgresql
I, [2022-11-25T09:00:51.183530 #1]  INFO -- : 
I, [2022-11-25T09:00:51.183660 #1]  INFO -- : > ln -s /shared/postgres_run /var/run/postgresql
I, [2022-11-25T09:00:51.185995 #1]  INFO -- : 
I, [2022-11-25T09:00:51.186116 #1]  INFO -- : > socat /dev/null UNIX-CONNECT:/shared/postgres_run/.s.PGSQL.5432 || exit 0 && echo postgres already running stop container ; exit 1
2022/11/25 09:00:51 socat[18] E connect(6, AF=1 "/shared/postgres_run/.s.PGSQL.5432", 36): No such file or directory
I, [2022-11-25T09:00:51.203185 #1]  INFO -- : 
I, [2022-11-25T09:00:51.203547 #1]  INFO -- : > rm -fr /shared/postgres_run/.s*
I, [2022-11-25T09:00:51.212117 #1]  INFO -- : 
I, [2022-11-25T09:00:51.212479 #1]  INFO -- : > rm -fr /shared/postgres_run/*.pid
I, [2022-11-25T09:00:51.220922 #1]  INFO -- : 
I, [2022-11-25T09:00:51.221289 #1]  INFO -- : > mkdir -p /shared/postgres_run/13-main.pg_stat_tmp
I, [2022-11-25T09:00:51.228453 #1]  INFO -- : 
I, [2022-11-25T09:00:51.229084 #1]  INFO -- : > chown postgres:postgres /shared/postgres_run/13-main.pg_stat_tmp
I, [2022-11-25T09:00:51.236038 #1]  INFO -- : 
I, [2022-11-25T09:00:51.251481 #1]  INFO -- : File > /etc/service/postgres/run  chmod: +x  chown: 
I, [2022-11-25T09:00:51.266078 #1]  INFO -- : File > /etc/service/postgres/log/run  chmod: +x  chown: 
I, [2022-11-25T09:00:51.280710 #1]  INFO -- : File > /etc/runit/3.d/99-postgres  chmod: +x  chown: 
I, [2022-11-25T09:00:51.295423 #1]  INFO -- : File > /root/upgrade_postgres  chmod: +x  chown: 
I, [2022-11-25T09:00:51.296150 #1]  INFO -- : > chown -R root /var/lib/postgresql/13/main
I, [2022-11-25T09:00:51.404140 #1]  INFO -- : 
I, [2022-11-25T09:00:51.404284 #1]  INFO -- : > [ ! -e /shared/postgres_data ] && install -d -m 0755 -o postgres -g postgres /shared/postgres_data && sudo -E -u postgres /usr/lib/postgresql/13/bin/initdb -D /shared/postgres_data || exit 0
I, [2022-11-25T09:00:51.406112 #1]  INFO -- : 
I, [2022-11-25T09:00:51.406196 #1]  INFO -- : > chown -R postgres:postgres /shared/postgres_data
I, [2022-11-25T09:00:51.417965 #1]  INFO -- : 
I, [2022-11-25T09:00:51.418059 #1]  INFO -- : > chown -R postgres:postgres /var/run/postgresql
I, [2022-11-25T09:00:51.420209 #1]  INFO -- : 
I, [2022-11-25T09:00:51.420320 #1]  INFO -- : > /root/upgrade_postgres
I, [2022-11-25T09:00:51.424814 #1]  INFO -- : 
I, [2022-11-25T09:00:51.424928 #1]  INFO -- : > rm /root/upgrade_postgres
I, [2022-11-25T09:00:51.426794 #1]  INFO -- : 
I, [2022-11-25T09:00:51.427083 #1]  INFO -- : Replacing data_directory = '/var/lib/postgresql/13/main' with data_directory = '/shared/postgres_data' in /etc/postgresql/13/main/postgresql.conf
I, [2022-11-25T09:00:51.427479 #1]  INFO -- : Replacing (?-mix:#?listen_addresses *=.*) with listen_addresses = '*' in /etc/postgresql/13/main/postgresql.conf
I, [2022-11-25T09:00:51.427699 #1]  INFO -- : Replacing (?-mix:#?synchronous_commit *=.*) with synchronous_commit = $db_synchronous_commit in /etc/postgresql/13/main/postgresql.conf
I, [2022-11-25T09:00:51.427898 #1]  INFO -- : Replacing (?-mix:#?shared_buffers *=.*) with shared_buffers = $db_shared_buffers in /etc/postgresql/13/main/postgresql.conf
I, [2022-11-25T09:00:51.428093 #1]  INFO -- : Replacing (?-mix:#?work_mem *=.*) with work_mem = $db_work_mem in /etc/postgresql/13/main/postgresql.conf
I, [2022-11-25T09:00:51.428287 #1]  INFO -- : Replacing (?-mix:#?default_text_search_config *=.*) with default_text_search_config = '$db_default_text_search_config' in /etc/postgresql/13/main/postgresql.conf
I, [2022-11-25T09:00:51.428462 #1]  INFO -- : > install -d -m 0755 -o postgres -g postgres /shared/postgres_backup
I, [2022-11-25T09:00:51.432155 #1]  INFO -- : 
I, [2022-11-25T09:00:51.432441 #1]  INFO -- : Replacing (?-mix:#?checkpoint_segments *=.*) with checkpoint_segments = $db_checkpoint_segments in /etc/postgresql/13/main/postgresql.conf
I, [2022-11-25T09:00:51.432668 #1]  INFO -- : Replacing (?-mix:#?logging_collector *=.*) with logging_collector = $db_logging_collector in /etc/postgresql/13/main/postgresql.conf
I, [2022-11-25T09:00:51.432878 #1]  INFO -- : Replacing (?-mix:#?log_min_duration_statement *=.*) with log_min_duration_statement = $db_log_min_duration_statement in /etc/postgresql/13/main/postgresql.conf
I, [2022-11-25T09:00:51.433089 #1]  INFO -- : Replacing (?-mix:^#local +replication +postgres +peer$) with local replication postgres  peer in /etc/postgresql/13/main/pg_hba.conf
I, [2022-11-25T09:00:51.433281 #1]  INFO -- : Replacing (?-mix:^host.*all.*all.*127.*$) with host all all 0.0.0.0/0 md5 in /etc/postgresql/13/main/pg_hba.conf
I, [2022-11-25T09:00:51.433452 #1]  INFO -- : Replacing (?-mix:^host.*all.*all.*::1\/128.*$) with host all all ::/0 md5 in /etc/postgresql/13/main/pg_hba.conf
I, [2022-11-25T09:00:51.433603 #1]  INFO -- : > HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main
I, [2022-11-25T09:00:51.434859 #1]  INFO -- : > sleep 10
2022-11-25 09:00:51.521 UTC [41] FATAL:  private key file "/etc/ssl/private/ssl-cert-snakeoil.key" must be owned by the database user or root
2022-11-25 09:00:51.521 UTC [41] LOG:  database system is shut down
I, [2022-11-25T09:01:01.437220 #1]  INFO -- : 
I, [2022-11-25T09:01:01.437718 #1]  INFO -- : > su postgres -c 'createdb discourse' || true
createdb: error: could not connect to database template1: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
	Is the server running locally and accepting connections on that socket?
I, [2022-11-25T09:01:01.503795 #1]  INFO -- : 
I, [2022-11-25T09:01:01.503958 #1]  INFO -- : > su postgres -c 'psql discourse -c "create user discourse;"' || true
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
	Is the server running locally and accepting connections on that socket?
I, [2022-11-25T09:01:01.548995 #1]  INFO -- : 
I, [2022-11-25T09:01:01.549398 #1]  INFO -- : > su postgres -c 'psql discourse -c "grant all privileges on database discourse to discourse;"' || true
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
	Is the server running locally and accepting connections on that socket?
I, [2022-11-25T09:01:01.593736 #1]  INFO -- : 
I, [2022-11-25T09:01:01.593899 #1]  INFO -- : > su postgres -c 'psql discourse -c "alter schema public owner to discourse;"'
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
	Is the server running locally and accepting connections on that socket?
I, [2022-11-25T09:01:01.631113 #1]  INFO -- : 
I, [2022-11-25T09:01:01.631506 #1]  INFO -- : Terminating async processes


FAILED
--------------------
Pups::ExecError: su postgres -c 'psql discourse -c "alter schema public owner to discourse;"' failed with return #<Process::Status: pid 53 exit 2>
Location of failure: /usr/local/lib/ruby/gems/2.7.0/gems/pups-1.1.1/lib/pups/exec_command.rb:117:in `spawn'
exec failed with the params "su postgres -c 'psql $db_name -c \"alter schema public owner to $db_user;\"'"
bootstrap failed with exit code 2
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.
d3281b165492d5420df227f46735618325f4b80a1c3a11643dcef514ec343406
nano@nano:/var/discourse$
1 Like
7

I’ve now tried with sleep 10 and sleep 60 but both give the same output (except the INFO – : > sleep 60 line)

1 Like
8

It looks like you might be checking the file on the host: you need to enter the app and check it within there.

1 Like
9

Mea culpa, and you’re right here the permissions seem to be quite wrong:

root@nano-app:/var/www/discourse# sudo ls -l /etc/ssl/private/ssl-cert-snakeoil.key
sudo: /etc/sudo.conf is owned by uid 1000, should be 0
sudo: /etc/sudo.conf is owned by uid 1000, should be 0
sudo: error in /etc/sudo.conf, line 0 while loading plugin "sudoers_policy"
sudo: /usr/lib/sudo/sudoers.so must be owned by uid 0
sudo: fatal error, unable to load plugins
root@nano-app:/var/www/discourse#

What do you suggest to fix this?

1 Like
10

That looks like a different thing: you’re already root, so you don’t need sudo in this case, so these errors needn’t be a bother.

What kind of installation are you trying to upgrade: is it a standard installation?

1 Like
11

Standard installation, don’t think we have any plugins (its a long time since I installed), its on a local machine, and fairly small (500 MB or so Backup)

I checked again without sudo and the owner is discourse:

root@nano-app:/var/www/discourse# ls -l /etc/ssl/private/ssl-cert-snakeoil.key
-rw-r----- 1 discourse ssl-cert 1704 Jul 20 00:14 /etc/ssl/private/ssl-cert-snakeoil.key

cannot run ./launcher enter app without sudo though

1 Like
12

It might be interesting to know how that file got the wrong ownership, but if you enter the app you should be able to

chown root /etc/ssl/private/ssl-cert-snakeoil.key
exit

and then a rebuild should go well.

1 Like
13

Thanks Ed!

I’ve changed the permission now:

root@nano-app:/var/www/discourse# ls -l  /etc/ssl/private/ssl-cert-snakeoil.key
-rw-r----- 1 root ssl-cert 1704 Jul 20 00:14 /etc/ssl/private/ssl-cert-snakeoil.key

But the rebuild still fails with

sudo ./launcher rebuild app
x86_64 arch detected.
WARNING: containers/app.yml file is world-readable. You can secure this file by running: chmod o-rwx containers/app.yml
Ensuring launcher is up to date
Fetching origin
Launcher is up-to-date
Stopping old container
+ /usr/bin/docker stop -t 60 app
app
2.0.20220818-0047: Pulling from discourse/base
Digest: sha256:7734701087766821ffb2ddcef423754798bd345c2ac0d550131c6e6905c268e8
Status: Image is up to date for discourse/base:2.0.20220818-0047
docker.io/discourse/base:2.0.20220818-0047
/usr/local/lib/ruby/gems/2.7.0/gems/pups-1.1.1/lib/pups.rb
/usr/local/bin/pups --stdin
I, [2022-11-25T17:38:07.880613 #1]  INFO -- : Reading from stdin
I, [2022-11-25T17:38:07.884660 #1]  INFO -- : > locale-gen $LANG && update-locale
I, [2022-11-25T17:38:07.915616 #1]  INFO -- : Generating locales (this might take a while)...
Generation complete.

I, [2022-11-25T17:38:07.915833 #1]  INFO -- : > mkdir -p /shared/postgres_run
I, [2022-11-25T17:38:07.918412 #1]  INFO -- : 
I, [2022-11-25T17:38:07.918633 #1]  INFO -- : > chown postgres:postgres /shared/postgres_run
I, [2022-11-25T17:38:07.920622 #1]  INFO -- : 
I, [2022-11-25T17:38:07.920765 #1]  INFO -- : > chmod 775 /shared/postgres_run
I, [2022-11-25T17:38:07.922483 #1]  INFO -- : 
I, [2022-11-25T17:38:07.922608 #1]  INFO -- : > rm -fr /var/run/postgresql
I, [2022-11-25T17:38:07.924505 #1]  INFO -- : 
I, [2022-11-25T17:38:07.924632 #1]  INFO -- : > ln -s /shared/postgres_run /var/run/postgresql
I, [2022-11-25T17:38:07.926371 #1]  INFO -- : 
I, [2022-11-25T17:38:07.926506 #1]  INFO -- : > socat /dev/null UNIX-CONNECT:/shared/postgres_run/.s.PGSQL.5432 || exit 0 && echo postgres already running stop container ; exit 1
2022/11/25 17:38:07 socat[19] E connect(6, AF=1 "/shared/postgres_run/.s.PGSQL.5432", 36): No such file or directory
I, [2022-11-25T17:38:07.930281 #1]  INFO -- : 
I, [2022-11-25T17:38:07.930387 #1]  INFO -- : > rm -fr /shared/postgres_run/.s*
I, [2022-11-25T17:38:07.932835 #1]  INFO -- : 
I, [2022-11-25T17:38:07.932934 #1]  INFO -- : > rm -fr /shared/postgres_run/*.pid
I, [2022-11-25T17:38:07.935169 #1]  INFO -- : 
I, [2022-11-25T17:38:07.935269 #1]  INFO -- : > mkdir -p /shared/postgres_run/13-main.pg_stat_tmp
I, [2022-11-25T17:38:07.937189 #1]  INFO -- : 
I, [2022-11-25T17:38:07.937320 #1]  INFO -- : > chown postgres:postgres /shared/postgres_run/13-main.pg_stat_tmp
I, [2022-11-25T17:38:07.939169 #1]  INFO -- : 
I, [2022-11-25T17:38:07.943171 #1]  INFO -- : File > /etc/service/postgres/run  chmod: +x  chown: 
I, [2022-11-25T17:38:07.947176 #1]  INFO -- : File > /etc/service/postgres/log/run  chmod: +x  chown: 
I, [2022-11-25T17:38:07.951156 #1]  INFO -- : File > /etc/runit/3.d/99-postgres  chmod: +x  chown: 
I, [2022-11-25T17:38:07.955181 #1]  INFO -- : File > /root/upgrade_postgres  chmod: +x  chown: 
I, [2022-11-25T17:38:07.955375 #1]  INFO -- : > chown -R root /var/lib/postgresql/13/main
I, [2022-11-25T17:38:08.024558 #1]  INFO -- : 
I, [2022-11-25T17:38:08.024689 #1]  INFO -- : > [ ! -e /shared/postgres_data ] && install -d -m 0755 -o postgres -g postgres /shared/postgres_data && sudo -E -u postgres /usr/lib/postgresql/13/bin/initdb -D /shared/postgres_data || exit 0
I, [2022-11-25T17:38:08.026753 #1]  INFO -- : 
I, [2022-11-25T17:38:08.026809 #1]  INFO -- : > chown -R postgres:postgres /shared/postgres_data
I, [2022-11-25T17:38:08.038131 #1]  INFO -- : 
I, [2022-11-25T17:38:08.038249 #1]  INFO -- : > chown -R postgres:postgres /var/run/postgresql
I, [2022-11-25T17:38:08.040545 #1]  INFO -- : 
I, [2022-11-25T17:38:08.040726 #1]  INFO -- : > /root/upgrade_postgres
I, [2022-11-25T17:38:08.044864 #1]  INFO -- : 
I, [2022-11-25T17:38:08.045002 #1]  INFO -- : > rm /root/upgrade_postgres
I, [2022-11-25T17:38:08.046952 #1]  INFO -- : 
I, [2022-11-25T17:38:08.047197 #1]  INFO -- : Replacing data_directory = '/var/lib/postgresql/13/main' with data_directory = '/shared/postgres_data' in /etc/postgresql/13/main/postgresql.conf
I, [2022-11-25T17:38:08.047658 #1]  INFO -- : Replacing (?-mix:#?listen_addresses *=.*) with listen_addresses = '*' in /etc/postgresql/13/main/postgresql.conf
I, [2022-11-25T17:38:08.047886 #1]  INFO -- : Replacing (?-mix:#?synchronous_commit *=.*) with synchronous_commit = $db_synchronous_commit in /etc/postgresql/13/main/postgresql.conf
I, [2022-11-25T17:38:08.048083 #1]  INFO -- : Replacing (?-mix:#?shared_buffers *=.*) with shared_buffers = $db_shared_buffers in /etc/postgresql/13/main/postgresql.conf
I, [2022-11-25T17:38:08.048276 #1]  INFO -- : Replacing (?-mix:#?work_mem *=.*) with work_mem = $db_work_mem in /etc/postgresql/13/main/postgresql.conf
I, [2022-11-25T17:38:08.048520 #1]  INFO -- : Replacing (?-mix:#?default_text_search_config *=.*) with default_text_search_config = '$db_default_text_search_config' in /etc/postgresql/13/main/postgresql.conf
I, [2022-11-25T17:38:08.048721 #1]  INFO -- : > install -d -m 0755 -o postgres -g postgres /shared/postgres_backup
I, [2022-11-25T17:38:08.051177 #1]  INFO -- : 
I, [2022-11-25T17:38:08.051547 #1]  INFO -- : Replacing (?-mix:#?checkpoint_segments *=.*) with checkpoint_segments = $db_checkpoint_segments in /etc/postgresql/13/main/postgresql.conf
I, [2022-11-25T17:38:08.051777 #1]  INFO -- : Replacing (?-mix:#?logging_collector *=.*) with logging_collector = $db_logging_collector in /etc/postgresql/13/main/postgresql.conf
I, [2022-11-25T17:38:08.051980 #1]  INFO -- : Replacing (?-mix:#?log_min_duration_statement *=.*) with log_min_duration_statement = $db_log_min_duration_statement in /etc/postgresql/13/main/postgresql.conf
I, [2022-11-25T17:38:08.052185 #1]  INFO -- : Replacing (?-mix:^#local +replication +postgres +peer$) with local replication postgres  peer in /etc/postgresql/13/main/pg_hba.conf
I, [2022-11-25T17:38:08.052376 #1]  INFO -- : Replacing (?-mix:^host.*all.*all.*127.*$) with host all all 0.0.0.0/0 md5 in /etc/postgresql/13/main/pg_hba.conf
I, [2022-11-25T17:38:08.052547 #1]  INFO -- : Replacing (?-mix:^host.*all.*all.*::1\/128.*$) with host all all ::/0 md5 in /etc/postgresql/13/main/pg_hba.conf
I, [2022-11-25T17:38:08.052696 #1]  INFO -- : > HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main
I, [2022-11-25T17:38:08.054007 #1]  INFO -- : > sleep 60
2022-11-25 17:38:08.085 UTC [42] FATAL:  private key file "/etc/ssl/private/ssl-cert-snakeoil.key" must be owned by the database user or root
2022-11-25 17:38:08.085 UTC [42] LOG:  database system is shut down
I, [2022-11-25T17:39:08.056117 #1]  INFO -- : 
I, [2022-11-25T17:39:08.056699 #1]  INFO -- : > su postgres -c 'createdb discourse' || true
createdb: error: could not connect to database template1: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
	Is the server running locally and accepting connections on that socket?
I, [2022-11-25T17:39:08.105263 #1]  INFO -- : 
I, [2022-11-25T17:39:08.105431 #1]  INFO -- : > su postgres -c 'psql discourse -c "create user discourse;"' || true
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
	Is the server running locally and accepting connections on that socket?
I, [2022-11-25T17:39:08.142885 #1]  INFO -- : 
I, [2022-11-25T17:39:08.143051 #1]  INFO -- : > su postgres -c 'psql discourse -c "grant all privileges on database discourse to discourse;"' || true
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
	Is the server running locally and accepting connections on that socket?
I, [2022-11-25T17:39:08.180416 #1]  INFO -- : 
I, [2022-11-25T17:39:08.180582 #1]  INFO -- : > su postgres -c 'psql discourse -c "alter schema public owner to discourse;"'
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
	Is the server running locally and accepting connections on that socket?
I, [2022-11-25T17:39:08.217735 #1]  INFO -- : 
I, [2022-11-25T17:39:08.218076 #1]  INFO -- : Terminating async processes


FAILED
--------------------
Pups::ExecError: su postgres -c 'psql discourse -c "alter schema public owner to discourse;"' failed with return #<Process::Status: pid 54 exit 2>
Location of failure: /usr/local/lib/ruby/gems/2.7.0/gems/pups-1.1.1/lib/pups/exec_command.rb:117:in `spawn'
exec failed with the params "su postgres -c 'psql $db_name -c \"alter schema public owner to $db_user;\"'"
bootstrap failed with exit code 2
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.

As to how the snakeoil.key permission changed, I do not know. There was some faffing around in the var/discourse folder changing permissions from user to root, but I was nowhere close to the /etc folder structure

I have cxhecked docker status and get some warnings here:

docker.service - Docker Application Container Engine
     Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2022-11-25 18:47:58 CET; 5min ago
TriggeredBy: â—Ź docker.socket
       Docs: https://docs.docker.com
   Main PID: 1793 (dockerd)
      Tasks: 39
     Memory: 118.1M
     CGroup: /system.slice/docker.service
             ├─1793 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
             ├─2128 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 443 -container-ip xxx.xx.x.x  -container-port 443
             ├─2134 /usr/bin/docker-proxy -proto tcp -host-ip :: -host-port 443 -container-ip xxx.xx.x.x -container-port 443
             ├─2148 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 80 -container-ip xxx.xx.x.x  -container-port 80
             └─2155 /usr/bin/docker-proxy -proto tcp -host-ip :: -host-port 80 -container-ip xxx.xx.x.x  -container-port 80

nov. 25 18:47:57 nano dockerd[1793]: time="2022-11-25T18:47:57.319069348+01:00" level=warning msg="Your kernel does not support cgroup blkio weight"
nov. 25 18:47:57 nano dockerd[1793]: time="2022-11-25T18:47:57.319076217+01:00" level=warning msg="Your kernel does not support cgroup blkio weight_device"
nov. 25 18:47:57 nano dockerd[1793]: time="2022-11-25T18:47:57.319414040+01:00" level=info msg="Loading containers: start."
nov. 25 18:47:57 nano dockerd[1793]: time="2022-11-25T18:47:57.679737834+01:00" level=info msg="Default bridge (docker0) is assigned with an IP address xxx.xx.x.x . Daemon option --bip can be used to set a preferred IP address"
nov. 25 18:47:57 nano dockerd[1793]: time="2022-11-25T18:47:57.770958200+01:00" level=warning msg="auplink flush failed: " error="exec: \"auplink\": executable file not found in $PATH" method=Unmount storage-driver=aufs
nov. 25 18:47:58 nano dockerd[1793]: time="2022-11-25T18:47:58.375910870+01:00" level=info msg="Loading containers: done."
nov. 25 18:47:58 nano dockerd[1793]: time="2022-11-25T18:47:58.391980297+01:00" level=info msg="Docker daemon" commit=3056208 graphdriver(s)=aufs version=20.10.21
nov. 25 18:47:58 nano dockerd[1793]: time="2022-11-25T18:47:58.392379728+01:00" level=info msg="Daemon has completed initialization"
nov. 25 18:47:58 nano systemd[1]: Started Docker Application Container Engine.
nov. 25 18:47:58 nano dockerd[1793]: time="2022-11-25T18:47:58.470697756+01:00" level=info msg="API listen on /run/docker.sock"
1 Like
14

Hmm, sorry, no more ideas at present.

As a last resort, one does see advice to take a backup, start a fresh install, and restore the backup. I’ve never done this myself!

1 Like
15

Could it be one of the enclosing folders with wrong permissions?
for the filetree i get:

drwxr-xr-x 178 root root 12288 des.   1 09:56 .
root@nano:/etc# cd ssl
root@nano:/etc/ssl# ls -ld
drwxr-xr-x 4 root root 4096 juli   7 06:36 .
root@nano:/etc/ssl#  cd private/
root@nano:/etc/ssl/private# ls -ld
drwx--x--- 2 root ssl-cert 4096 jan.  11  2017 .
1 Like
16

I don’t think those permissions are wrong. But that’s a very old datestamp. I would have expected these things to come in the shipped containers so it seems possible you are very much behind in updates, somehow.

This is what I see (from outside the container)

/var/discourse# docker image ls 
REPOSITORY            TAG                 IMAGE ID            CREATED             SIZE
local_discourse/app   latest              1727731a0700        4 months ago        3.19GB
discourse/base        2.0.20220720-0049   13ff18ca36a6        4 months ago        2.65GB
1 Like
17

So this might not be helpful for this particular issue (or maybe it is?) but maybe someone else reading it could benefit.

I was also having trouble updating my site last night, with basically the same error message:

Pups::ExecError: cd /var/www/discourse ....

I isolated the cause to two plugins I wrote. I was using the Discourse Signature plugin as boilerplate code to learn how to write plugins at the time. Because I was following it as an example, my two plugins also had this issue that was blocking update:

Commenting out the plugin lines in my yml file worked. Removing the same line on the plugin repos also fixed the issue and allowed me to update.

1 Like
18

Unfortunately, that’s not really the same error message: it’s the summary that a command failed, but the reason for the command failing is earlier in the log file, possibly quite a lot earlier. You’ll notice it’s common here to ask for more of the log file - it’s common to paste too little to do any diagnosis.

1 Like
19

Yes, you’re right, thats fair. My phrasing is not ideal.The relevant log lines were indeed above the final error being displayed but the halting point is the same

In my case the point of failure happened as the plugin repos were being processed

2 Likes