Hi
I am having trouble rescuing a failed upgrade on a Ubuntu 20.04.5 LTS (Focal Fossa)" local machine. Following the failed update, from the command line I ran first
sudo apt-get update
sudo apt-get upgrade
cd /var/discourse
git pull
sudo ./launcher rebuild app
the last part of the output was
- dist/javascripts/workbox/workbox-expiration.dev.js: 20.83 KB (5.48 KB gzipped)
- dist/javascripts/workbox/workbox-expiration.prod.js: 2.82 KB (1.22 KB gzipped)
- dist/javascripts/workbox/workbox-routing.dev.js: 31.54 KB (7.58 KB gzipped)
- dist/javascripts/workbox/workbox-routing.prod.js: 3.32 KB (1.44 KB gzipped)
- dist/javascripts/workbox/workbox-strategies.dev.js: 34.65 KB (4.79 KB gzipped)
- dist/javascripts/workbox/workbox-strategies.prod.js: 4.75 KB (1.16 KB gzipped)
- dist/javascripts/workbox/workbox-sw.js: 1.3 KB (741 B gzipped)
Done in 90.53s.
I, [2022-11-24T15:11:00.523692 #1] INFO -- : Terminating async processes
I, [2022-11-24T15:11:00.523748 #1] INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main pid: 42
I, [2022-11-24T15:11:00.523808 #1] INFO -- : Sending TERM to exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf pid: 103
103:signal-handler (1669302660) Received SIGTERM scheduling shutdown...
2022-11-24 15:11:00.524 UTC [42] LOG: received fast shutdown request
2022-11-24 15:11:00.540 UTC [42] LOG: aborting any active transactions
2022-11-24 15:11:00.542 UTC [42] LOG: background worker "logical replication launcher" (PID 51) exited with exit code 1
2022-11-24 15:11:00.544 UTC [46] LOG: shutting down
103:M 24 Nov 2022 15:11:00.546 # User requested shutdown...
103:M 24 Nov 2022 15:11:00.546 * Saving the final RDB snapshot before exiting.
103:M 24 Nov 2022 15:11:00.597 * DB saved on disk
103:M 24 Nov 2022 15:11:00.597 # Redis is now ready to exit, bye bye...
2022-11-24 15:11:00.680 UTC [42] LOG: database system is shut down
FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && su discourse -c 'bundle exec rake themes:update assets:precompile' failed with return #<Process::Status: pid 1415 exit 1>
Location of failure: /usr/local/lib/ruby/gems/2.7.0/gems/pups-1.1.1/lib/pups/exec_command.rb:117:in `spawn'
exec failed with the params {"cd"=>"$home", "hook"=>"assets_precompile", "cmd"=>["su discourse -c 'bundle exec rake themes:update assets:precompile'"]}
bootstrap failed with exit code 1
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.
abfc5da15ae62310ad82ccb495992f76ef7b9a87b7bb024ca4bf327ea993bc54
I then tried
sudo apt-get dist-upgrade
cd /var/discourse
git pull
sudo ./launcher rebuild app
and the output is now
sudo ./launcher rebuild app
WARNING: Docker version 17.05.0-ce deprecated, recommend upgrade to 17.06.2 or newer.
x86_64 arch detected.
WARNING: containers/app.yml file is world-readable. You can secure this file by running: chmod o-rwx containers/app.yml
Ensuring launcher is up to date
Fetching origin
Launcher is up-to-date
Stopping old container
+ /usr/bin/docker stop -t 60 app
app
2.0.20220818-0047: Pulling from discourse/base
Digest: sha256:7734701087766821ffb2ddcef423754798bd345c2ac0d550131c6e6905c268e8
Status: Image is up to date for discourse/base:2.0.20220818-0047
/usr/local/lib/ruby/gems/2.7.0/gems/pups-1.1.1/lib/pups.rb
/usr/local/bin/pups --stdin
I, [2022-11-24T17:05:57.385657 #1] INFO -- : Reading from stdin
I, [2022-11-24T17:05:57.389797 #1] INFO -- : > locale-gen $LANG && update-locale
I, [2022-11-24T17:05:57.422186 #1] INFO -- : Generating locales (this might take a while)...
Generation complete.
I, [2022-11-24T17:05:57.422436 #1] INFO -- : > mkdir -p /shared/postgres_run
I, [2022-11-24T17:05:57.425262 #1] INFO -- :
I, [2022-11-24T17:05:57.425450 #1] INFO -- : > chown postgres:postgres /shared/postgres_run
I, [2022-11-24T17:05:57.427977 #1] INFO -- :
I, [2022-11-24T17:05:57.428106 #1] INFO -- : > chmod 775 /shared/postgres_run
I, [2022-11-24T17:05:57.429878 #1] INFO -- :
I, [2022-11-24T17:05:57.429989 #1] INFO -- : > rm -fr /var/run/postgresql
I, [2022-11-24T17:05:57.431844 #1] INFO -- :
I, [2022-11-24T17:05:57.431957 #1] INFO -- : > ln -s /shared/postgres_run /var/run/postgresql
I, [2022-11-24T17:05:57.433756 #1] INFO -- :
I, [2022-11-24T17:05:57.433876 #1] INFO -- : > socat /dev/null UNIX-CONNECT:/shared/postgres_run/.s.PGSQL.5432 || exit 0 && echo postgres already running stop container ; exit 1
2022/11/24 17:05:57 socat[19] E connect(6, AF=1 "/shared/postgres_run/.s.PGSQL.5432", 36): No such file or directory
I, [2022-11-24T17:05:57.439214 #1] INFO -- :
I, [2022-11-24T17:05:57.439321 #1] INFO -- : > rm -fr /shared/postgres_run/.s*
I, [2022-11-24T17:05:57.441654 #1] INFO -- :
I, [2022-11-24T17:05:57.441752 #1] INFO -- : > rm -fr /shared/postgres_run/*.pid
I, [2022-11-24T17:05:57.444021 #1] INFO -- :
I, [2022-11-24T17:05:57.444120 #1] INFO -- : > mkdir -p /shared/postgres_run/13-main.pg_stat_tmp
I, [2022-11-24T17:05:57.446040 #1] INFO -- :
I, [2022-11-24T17:05:57.446171 #1] INFO -- : > chown postgres:postgres /shared/postgres_run/13-main.pg_stat_tmp
I, [2022-11-24T17:05:57.448063 #1] INFO -- :
I, [2022-11-24T17:05:57.452270 #1] INFO -- : File > /etc/service/postgres/run chmod: +x chown:
I, [2022-11-24T17:05:57.456198 #1] INFO -- : File > /etc/service/postgres/log/run chmod: +x chown:
I, [2022-11-24T17:05:57.460141 #1] INFO -- : File > /etc/runit/3.d/99-postgres chmod: +x chown:
I, [2022-11-24T17:05:57.464080 #1] INFO -- : File > /root/upgrade_postgres chmod: +x chown:
I, [2022-11-24T17:05:57.464267 #1] INFO -- : > chown -R root /var/lib/postgresql/13/main
I, [2022-11-24T17:05:57.537077 #1] INFO -- :
I, [2022-11-24T17:05:57.537178 #1] INFO -- : > [ ! -e /shared/postgres_data ] && install -d -m 0755 -o postgres -g postgres /shared/postgres_data && sudo -E -u postgres /usr/lib/postgresql/13/bin/initdb -D /shared/postgres_data || exit 0
I, [2022-11-24T17:05:57.538742 #1] INFO -- :
I, [2022-11-24T17:05:57.538799 #1] INFO -- : > chown -R postgres:postgres /shared/postgres_data
I, [2022-11-24T17:05:57.550302 #1] INFO -- :
I, [2022-11-24T17:05:57.550394 #1] INFO -- : > chown -R postgres:postgres /var/run/postgresql
I, [2022-11-24T17:05:57.552387 #1] INFO -- :
I, [2022-11-24T17:05:57.552528 #1] INFO -- : > /root/upgrade_postgres
I, [2022-11-24T17:05:57.556422 #1] INFO -- :
I, [2022-11-24T17:05:57.556533 #1] INFO -- : > rm /root/upgrade_postgres
I, [2022-11-24T17:05:57.558395 #1] INFO -- :
I, [2022-11-24T17:05:57.558734 #1] INFO -- : Replacing data_directory = '/var/lib/postgresql/13/main' with data_directory = '/shared/postgres_data' in /etc/postgresql/13/main/postgresql.conf
I, [2022-11-24T17:05:57.559138 #1] INFO -- : Replacing (?-mix:#?listen_addresses *=.*) with listen_addresses = '*' in /etc/postgresql/13/main/postgresql.conf
I, [2022-11-24T17:05:57.559350 #1] INFO -- : Replacing (?-mix:#?synchronous_commit *=.*) with synchronous_commit = $db_synchronous_commit in /etc/postgresql/13/main/postgresql.conf
I, [2022-11-24T17:05:57.559550 #1] INFO -- : Replacing (?-mix:#?shared_buffers *=.*) with shared_buffers = $db_shared_buffers in /etc/postgresql/13/main/postgresql.conf
I, [2022-11-24T17:05:57.559748 #1] INFO -- : Replacing (?-mix:#?work_mem *=.*) with work_mem = $db_work_mem in /etc/postgresql/13/main/postgresql.conf
I, [2022-11-24T17:05:57.559945 #1] INFO -- : Replacing (?-mix:#?default_text_search_config *=.*) with default_text_search_config = '$db_default_text_search_config' in /etc/postgresql/13/main/postgresql.conf
I, [2022-11-24T17:05:57.560122 #1] INFO -- : > install -d -m 0755 -o postgres -g postgres /shared/postgres_backup
I, [2022-11-24T17:05:57.562571 #1] INFO -- :
I, [2022-11-24T17:05:57.562855 #1] INFO -- : Replacing (?-mix:#?checkpoint_segments *=.*) with checkpoint_segments = $db_checkpoint_segments in /etc/postgresql/13/main/postgresql.conf
I, [2022-11-24T17:05:57.563086 #1] INFO -- : Replacing (?-mix:#?logging_collector *=.*) with logging_collector = $db_logging_collector in /etc/postgresql/13/main/postgresql.conf
I, [2022-11-24T17:05:57.563295 #1] INFO -- : Replacing (?-mix:#?log_min_duration_statement *=.*) with log_min_duration_statement = $db_log_min_duration_statement in /etc/postgresql/13/main/postgresql.conf
I, [2022-11-24T17:05:57.563506 #1] INFO -- : Replacing (?-mix:^#local +replication +postgres +peer$) with local replication postgres peer in /etc/postgresql/13/main/pg_hba.conf
I, [2022-11-24T17:05:57.563702 #1] INFO -- : Replacing (?-mix:^host.*all.*all.*127.*$) with host all all 0.0.0.0/0 md5 in /etc/postgresql/13/main/pg_hba.conf
I, [2022-11-24T17:05:57.563874 #1] INFO -- : Replacing (?-mix:^host.*all.*all.*::1\/128.*$) with host all all ::/0 md5 in /etc/postgresql/13/main/pg_hba.conf
I, [2022-11-24T17:05:57.564025 #1] INFO -- : > HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main
I, [2022-11-24T17:05:57.565221 #1] INFO -- : > sleep 5
2022-11-24 17:05:57.591 UTC [42] FATAL: private key file "/etc/ssl/private/ssl-cert-snakeoil.key" must be owned by the database user or root
2022-11-24 17:05:57.591 UTC [42] LOG: database system is shut down
I, [2022-11-24T17:06:02.567410 #1] INFO -- :
I, [2022-11-24T17:06:02.567949 #1] INFO -- : > su postgres -c 'createdb discourse' || true
createdb: error: could not connect to database template1: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
Is the server running locally and accepting connections on that socket?
I, [2022-11-24T17:06:02.612636 #1] INFO -- :
I, [2022-11-24T17:06:02.612780 #1] INFO -- : > su postgres -c 'psql discourse -c "create user discourse;"' || true
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
Is the server running locally and accepting connections on that socket?
I, [2022-11-24T17:06:02.649900 #1] INFO -- :
I, [2022-11-24T17:06:02.650042 #1] INFO -- : > su postgres -c 'psql discourse -c "grant all privileges on database discourse to discourse;"' || true
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
Is the server running locally and accepting connections on that socket?
I, [2022-11-24T17:06:02.688244 #1] INFO -- :
I, [2022-11-24T17:06:02.688412 #1] INFO -- : > su postgres -c 'psql discourse -c "alter schema public owner to discourse;"'
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
Is the server running locally and accepting connections on that socket?
I, [2022-11-24T17:06:02.725728 #1] INFO -- :
I, [2022-11-24T17:06:02.725969 #1] INFO -- : Terminating async processes
FAILED
--------------------
Pups::ExecError: su postgres -c 'psql discourse -c "alter schema public owner to discourse;"' failed with return #<Process::Status: pid 54 exit 2>
Location of failure: /usr/local/lib/ruby/gems/2.7.0/gems/pups-1.1.1/lib/pups/exec_command.rb:117:in `spawn'
exec failed with the params "su postgres -c 'psql $db_name -c \"alter schema public owner to $db_user;\"'"
bootstrap failed with exit code 2
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.
8c252b2f3146df27193b8a6966b36a75230460f92397cd206aa5540082090660
Can anyone help?
