HELP: Discourse Update broke app -- fails to rebuild

littleviolette · January 8, 2023, 10:15am

When I tried to update to the latest version today, I encountered the follow error:

fatal: unable to access 'https://github.com/discourse/discourse.git/': Could not resolve host: github.com

I am running the app on a DigitalOcean Droplet Ubuntu 18.04.6 LTS (GNU/Linux 3.13.0-52-generic x86_64)

I am not familiar with docker or server-side programming. I would really appreciate your help!

Here is the error code in its entirety:

root@forum:/var/discourse# ./launcher rebuild app
x86_64 arch detected.
WARNING: containers/app.yml file is world-readable. You can secure this file by running: chmod o-rwx containers/app.yml
Ensuring launcher is up to date
Fetching origin
Launcher is up-to-date
Stopping old container
+ /usr/bin/docker stop -t 60 app
app
2.0.20221221-0050: Pulling from discourse/base
Digest: sha256:7ac6489e34a0f40f32457bffaa331940a98447b46451a6a1279a79868ee5420b
Status: Image is up to date for discourse/base:2.0.20221221-0050
docker.io/discourse/base:2.0.20221221-0050
/usr/local/lib/ruby/gems/3.1.0/gems/pups-1.1.1/lib/pups.rb
/usr/local/bin/pups --stdin
I, [2023-01-08T10:02:23.027325 #1]  INFO -- : Reading from stdin
I, [2023-01-08T10:02:23.037278 #1]  INFO -- : > locale-gen $LANG && update-locale
I, [2023-01-08T10:02:23.114042 #1]  INFO -- : Generating locales (this might take a while)...
Generation complete.

I, [2023-01-08T10:02:23.114335 #1]  INFO -- : > mkdir -p /shared/postgres_run
I, [2023-01-08T10:02:23.121579 #1]  INFO -- : 
I, [2023-01-08T10:02:23.122417 #1]  INFO -- : > chown postgres:postgres /shared/postgres_run
I, [2023-01-08T10:02:23.126437 #1]  INFO -- : 
I, [2023-01-08T10:02:23.126670 #1]  INFO -- : > chmod 775 /shared/postgres_run
I, [2023-01-08T10:02:23.129755 #1]  INFO -- : 
I, [2023-01-08T10:02:23.129992 #1]  INFO -- : > rm -fr /var/run/postgresql
I, [2023-01-08T10:02:23.133413 #1]  INFO -- : 
I, [2023-01-08T10:02:23.133646 #1]  INFO -- : > ln -s /shared/postgres_run /var/run/postgresql
I, [2023-01-08T10:02:23.137014 #1]  INFO -- : 
I, [2023-01-08T10:02:23.137231 #1]  INFO -- : > socat /dev/null UNIX-CONNECT:/shared/postgres_run/.s.PGSQL.5432 || exit 0 && echo postgres already running stop container ; exit 1
2023/01/08 10:02:23 socat[18] E connect(6, AF=1 "/shared/postgres_run/.s.PGSQL.5432", 36): No such file or directory
I, [2023-01-08T10:02:23.163463 #1]  INFO -- : 
I, [2023-01-08T10:02:23.163683 #1]  INFO -- : > rm -fr /shared/postgres_run/.s*
I, [2023-01-08T10:02:23.167367 #1]  INFO -- : 
I, [2023-01-08T10:02:23.167548 #1]  INFO -- : > rm -fr /shared/postgres_run/*.pid
I, [2023-01-08T10:02:23.170830 #1]  INFO -- : 
I, [2023-01-08T10:02:23.171037 #1]  INFO -- : > mkdir -p /shared/postgres_run/13-main.pg_stat_tmp
I, [2023-01-08T10:02:23.174338 #1]  INFO -- : 
I, [2023-01-08T10:02:23.174539 #1]  INFO -- : > chown postgres:postgres /shared/postgres_run/13-main.pg_stat_tmp
I, [2023-01-08T10:02:23.177456 #1]  INFO -- : 
I, [2023-01-08T10:02:23.183562 #1]  INFO -- : File > /etc/service/postgres/run  chmod: +x  chown: 
I, [2023-01-08T10:02:23.189225 #1]  INFO -- : File > /etc/service/postgres/log/run  chmod: +x  chown: 
I, [2023-01-08T10:02:23.195374 #1]  INFO -- : File > /etc/runit/3.d/99-postgres  chmod: +x  chown: 
I, [2023-01-08T10:02:23.201124 #1]  INFO -- : File > /root/upgrade_postgres  chmod: +x  chown: 
I, [2023-01-08T10:02:23.201450 #1]  INFO -- : > chown -R root /var/lib/postgresql/13/main
I, [2023-01-08T10:02:23.616900 #1]  INFO -- : 
I, [2023-01-08T10:02:23.617006 #1]  INFO -- : > [ ! -e /shared/postgres_data ] && install -d -m 0755 -o postgres -g postgres /shared/postgres_data && sudo -E -u postgres /usr/lib/postgresql/13/bin/initdb -D /shared/postgres_data || exit 0
I, [2023-01-08T10:02:23.619878 #1]  INFO -- : 
I, [2023-01-08T10:02:23.619967 #1]  INFO -- : > chown -R postgres:postgres /shared/postgres_data
I, [2023-01-08T10:02:23.673160 #1]  INFO -- : 
I, [2023-01-08T10:02:23.673318 #1]  INFO -- : > chown -R postgres:postgres /var/run/postgresql
I, [2023-01-08T10:02:23.677342 #1]  INFO -- : 
I, [2023-01-08T10:02:23.677523 #1]  INFO -- : > /root/upgrade_postgres
I, [2023-01-08T10:02:23.683764 #1]  INFO -- : 
I, [2023-01-08T10:02:23.683966 #1]  INFO -- : > rm /root/upgrade_postgres
I, [2023-01-08T10:02:23.686507 #1]  INFO -- : 
I, [2023-01-08T10:02:23.687630 #1]  INFO -- : Replacing data_directory = '/var/lib/postgresql/13/main' with data_directory = '/shared/postgres_data' in /etc/postgresql/13/main/postgresql.conf
I, [2023-01-08T10:02:23.688285 #1]  INFO -- : Replacing (?-mix:#?listen_addresses *=.*) with listen_addresses = '*' in /etc/postgresql/13/main/postgresql.conf
I, [2023-01-08T10:02:23.688707 #1]  INFO -- : Replacing (?-mix:#?synchronous_commit *=.*) with synchronous_commit = $db_synchronous_commit in /etc/postgresql/13/main/postgresql.conf
I, [2023-01-08T10:02:23.689138 #1]  INFO -- : Replacing (?-mix:#?shared_buffers *=.*) with shared_buffers = $db_shared_buffers in /etc/postgresql/13/main/postgresql.conf
I, [2023-01-08T10:02:23.689521 #1]  INFO -- : Replacing (?-mix:#?work_mem *=.*) with work_mem = $db_work_mem in /etc/postgresql/13/main/postgresql.conf
I, [2023-01-08T10:02:23.689954 #1]  INFO -- : Replacing (?-mix:#?default_text_search_config *=.*) with default_text_search_config = '$db_default_text_search_config' in /etc/postgresql/13/main/postgresql.conf
I, [2023-01-08T10:02:23.690327 #1]  INFO -- : > install -d -m 0755 -o postgres -g postgres /shared/postgres_backup
I, [2023-01-08T10:02:23.696176 #1]  INFO -- : 
I, [2023-01-08T10:02:23.696554 #1]  INFO -- : Replacing (?-mix:#?checkpoint_segments *=.*) with checkpoint_segments = $db_checkpoint_segments in /etc/postgresql/13/main/postgresql.conf
I, [2023-01-08T10:02:23.696973 #1]  INFO -- : Replacing (?-mix:#?logging_collector *=.*) with logging_collector = $db_logging_collector in /etc/postgresql/13/main/postgresql.conf
I, [2023-01-08T10:02:23.697400 #1]  INFO -- : Replacing (?-mix:#?log_min_duration_statement *=.*) with log_min_duration_statement = $db_log_min_duration_statement in /etc/postgresql/13/main/postgresql.conf
I, [2023-01-08T10:02:23.698030 #1]  INFO -- : Replacing (?-mix:^#local +replication +postgres +peer$) with local replication postgres  peer in /etc/postgresql/13/main/pg_hba.conf
I, [2023-01-08T10:02:23.698390 #1]  INFO -- : Replacing (?-mix:^host.*all.*all.*127.*$) with host all all 0.0.0.0/0 md5 in /etc/postgresql/13/main/pg_hba.conf
I, [2023-01-08T10:02:23.698734 #1]  INFO -- : Replacing (?-mix:^host.*all.*all.*::1\/128.*$) with host all all ::/0 md5 in /etc/postgresql/13/main/pg_hba.conf
I, [2023-01-08T10:02:23.699085 #1]  INFO -- : > HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main
I, [2023-01-08T10:02:23.701001 #1]  INFO -- : > sleep 5
2023-01-08 10:02:23.892 UTC [41] LOG:  starting PostgreSQL 13.9 (Debian 13.9-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
2023-01-08 10:02:23.893 UTC [41] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2023-01-08 10:02:23.893 UTC [41] LOG:  listening on IPv6 address "::", port 5432
2023-01-08 10:02:23.895 UTC [41] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2023-01-08 10:02:23.900 UTC [44] LOG:  database system was shut down at 2023-01-08 10:02:12 UTC
2023-01-08 10:02:23.914 UTC [41] LOG:  database system is ready to accept connections
I, [2023-01-08T10:02:28.704562 #1]  INFO -- : 
I, [2023-01-08T10:02:28.705165 #1]  INFO -- : > su postgres -c 'createdb discourse' || true
2023-01-08 10:02:28.812 UTC [54] postgres@postgres ERROR:  database "discourse" already exists
2023-01-08 10:02:28.812 UTC [54] postgres@postgres STATEMENT:  CREATE DATABASE discourse;
createdb: error: database creation failed: ERROR:  database "discourse" already exists
I, [2023-01-08T10:02:28.815155 #1]  INFO -- : 
I, [2023-01-08T10:02:28.815612 #1]  INFO -- : > su postgres -c 'psql discourse -c "create user discourse;"' || true
2023-01-08 10:02:28.891 UTC [58] postgres@discourse ERROR:  role "discourse" already exists
2023-01-08 10:02:28.891 UTC [58] postgres@discourse STATEMENT:  create user discourse;
ERROR:  role "discourse" already exists
I, [2023-01-08T10:02:28.894445 #1]  INFO -- : 
I, [2023-01-08T10:02:28.894925 #1]  INFO -- : > su postgres -c 'psql discourse -c "grant all privileges on database discourse to discourse;"' || true
I, [2023-01-08T10:02:28.966522 #1]  INFO -- : GRANT

I, [2023-01-08T10:02:28.966978 #1]  INFO -- : > su postgres -c 'psql discourse -c "alter schema public owner to discourse;"'
I, [2023-01-08T10:02:29.031133 #1]  INFO -- : ALTER SCHEMA

I, [2023-01-08T10:02:29.031397 #1]  INFO -- : > su postgres -c 'psql template1 -c "create extension if not exists hstore;"'
NOTICE:  extension "hstore" already exists, skipping
I, [2023-01-08T10:02:29.102970 #1]  INFO -- : CREATE EXTENSION

I, [2023-01-08T10:02:29.103264 #1]  INFO -- : > su postgres -c 'psql template1 -c "create extension if not exists pg_trgm;"'
NOTICE:  extension "pg_trgm" already exists, skipping
I, [2023-01-08T10:02:29.167237 #1]  INFO -- : CREATE EXTENSION

I, [2023-01-08T10:02:29.167506 #1]  INFO -- : > su postgres -c 'psql discourse -c "create extension if not exists hstore;"'
NOTICE:  extension "hstore" already exists, skipping
I, [2023-01-08T10:02:29.233713 #1]  INFO -- : CREATE EXTENSION

I, [2023-01-08T10:02:29.233931 #1]  INFO -- : > su postgres -c 'psql discourse -c "create extension if not exists pg_trgm;"'
NOTICE:  extension "pg_trgm" already exists, skipping
I, [2023-01-08T10:02:29.296228 #1]  INFO -- : CREATE EXTENSION

I, [2023-01-08T10:02:29.296631 #1]  INFO -- : > sudo -u postgres psql discourse
I, [2023-01-08T10:02:29.300403 #1]  INFO -- : update pg_database set encoding = pg_char_to_encoding('UTF8') where datname = 'discourse' AND encoding = pg_char_to_encoding('SQL_ASCII');

I, [2023-01-08T10:02:29.392287 #1]  INFO -- : File > /var/lib/postgresql/take-database-backup  chmod: +x  chown: postgres:postgres
I, [2023-01-08T10:02:29.396585 #1]  INFO -- : File > /var/spool/cron/crontabs/postgres  chmod:   chown: 
I, [2023-01-08T10:02:29.396782 #1]  INFO -- : > echo postgres installed!
I, [2023-01-08T10:02:29.399217 #1]  INFO -- : postgres installed!

I, [2023-01-08T10:02:29.404860 #1]  INFO -- : File > /etc/service/redis/run  chmod: +x  chown: 
I, [2023-01-08T10:02:29.410365 #1]  INFO -- : File > /etc/service/redis/log/run  chmod: +x  chown: 
I, [2023-01-08T10:02:29.415874 #1]  INFO -- : File > /etc/runit/3.d/10-redis  chmod: +x  chown: 
I, [2023-01-08T10:02:29.417263 #1]  INFO -- : Replacing daemonize yes with  in /etc/redis/redis.conf
I, [2023-01-08T10:02:29.418070 #1]  INFO -- : Replacing (?-mix:^pidfile.*$) with  in /etc/redis/redis.conf
I, [2023-01-08T10:02:29.419197 #1]  INFO -- : > install -d -m 0755 -o redis -g redis /shared/redis_data
I, [2023-01-08T10:02:29.422449 #1]  INFO -- : 
I, [2023-01-08T10:02:29.422962 #1]  INFO -- : Replacing (?-mix:^logfile.*$) with logfile "" in /etc/redis/redis.conf
I, [2023-01-08T10:02:29.423516 #1]  INFO -- : Replacing (?-mix:^bind .*$) with  in /etc/redis/redis.conf
I, [2023-01-08T10:02:29.424221 #1]  INFO -- : Replacing (?-mix:^dir .*$) with dir /shared/redis_data in /etc/redis/redis.conf
I, [2023-01-08T10:02:29.424956 #1]  INFO -- : Replacing (?-mix:^protected-mode yes) with protected-mode no in /etc/redis/redis.conf
I, [2023-01-08T10:02:29.425580 #1]  INFO -- : Replacing # io-threads 4 with io-threads $redis_io_threads in /etc/redis/redis.conf
I, [2023-01-08T10:02:29.426230 #1]  INFO -- : > echo redis installed
I, [2023-01-08T10:02:29.428842 #1]  INFO -- : redis installed

I, [2023-01-08T10:02:29.429031 #1]  INFO -- : > cat /etc/redis/redis.conf | grep logfile
I, [2023-01-08T10:02:29.435746 #1]  INFO -- : logfile ""

I, [2023-01-08T10:02:29.435909 #1]  INFO -- : > exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf
I, [2023-01-08T10:02:29.438431 #1]  INFO -- : > sleep 10
102:C 08 Jan 2023 10:02:29.468 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
102:C 08 Jan 2023 10:02:29.468 # Redis version=7.0.7, bits=64, commit=00000000, modified=0, pid=102, just started
102:C 08 Jan 2023 10:02:29.468 # Configuration loaded
102:M 08 Jan 2023 10:02:29.469 * monotonic clock: POSIX clock_gettime
102:M 08 Jan 2023 10:02:29.471 * Running mode=standalone, port=6379.
102:M 08 Jan 2023 10:02:29.471 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
102:M 08 Jan 2023 10:02:29.471 # Server initialized
102:M 08 Jan 2023 10:02:29.471 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
102:M 08 Jan 2023 10:02:29.471 # WARNING You have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo madvise > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled (set to 'madvise' or 'never').
102:M 08 Jan 2023 10:02:29.472 * Loading RDB produced by version 7.0.7
102:M 08 Jan 2023 10:02:29.472 * RDB age 4308 seconds
102:M 08 Jan 2023 10:02:29.472 * RDB memory usage when created 269.18 Mb
102:M 08 Jan 2023 10:02:31.177 * Done loading RDB, keys loaded: 43865, keys expired: 3782.
102:M 08 Jan 2023 10:02:31.177 * DB loaded from disk: 1.706 seconds
102:M 08 Jan 2023 10:02:31.177 * Ready to accept connections
I, [2023-01-08T10:02:39.441240 #1]  INFO -- : 
I, [2023-01-08T10:02:39.442074 #1]  INFO -- : > thpoff echo "thpoff is installed!"
I, [2023-01-08T10:02:39.447520 #1]  INFO -- : thpoff is installed!

I, [2023-01-08T10:02:39.447934 #1]  INFO -- : > /usr/local/bin/ruby -e 'if ENV["DISCOURSE_SMTP_ADDRESS"] == "smtp.example.com"; puts "Aborting! Mail is not configured!"; exit 1; end'
I, [2023-01-08T10:02:39.639965 #1]  INFO -- : 
I, [2023-01-08T10:02:39.640606 #1]  INFO -- : > /usr/local/bin/ruby -e 'if ENV["DISCOURSE_HOSTNAME"] == "discourse.example.com"; puts "Aborting! Domain is not configured!"; exit 1; end'
I, [2023-01-08T10:02:39.784645 #1]  INFO -- : 
I, [2023-01-08T10:02:39.785173 #1]  INFO -- : > /usr/local/bin/ruby -e 'if (ENV["DISCOURSE_CDN_URL"] || "")[0..1] == "//"; puts "Aborting! CDN must have a protocol specified. Once fixed you should rebake your posts now to correct all posts."; exit 1; end'
I, [2023-01-08T10:02:39.919869 #1]  INFO -- : 
I, [2023-01-08T10:02:39.920430 #1]  INFO -- : > rm -f /etc/cron.d/anacron
I, [2023-01-08T10:02:39.924493 #1]  INFO -- : 
I, [2023-01-08T10:02:39.928198 #1]  INFO -- : File > /etc/cron.d/anacron  chmod:   chown: 
I, [2023-01-08T10:02:39.935098 #1]  INFO -- : File > /etc/runit/1.d/copy-env  chmod: +x  chown: 
I, [2023-01-08T10:02:39.941466 #1]  INFO -- : File > /etc/service/unicorn/run  chmod: +x  chown: 
I, [2023-01-08T10:02:39.947679 #1]  INFO -- : File > /etc/service/nginx/run  chmod: +x  chown: 
I, [2023-01-08T10:02:39.953746 #1]  INFO -- : File > /etc/runit/3.d/01-nginx  chmod: +x  chown: 
I, [2023-01-08T10:02:39.959831 #1]  INFO -- : File > /etc/runit/3.d/02-unicorn  chmod: +x  chown: 
I, [2023-01-08T10:02:39.960073 #1]  INFO -- : Replacing # postgres with sv start postgres || exit 1 in /etc/service/unicorn/run
I, [2023-01-08T10:02:39.960480 #1]  INFO -- : > exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf
I, [2023-01-08T10:02:39.963937 #1]  INFO -- : > cd /var/www/discourse && git reset --hard
129:C 08 Jan 2023 10:02:39.973 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
129:C 08 Jan 2023 10:02:39.973 # Redis version=7.0.7, bits=64, commit=00000000, modified=0, pid=129, just started
129:C 08 Jan 2023 10:02:39.973 # Configuration loaded
129:M 08 Jan 2023 10:02:39.973 * monotonic clock: POSIX clock_gettime
129:M 08 Jan 2023 10:02:39.974 # Warning: Could not create server TCP listening socket *:6379: bind: Address already in use
129:M 08 Jan 2023 10:02:39.974 # Failed listening on port 6379 (TCP), aborting.
Updating files: 100% (32637/32637), done.
I, [2023-01-08T10:02:44.070762 #1]  INFO -- : HEAD is now at e775ed1c DEV: Properly include spec examples (#19532)

I, [2023-01-08T10:02:44.070958 #1]  INFO -- : > cd /var/www/discourse && git clean -f
I, [2023-01-08T10:02:44.243596 #1]  INFO -- : 
I, [2023-01-08T10:02:44.243778 #1]  INFO -- : > cd /var/www/discourse && git remote set-branches --add origin main
I, [2023-01-08T10:02:44.248754 #1]  INFO -- : 
I, [2023-01-08T10:02:44.248928 #1]  INFO -- : > cd /var/www/discourse && git remote set-branches origin tests-passed
I, [2023-01-08T10:02:44.253401 #1]  INFO -- : 
I, [2023-01-08T10:02:44.253516 #1]  INFO -- : > cd /var/www/discourse && git fetch --depth 1 origin tests-passed
fatal: unable to access 'https://github.com/discourse/discourse.git/': Could not resolve host: github.com
I, [2023-01-08T10:03:04.333061 #1]  INFO -- : 
I, [2023-01-08T10:03:04.333557 #1]  INFO -- : Terminating async processes
I, [2023-01-08T10:03:04.333603 #1]  INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main pid: 41
I, [2023-01-08T10:03:04.333661 #1]  INFO -- : Sending TERM to exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf pid: 102
2023-01-08 10:03:04.333 UTC [41] LOG:  received fast shutdown request
102:signal-handler (1673172184) Received SIGTERM scheduling shutdown...
2023-01-08 10:03:04.335 UTC [41] LOG:  aborting any active transactions
2023-01-08 10:03:04.340 UTC [41] LOG:  background worker "logical replication launcher" (PID 50) exited with exit code 1
2023-01-08 10:03:04.340 UTC [45] LOG:  shutting down
102:M 08 Jan 2023 10:03:04.375 # User requested shutdown...
102:M 08 Jan 2023 10:03:04.375 * Saving the final RDB snapshot before exiting.
2023-01-08 10:03:04.387 UTC [41] LOG:  database system is shut down
102:M 08 Jan 2023 10:03:05.898 * DB saved on disk
102:M 08 Jan 2023 10:03:05.898 # Redis is now ready to exit, bye bye...


FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && git fetch --depth 1 origin tests-passed failed with return #<Process::Status: pid 140 exit 128>
Location of failure: /usr/local/lib/ruby/gems/3.1.0/gems/pups-1.1.1/lib/pups/exec_command.rb:117:in `spawn'
exec failed with the params {"cd"=>"$home", "hook"=>"code", "cmd"=>["git reset --hard", "git clean -f", "git remote set-branches --add origin main", "git remote set-branches origin $version", "git fetch --depth 1 origin $version", "git checkout $version", "mkdir -p tmp", "chown discourse:www-data tmp", "mkdir -p tmp/pids", "mkdir -p tmp/sockets", "touch tmp/.gitkeep", "mkdir -p                    /shared/log/rails", "bash -c \"touch -a           /shared/log/rails/{production,production_errors,unicorn.stdout,unicorn.stderr,sidekiq}.log\"", "bash -c \"ln    -s           /shared/log/rails/{production,production_errors,unicorn.stdout,unicorn.stderr,sidekiq}.log $home/log\"", "bash -c \"mkdir -p           /shared/{uploads,backups}\"", "bash -c \"ln    -s           /shared/{uploads,backups} $home/public\"", "bash -c \"mkdir -p           /shared/tmp/{backups,restores}\"", "bash -c \"ln    -s           /shared/tmp/{backups,restores} $home/tmp\"", "chown -R discourse:www-data /shared/log/rails /shared/uploads /shared/backups /shared/tmp", "[ ! -d public/plugins ] || find public/plugins/ -maxdepth 1 -xtype l -delete"]}
bootstrap failed with exit code 128
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.
021af933a82ed69c2ea566b7ac1e5cdee45b11d1da210d1359d43e849f4b5025
root@forum:/var/discourse# ```

hiddenseal · January 8, 2023, 10:57am

Hm. I had the exact same issue with a different hosting provider and resolved it by moving to digital ocean.

What geographic location did you chose for the droplet?

pfaffman · January 8, 2023, 12:30pm

Exit code 128 is out of memory. Increase your ram or swap.

RGJ · January 8, 2023, 3:58pm

No it’s not? 128 is a generic error code. Out of memory results in 137 most of the time.

could not resolve host is a network of DNS issue.

littleviolette · January 8, 2023, 5:25pm

@pfaffman After increasing the vCPU from 6 - 8 (the only thing I can upgrade on DigitalOcean), I am still getting the same Exit Code 128 error.

My server’s spec: 8vCPUs/16GB Memory/320GB SSD/6TB Transfer

  System load:              0.2
  Usage of /:               55.6% of 314.85GB
  Memory usage:             2%
  Swap usage:               0%
  Processes:                165

Here is the complete error log:

root@forum:/var/discourse# ./launcher rebuild app
x86_64 arch detected.
WARNING: containers/app.yml file is world-readable. You can secure this file by running: chmod o-rwx containers/app.yml
Ensuring launcher is up to date
Fetching origin
Launcher is up-to-date
Stopping old container
+ /usr/bin/docker stop -t 60 app
app
2.0.20221221-0050: Pulling from discourse/base
Digest: sha256:7ac6489e34a0f40f32457bffaa331940a98447b46451a6a1279a79868ee5420b
Status: Image is up to date for discourse/base:2.0.20221221-0050
docker.io/discourse/base:2.0.20221221-0050
/usr/local/lib/ruby/gems/3.1.0/gems/pups-1.1.1/lib/pups.rb
/usr/local/bin/pups --stdin
I, [2023-01-08T17:17:51.784634 #1]  INFO -- : Reading from stdin
I, [2023-01-08T17:17:51.789935 #1]  INFO -- : > locale-gen $LANG && update-locale
I, [2023-01-08T17:17:51.866583 #1]  INFO -- : Generating locales (this might take a while)...
Generation complete.

I, [2023-01-08T17:17:51.866813 #1]  INFO -- : > mkdir -p /shared/postgres_run
I, [2023-01-08T17:17:51.873210 #1]  INFO -- : 
I, [2023-01-08T17:17:51.874108 #1]  INFO -- : > chown postgres:postgres /shared/postgres_run
I, [2023-01-08T17:17:51.877428 #1]  INFO -- : 
I, [2023-01-08T17:17:51.877600 #1]  INFO -- : > chmod 775 /shared/postgres_run
I, [2023-01-08T17:17:51.880335 #1]  INFO -- : 
I, [2023-01-08T17:17:51.880503 #1]  INFO -- : > rm -fr /var/run/postgresql
I, [2023-01-08T17:17:51.883210 #1]  INFO -- : 
I, [2023-01-08T17:17:51.883316 #1]  INFO -- : > ln -s /shared/postgres_run /var/run/postgresql
I, [2023-01-08T17:17:51.886051 #1]  INFO -- : 
I, [2023-01-08T17:17:51.886148 #1]  INFO -- : > socat /dev/null UNIX-CONNECT:/shared/postgres_run/.s.PGSQL.5432 || exit 0 && echo postgres already running stop container ; exit 1
2023/01/08 17:17:51 socat[18] E connect(6, AF=1 "/shared/postgres_run/.s.PGSQL.5432", 36): No such file or directory
I, [2023-01-08T17:17:51.909808 #1]  INFO -- : 
I, [2023-01-08T17:17:51.909975 #1]  INFO -- : > rm -fr /shared/postgres_run/.s*
I, [2023-01-08T17:17:51.913236 #1]  INFO -- : 
I, [2023-01-08T17:17:51.913423 #1]  INFO -- : > rm -fr /shared/postgres_run/*.pid
I, [2023-01-08T17:17:51.916137 #1]  INFO -- : 
I, [2023-01-08T17:17:51.916229 #1]  INFO -- : > mkdir -p /shared/postgres_run/13-main.pg_stat_tmp
I, [2023-01-08T17:17:51.918500 #1]  INFO -- : 
I, [2023-01-08T17:17:51.918680 #1]  INFO -- : > chown postgres:postgres /shared/postgres_run/13-main.pg_stat_tmp
I, [2023-01-08T17:17:51.920791 #1]  INFO -- : 
I, [2023-01-08T17:17:51.925959 #1]  INFO -- : File > /etc/service/postgres/run  chmod: +x  chown: 
I, [2023-01-08T17:17:51.930666 #1]  INFO -- : File > /etc/service/postgres/log/run  chmod: +x  chown: 
I, [2023-01-08T17:17:51.935630 #1]  INFO -- : File > /etc/runit/3.d/99-postgres  chmod: +x  chown: 
I, [2023-01-08T17:17:51.940501 #1]  INFO -- : File > /root/upgrade_postgres  chmod: +x  chown: 
I, [2023-01-08T17:17:51.940815 #1]  INFO -- : > chown -R root /var/lib/postgresql/13/main
I, [2023-01-08T17:17:52.327524 #1]  INFO -- : 
I, [2023-01-08T17:17:52.327809 #1]  INFO -- : > [ ! -e /shared/postgres_data ] && install -d -m 0755 -o postgres -g postgres /shared/postgres_data && sudo -E -u postgres /usr/lib/postgresql/13/bin/initdb -D /shared/postgres_data || exit 0
I, [2023-01-08T17:17:52.330153 #1]  INFO -- : 
I, [2023-01-08T17:17:52.330206 #1]  INFO -- : > chown -R postgres:postgres /shared/postgres_data
I, [2023-01-08T17:17:52.385884 #1]  INFO -- : 
I, [2023-01-08T17:17:52.386017 #1]  INFO -- : > chown -R postgres:postgres /var/run/postgresql
I, [2023-01-08T17:17:52.388853 #1]  INFO -- : 
I, [2023-01-08T17:17:52.389039 #1]  INFO -- : > /root/upgrade_postgres
I, [2023-01-08T17:17:52.394749 #1]  INFO -- : 
I, [2023-01-08T17:17:52.394868 #1]  INFO -- : > rm /root/upgrade_postgres
I, [2023-01-08T17:17:52.397391 #1]  INFO -- : 
I, [2023-01-08T17:17:52.398496 #1]  INFO -- : Replacing data_directory = '/var/lib/postgresql/13/main' with data_directory = '/shared/postgres_data' in /etc/postgresql/13/main/postgresql.conf
I, [2023-01-08T17:17:52.399030 #1]  INFO -- : Replacing (?-mix:#?listen_addresses *=.*) with listen_addresses = '*' in /etc/postgresql/13/main/postgresql.conf
I, [2023-01-08T17:17:52.399339 #1]  INFO -- : Replacing (?-mix:#?synchronous_commit *=.*) with synchronous_commit = $db_synchronous_commit in /etc/postgresql/13/main/postgresql.conf
I, [2023-01-08T17:17:52.399638 #1]  INFO -- : Replacing (?-mix:#?shared_buffers *=.*) with shared_buffers = $db_shared_buffers in /etc/postgresql/13/main/postgresql.conf
I, [2023-01-08T17:17:52.399932 #1]  INFO -- : Replacing (?-mix:#?work_mem *=.*) with work_mem = $db_work_mem in /etc/postgresql/13/main/postgresql.conf
I, [2023-01-08T17:17:52.400191 #1]  INFO -- : Replacing (?-mix:#?default_text_search_config *=.*) with default_text_search_config = '$db_default_text_search_config' in /etc/postgresql/13/main/postgresql.conf
I, [2023-01-08T17:17:52.400449 #1]  INFO -- : > install -d -m 0755 -o postgres -g postgres /shared/postgres_backup
I, [2023-01-08T17:17:52.405233 #1]  INFO -- : 
I, [2023-01-08T17:17:52.405420 #1]  INFO -- : Replacing (?-mix:#?checkpoint_segments *=.*) with checkpoint_segments = $db_checkpoint_segments in /etc/postgresql/13/main/postgresql.conf
I, [2023-01-08T17:17:52.405722 #1]  INFO -- : Replacing (?-mix:#?logging_collector *=.*) with logging_collector = $db_logging_collector in /etc/postgresql/13/main/postgresql.conf
I, [2023-01-08T17:17:52.406030 #1]  INFO -- : Replacing (?-mix:#?log_min_duration_statement *=.*) with log_min_duration_statement = $db_log_min_duration_statement in /etc/postgresql/13/main/postgresql.conf
I, [2023-01-08T17:17:52.406566 #1]  INFO -- : Replacing (?-mix:^#local +replication +postgres +peer$) with local replication postgres  peer in /etc/postgresql/13/main/pg_hba.conf
I, [2023-01-08T17:17:52.406842 #1]  INFO -- : Replacing (?-mix:^host.*all.*all.*127.*$) with host all all 0.0.0.0/0 md5 in /etc/postgresql/13/main/pg_hba.conf
I, [2023-01-08T17:17:52.407086 #1]  INFO -- : Replacing (?-mix:^host.*all.*all.*::1\/128.*$) with host all all ::/0 md5 in /etc/postgresql/13/main/pg_hba.conf
I, [2023-01-08T17:17:52.407305 #1]  INFO -- : > HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main
I, [2023-01-08T17:17:52.409278 #1]  INFO -- : > sleep 5
2023-01-08 17:17:52.593 UTC [41] LOG:  starting PostgreSQL 13.9 (Debian 13.9-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
2023-01-08 17:17:52.594 UTC [41] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2023-01-08 17:17:52.594 UTC [41] LOG:  listening on IPv6 address "::", port 5432
2023-01-08 17:17:52.598 UTC [41] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2023-01-08 17:17:52.605 UTC [44] LOG:  database system was shut down at 2023-01-08 17:17:42 UTC
2023-01-08 17:17:52.621 UTC [41] LOG:  database system is ready to accept connections
I, [2023-01-08T17:17:57.412189 #1]  INFO -- : 
I, [2023-01-08T17:17:57.412815 #1]  INFO -- : > su postgres -c 'createdb discourse' || true
2023-01-08 17:17:57.500 UTC [54] postgres@postgres ERROR:  database "discourse" already exists
2023-01-08 17:17:57.500 UTC [54] postgres@postgres STATEMENT:  CREATE DATABASE discourse;
createdb: error: database creation failed: ERROR:  database "discourse" already exists
I, [2023-01-08T17:17:57.502683 #1]  INFO -- : 
I, [2023-01-08T17:17:57.503102 #1]  INFO -- : > su postgres -c 'psql discourse -c "create user discourse;"' || true
2023-01-08 17:17:57.571 UTC [58] postgres@discourse ERROR:  role "discourse" already exists
2023-01-08 17:17:57.571 UTC [58] postgres@discourse STATEMENT:  create user discourse;
ERROR:  role "discourse" already exists
I, [2023-01-08T17:17:57.573448 #1]  INFO -- : 
I, [2023-01-08T17:17:57.573809 #1]  INFO -- : > su postgres -c 'psql discourse -c "grant all privileges on database discourse to discourse;"' || true
I, [2023-01-08T17:17:57.632967 #1]  INFO -- : GRANT

I, [2023-01-08T17:17:57.633283 #1]  INFO -- : > su postgres -c 'psql discourse -c "alter schema public owner to discourse;"'
I, [2023-01-08T17:17:57.683299 #1]  INFO -- : ALTER SCHEMA

I, [2023-01-08T17:17:57.683765 #1]  INFO -- : > su postgres -c 'psql template1 -c "create extension if not exists hstore;"'
NOTICE:  extension "hstore" already exists, skipping
I, [2023-01-08T17:17:57.737889 #1]  INFO -- : CREATE EXTENSION

I, [2023-01-08T17:17:57.738056 #1]  INFO -- : > su postgres -c 'psql template1 -c "create extension if not exists pg_trgm;"'
NOTICE:  extension "pg_trgm" already exists, skipping
I, [2023-01-08T17:17:57.788026 #1]  INFO -- : CREATE EXTENSION

I, [2023-01-08T17:17:57.788254 #1]  INFO -- : > su postgres -c 'psql discourse -c "create extension if not exists hstore;"'
NOTICE:  extension "hstore" already exists, skipping
I, [2023-01-08T17:17:57.834234 #1]  INFO -- : CREATE EXTENSION

I, [2023-01-08T17:17:57.834647 #1]  INFO -- : > su postgres -c 'psql discourse -c "create extension if not exists pg_trgm;"'
NOTICE:  extension "pg_trgm" already exists, skipping
I, [2023-01-08T17:17:57.882288 #1]  INFO -- : CREATE EXTENSION

I, [2023-01-08T17:17:57.882631 #1]  INFO -- : > sudo -u postgres psql discourse
I, [2023-01-08T17:17:57.885847 #1]  INFO -- : update pg_database set encoding = pg_char_to_encoding('UTF8') where datname = 'discourse' AND encoding = pg_char_to_encoding('SQL_ASCII');

I, [2023-01-08T17:17:57.954282 #1]  INFO -- : File > /var/lib/postgresql/take-database-backup  chmod: +x  chown: postgres:postgres
I, [2023-01-08T17:17:57.957842 #1]  INFO -- : File > /var/spool/cron/crontabs/postgres  chmod:   chown: 
I, [2023-01-08T17:17:57.957973 #1]  INFO -- : > echo postgres installed!
I, [2023-01-08T17:17:57.959975 #1]  INFO -- : postgres installed!

I, [2023-01-08T17:17:57.964538 #1]  INFO -- : File > /etc/service/redis/run  chmod: +x  chown: 
I, [2023-01-08T17:17:57.969401 #1]  INFO -- : File > /etc/service/redis/log/run  chmod: +x  chown: 
I, [2023-01-08T17:17:57.974012 #1]  INFO -- : File > /etc/runit/3.d/10-redis  chmod: +x  chown: 
I, [2023-01-08T17:17:57.975225 #1]  INFO -- : Replacing daemonize yes with  in /etc/redis/redis.conf
I, [2023-01-08T17:17:57.975797 #1]  INFO -- : Replacing (?-mix:^pidfile.*$) with  in /etc/redis/redis.conf
I, [2023-01-08T17:17:57.977653 #1]  INFO -- : > install -d -m 0755 -o redis -g redis /shared/redis_data
I, [2023-01-08T17:17:57.980144 #1]  INFO -- : 
I, [2023-01-08T17:17:57.980429 #1]  INFO -- : Replacing (?-mix:^logfile.*$) with logfile "" in /etc/redis/redis.conf
I, [2023-01-08T17:17:57.980926 #1]  INFO -- : Replacing (?-mix:^bind .*$) with  in /etc/redis/redis.conf
I, [2023-01-08T17:17:57.981625 #1]  INFO -- : Replacing (?-mix:^dir .*$) with dir /shared/redis_data in /etc/redis/redis.conf
I, [2023-01-08T17:17:57.982227 #1]  INFO -- : Replacing (?-mix:^protected-mode yes) with protected-mode no in /etc/redis/redis.conf
I, [2023-01-08T17:17:57.982908 #1]  INFO -- : Replacing # io-threads 4 with io-threads $redis_io_threads in /etc/redis/redis.conf
I, [2023-01-08T17:17:57.983543 #1]  INFO -- : > echo redis installed
I, [2023-01-08T17:17:57.985612 #1]  INFO -- : redis installed

I, [2023-01-08T17:17:57.985800 #1]  INFO -- : > cat /etc/redis/redis.conf | grep logfile
I, [2023-01-08T17:17:57.992316 #1]  INFO -- : logfile ""

I, [2023-01-08T17:17:57.992577 #1]  INFO -- : > exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf
I, [2023-01-08T17:17:57.994136 #1]  INFO -- : > sleep 10
102:C 08 Jan 2023 17:17:58.023 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
102:C 08 Jan 2023 17:17:58.023 # Redis version=7.0.7, bits=64, commit=00000000, modified=0, pid=102, just started
102:C 08 Jan 2023 17:17:58.023 # Configuration loaded
102:M 08 Jan 2023 17:17:58.023 * monotonic clock: POSIX clock_gettime
102:M 08 Jan 2023 17:17:58.025 * Running mode=standalone, port=6379.
102:M 08 Jan 2023 17:17:58.025 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
102:M 08 Jan 2023 17:17:58.025 # Server initialized
102:M 08 Jan 2023 17:17:58.025 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
102:M 08 Jan 2023 17:17:58.025 # WARNING You have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo madvise > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled (set to 'madvise' or 'never').
102:M 08 Jan 2023 17:17:58.026 * Loading RDB produced by version 7.0.7
102:M 08 Jan 2023 17:17:58.026 * RDB age 26094 seconds
102:M 08 Jan 2023 17:17:58.026 * RDB memory usage when created 267.24 Mb
102:M 08 Jan 2023 17:17:59.205 * Done loading RDB, keys loaded: 28568, keys expired: 15293.
102:M 08 Jan 2023 17:17:59.205 * DB loaded from disk: 1.179 seconds
102:M 08 Jan 2023 17:17:59.205 * Ready to accept connections
I, [2023-01-08T17:18:07.996578 #1]  INFO -- : 
I, [2023-01-08T17:18:07.997276 #1]  INFO -- : > thpoff echo "thpoff is installed!"
I, [2023-01-08T17:18:08.001894 #1]  INFO -- : thpoff is installed!

I, [2023-01-08T17:18:08.002222 #1]  INFO -- : > /usr/local/bin/ruby -e 'if ENV["DISCOURSE_SMTP_ADDRESS"] == "smtp.example.com"; puts "Aborting! Mail is not configured!"; exit 1; end'
I, [2023-01-08T17:18:08.163344 #1]  INFO -- : 
I, [2023-01-08T17:18:08.163791 #1]  INFO -- : > /usr/local/bin/ruby -e 'if ENV["DISCOURSE_HOSTNAME"] == "discourse.example.com"; puts "Aborting! Domain is not configured!"; exit 1; end'
I, [2023-01-08T17:18:08.256476 #1]  INFO -- : 
I, [2023-01-08T17:18:08.257079 #1]  INFO -- : > /usr/local/bin/ruby -e 'if (ENV["DISCOURSE_CDN_URL"] || "")[0..1] == "//"; puts "Aborting! CDN must have a protocol specified. Once fixed you should rebake your posts now to correct all posts."; exit 1; end'
I, [2023-01-08T17:18:08.335922 #1]  INFO -- : 
I, [2023-01-08T17:18:08.336428 #1]  INFO -- : > rm -f /etc/cron.d/anacron
I, [2023-01-08T17:18:08.340179 #1]  INFO -- : 
I, [2023-01-08T17:18:08.343109 #1]  INFO -- : File > /etc/cron.d/anacron  chmod:   chown: 
I, [2023-01-08T17:18:08.348517 #1]  INFO -- : File > /etc/runit/1.d/copy-env  chmod: +x  chown: 
I, [2023-01-08T17:18:08.353394 #1]  INFO -- : File > /etc/service/unicorn/run  chmod: +x  chown: 
I, [2023-01-08T17:18:08.357978 #1]  INFO -- : File > /etc/service/nginx/run  chmod: +x  chown: 
I, [2023-01-08T17:18:08.362553 #1]  INFO -- : File > /etc/runit/3.d/01-nginx  chmod: +x  chown: 
I, [2023-01-08T17:18:08.367045 #1]  INFO -- : File > /etc/runit/3.d/02-unicorn  chmod: +x  chown: 
I, [2023-01-08T17:18:08.367288 #1]  INFO -- : Replacing # postgres with sv start postgres || exit 1 in /etc/service/unicorn/run
I, [2023-01-08T17:18:08.367561 #1]  INFO -- : > exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf
I, [2023-01-08T17:18:08.370035 #1]  INFO -- : > cd /var/www/discourse && git reset --hard
129:C 08 Jan 2023 17:18:08.376 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
129:C 08 Jan 2023 17:18:08.376 # Redis version=7.0.7, bits=64, commit=00000000, modified=0, pid=129, just started
129:C 08 Jan 2023 17:18:08.376 # Configuration loaded
129:M 08 Jan 2023 17:18:08.377 * monotonic clock: POSIX clock_gettime
129:M 08 Jan 2023 17:18:08.377 # Warning: Could not create server TCP listening socket *:6379: bind: Address already in use
129:M 08 Jan 2023 17:18:08.377 # Failed listening on port 6379 (TCP), aborting.
Updating files: 100% (32637/32637), done.
I, [2023-01-08T17:18:11.718607 #1]  INFO -- : HEAD is now at e775ed1c DEV: Properly include spec examples (#19532)

I, [2023-01-08T17:18:11.718853 #1]  INFO -- : > cd /var/www/discourse && git clean -f
I, [2023-01-08T17:18:11.867916 #1]  INFO -- : 
I, [2023-01-08T17:18:11.868312 #1]  INFO -- : > cd /var/www/discourse && git remote set-branches --add origin main
I, [2023-01-08T17:18:11.873068 #1]  INFO -- : 
I, [2023-01-08T17:18:11.873230 #1]  INFO -- : > cd /var/www/discourse && git remote set-branches origin tests-passed
I, [2023-01-08T17:18:11.877351 #1]  INFO -- : 
I, [2023-01-08T17:18:11.877407 #1]  INFO -- : > cd /var/www/discourse && git fetch --depth 1 origin tests-passed
fatal: unable to access 'https://github.com/discourse/discourse.git/': Could not resolve host: github.com
I, [2023-01-08T17:18:31.945158 #1]  INFO -- : 
I, [2023-01-08T17:18:31.945552 #1]  INFO -- : Terminating async processes
I, [2023-01-08T17:18:31.945652 #1]  INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main pid: 41
2023-01-08 17:18:31.945 UTC [41] LOG:  received fast shutdown request
I, [2023-01-08T17:18:31.945947 #1]  INFO -- : Sending TERM to exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf pid: 102
102:signal-handler (1673198311) Received SIGTERM scheduling shutdown...
2023-01-08 17:18:31.949 UTC [41] LOG:  aborting any active transactions
2023-01-08 17:18:31.952 UTC [41] LOG:  background worker "logical replication launcher" (PID 50) exited with exit code 1
2023-01-08 17:18:31.952 UTC [45] LOG:  shutting down
102:M 08 Jan 2023 17:18:31.994 # User requested shutdown...
102:M 08 Jan 2023 17:18:31.994 * Saving the final RDB snapshot before exiting.
2023-01-08 17:18:32.003 UTC [41] LOG:  database system is shut down
102:M 08 Jan 2023 17:18:33.194 * DB saved on disk
102:M 08 Jan 2023 17:18:33.194 # Redis is now ready to exit, bye bye...


FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && git fetch --depth 1 origin tests-passed failed with return #<Process::Status: pid 139 exit 128>
Location of failure: /usr/local/lib/ruby/gems/3.1.0/gems/pups-1.1.1/lib/pups/exec_command.rb:117:in `spawn'
exec failed with the params {"cd"=>"$home", "hook"=>"code", "cmd"=>["git reset --hard", "git clean -f", "git remote set-branches --add origin main", "git remote set-branches origin $version", "git fetch --depth 1 origin $version", "git checkout $version", "mkdir -p tmp", "chown discourse:www-data tmp", "mkdir -p tmp/pids", "mkdir -p tmp/sockets", "touch tmp/.gitkeep", "mkdir -p                    /shared/log/rails", "bash -c \"touch -a           /shared/log/rails/{production,production_errors,unicorn.stdout,unicorn.stderr,sidekiq}.log\"", "bash -c \"ln    -s           /shared/log/rails/{production,production_errors,unicorn.stdout,unicorn.stderr,sidekiq}.log $home/log\"", "bash -c \"mkdir -p           /shared/{uploads,backups}\"", "bash -c \"ln    -s           /shared/{uploads,backups} $home/public\"", "bash -c \"mkdir -p           /shared/tmp/{backups,restores}\"", "bash -c \"ln    -s           /shared/tmp/{backups,restores} $home/tmp\"", "chown -R discourse:www-data /shared/log/rails /shared/uploads /shared/backups /shared/tmp", "[ ! -d public/plugins ] || find public/plugins/ -maxdepth 1 -xtype l -delete"]}
bootstrap failed with exit code 128
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.
75d71c3f22d42c746becd5f6fbeb5529fd01ff4a7176d2b942930aad6f973b81

littleviolette · January 8, 2023, 5:26pm

@RGJ How can I resolve the DNS issue? Are there other issues at play?

pfaffman · January 8, 2023, 6:01pm

Yeah. Sorry. I didn’t know you had that much ram.

fatal: unable to access 'https://github.com/discourse/discourse.git/': Could not resolve host: github.com

That’s the issue. Don’t know how you’d get that on a digital ocean droplet. Must be something with your docker network. But I don’t have any ideas.

Edit: here’s one, make sure docker is up to date and reboot. See if you can

 docker run hello-world

littleviolette · January 8, 2023, 6:50pm

SFO1 DigitalOcean server

littleviolette · January 8, 2023, 9:40pm

Docker seems to be running correctly. Anything else I can try?

root@forum:~# docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
2db29710123e: Pull complete 
Digest: sha256:94ebc7edf3401f299cd3376a1669bc0a49aef92d6d2669005f9bc5ef028dc333
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

littleviolette · January 8, 2023, 9:52pm

The below didn’t work. Still getting the same error.

git config --global --unset https.proxy

littleviolette · January 9, 2023, 12:29am

Could this be the cause of the issue? How could I resolve it?

# Warning: Could not create server TCP listening socket *:6379: bind: Address already in use
# Failed listening on port 6379 (TCP), aborting.

Thank you!

littleviolette · January 9, 2023, 1:44am

I restored a backup of our droplet and try rebuilding the app again. I was returned with the error:

bootstrap failed with exit code 1

This old image runs on Ubuntu 16.04.7 LTS (GNU/Linux 3.13.0-52-generic x86_64).

Anyone would be able to help? Where else can I seek help? Thank you!

Here is the full log:

root@lh-forum-4:/var/discourse# ./launcher rebuild app
x86_64 arch detected.
WARNING: containers/app.yml file is world-readable. You can secure this file by running: chmod o-rwx containers/app.yml
Ensuring launcher is up to date
Fetching origin
remote: Enumerating objects: 75, done.
remote: Counting objects: 100% (46/46), done.
remote: Compressing objects: 100% (6/6), done.
remote: Total 75 (delta 38), reused 42 (delta 38), pack-reused 29
Unpacking objects: 100% (75/75), done.
From https://github.com/discourse/discourse_docker
   e0f27d1..e6cb71c  main       -> origin/main
 * [new branch]      build-27   -> origin/build-27
 * [new branch]      revert-654-old-image-for-old-discourse -> origin/revert-654-old-image-for-old-discourse
 + 4fae4a0...2e142d4 ruby-3.2   -> origin/ruby-3.2  (forced update)
   bdc1b59..4ffccf2  yarn-timeout -> origin/yarn-timeout
Updating Launcher...
Updating e0f27d1..e6cb71c
Fast-forward
 .github/workflows/build.yml                   | 26 +++++++++++++-------------
 image/base/VERSION                            |  1 -
 image/base/install-imagemagick                |  4 ++--
 image/base/install-jemalloc                   | 10 +++++-----
 image/base/install-nginx                      |  4 ++--
 image/base/install-oxipng                     |  4 ++--
 image/base/install-redis                      |  4 ++--
 image/base/install-ruby                       |  7 ++++++-
 image/base/install-rust                       | 30 ++++++++++++++++++++++++++++++
 image/base/slim.Dockerfile                    | 12 +++++++++---
 image/discourse_dev/postgres_dev.template.yml |  1 +
 image/discourse_test/Dockerfile               |  3 ++-
 launcher                                      | 11 ++++++++++-
 templates/web.template.yml                    |  4 ++--
 14 files changed, 86 insertions(+), 35 deletions(-)
 delete mode 100644 image/base/VERSION
 create mode 100755 image/base/install-rust
Launcher updated, restarting...
x86_64 arch detected.

WARNING: We are about to start downloading the Discourse base image
This process may take anywhere between a few minutes to an hour, depending on your network speed

Please be patient

2.0.20221221-0050: Pulling from discourse/base
025c56f98b67: Pull complete 
4daec98b5622: Pull complete 
05576b07f5fb: Pull complete 
Digest: sha256:7ac6489e34a0f40f32457bffaa331940a98447b46451a6a1279a79868ee5420b
Status: Downloaded newer image for discourse/base:2.0.20221221-0050
WARNING: containers/app.yml file is world-readable. You can secure this file by running: chmod o-rwx containers/app.yml
Ensuring launcher is up to date
Fetching origin
Launcher is up-to-date
Stopping old container
+ /usr/bin/docker stop -t 60 app
app
2.0.20221221-0050: Pulling from discourse/base
Digest: sha256:7ac6489e34a0f40f32457bffaa331940a98447b46451a6a1279a79868ee5420b
Status: Image is up to date for discourse/base:2.0.20221221-0050
/usr/local/lib/ruby/gems/3.1.0/gems/pups-1.1.1/lib/pups.rb
/usr/local/bin/pups --stdin
I, [2023-01-09T01:24:11.633490 #1]  INFO -- : Reading from stdin
I, [2023-01-09T01:24:11.638961 #1]  INFO -- : > locale-gen $LANG && update-locale
I, [2023-01-09T01:24:11.697592 #1]  INFO -- : Generating locales (this might take a while)...
Generation complete.

I, [2023-01-09T01:24:11.697856 #1]  INFO -- : > mkdir -p /shared/postgres_run
I, [2023-01-09T01:24:11.704671 #1]  INFO -- : 
I, [2023-01-09T01:24:11.705524 #1]  INFO -- : > chown postgres:postgres /shared/postgres_run
I, [2023-01-09T01:24:11.710149 #1]  INFO -- : 
I, [2023-01-09T01:24:11.710385 #1]  INFO -- : > chmod 775 /shared/postgres_run
I, [2023-01-09T01:24:11.713993 #1]  INFO -- : 
I, [2023-01-09T01:24:11.714224 #1]  INFO -- : > rm -fr /var/run/postgresql
I, [2023-01-09T01:24:11.717249 #1]  INFO -- : 
I, [2023-01-09T01:24:11.717484 #1]  INFO -- : > ln -s /shared/postgres_run /var/run/postgresql
I, [2023-01-09T01:24:11.720899 #1]  INFO -- : 
I, [2023-01-09T01:24:11.721113 #1]  INFO -- : > socat /dev/null UNIX-CONNECT:/shared/postgres_run/.s.PGSQL.5432 || exit 0 && echo postgres already running stop container ; exit 1
2023/01/09 01:24:11 socat[19] E connect(6, AF=1 "/shared/postgres_run/.s.PGSQL.5432", 36): Connection refused
I, [2023-01-09T01:24:11.747602 #1]  INFO -- : 
I, [2023-01-09T01:24:11.747838 #1]  INFO -- : > rm -fr /shared/postgres_run/.s*
I, [2023-01-09T01:24:11.752658 #1]  INFO -- : 
I, [2023-01-09T01:24:11.752860 #1]  INFO -- : > rm -fr /shared/postgres_run/*.pid
I, [2023-01-09T01:24:11.756646 #1]  INFO -- : 
I, [2023-01-09T01:24:11.756769 #1]  INFO -- : > mkdir -p /shared/postgres_run/13-main.pg_stat_tmp
I, [2023-01-09T01:24:11.759946 #1]  INFO -- : 
I, [2023-01-09T01:24:11.760120 #1]  INFO -- : > chown postgres:postgres /shared/postgres_run/13-main.pg_stat_tmp
I, [2023-01-09T01:24:11.763132 #1]  INFO -- : 
I, [2023-01-09T01:24:11.769380 #1]  INFO -- : File > /etc/service/postgres/run  chmod: +x  chown: 
I, [2023-01-09T01:24:11.775772 #1]  INFO -- : File > /etc/service/postgres/log/run  chmod: +x  chown: 
I, [2023-01-09T01:24:11.782041 #1]  INFO -- : File > /etc/runit/3.d/99-postgres  chmod: +x  chown: 
I, [2023-01-09T01:24:11.788362 #1]  INFO -- : File > /root/upgrade_postgres  chmod: +x  chown: 
I, [2023-01-09T01:24:11.788781 #1]  INFO -- : > chown -R root /var/lib/postgresql/13/main
I, [2023-01-09T01:24:11.886134 #1]  INFO -- : 
I, [2023-01-09T01:24:11.886778 #1]  INFO -- : > [ ! -e /shared/postgres_data ] && install -d -m 0755 -o postgres -g postgres /shared/postgres_data && sudo -E -u postgres /usr/lib/postgresql/13/bin/initdb -D /shared/postgres_data || exit 0
I, [2023-01-09T01:24:11.890560 #1]  INFO -- : 
I, [2023-01-09T01:24:11.890642 #1]  INFO -- : > chown -R postgres:postgres /shared/postgres_data
I, [2023-01-09T01:24:11.937820 #1]  INFO -- : 
I, [2023-01-09T01:24:11.937984 #1]  INFO -- : > chown -R postgres:postgres /var/run/postgresql
I, [2023-01-09T01:24:11.941646 #1]  INFO -- : 
I, [2023-01-09T01:24:11.941855 #1]  INFO -- : > /root/upgrade_postgres
I, [2023-01-09T01:24:11.948144 #1]  INFO -- : 
I, [2023-01-09T01:24:11.948323 #1]  INFO -- : > rm /root/upgrade_postgres
I, [2023-01-09T01:24:11.950684 #1]  INFO -- : 
I, [2023-01-09T01:24:11.951813 #1]  INFO -- : Replacing data_directory = '/var/lib/postgresql/13/main' with data_directory = '/shared/postgres_data' in /etc/postgresql/13/main/postgresql.conf
I, [2023-01-09T01:24:11.952432 #1]  INFO -- : Replacing (?-mix:#?listen_addresses *=.*) with listen_addresses = '*' in /etc/postgresql/13/main/postgresql.conf
I, [2023-01-09T01:24:11.952806 #1]  INFO -- : Replacing (?-mix:#?synchronous_commit *=.*) with synchronous_commit = $db_synchronous_commit in /etc/postgresql/13/main/postgresql.conf
I, [2023-01-09T01:24:11.953169 #1]  INFO -- : Replacing (?-mix:#?shared_buffers *=.*) with shared_buffers = $db_shared_buffers in /etc/postgresql/13/main/postgresql.conf
I, [2023-01-09T01:24:11.953567 #1]  INFO -- : Replacing (?-mix:#?work_mem *=.*) with work_mem = $db_work_mem in /etc/postgresql/13/main/postgresql.conf
I, [2023-01-09T01:24:11.954485 #1]  INFO -- : Replacing (?-mix:#?default_text_search_config *=.*) with default_text_search_config = '$db_default_text_search_config' in /etc/postgresql/13/main/postgresql.conf
I, [2023-01-09T01:24:11.954943 #1]  INFO -- : > install -d -m 0755 -o postgres -g postgres /shared/postgres_backup
I, [2023-01-09T01:24:11.960686 #1]  INFO -- : 
I, [2023-01-09T01:24:11.960968 #1]  INFO -- : Replacing (?-mix:#?checkpoint_segments *=.*) with checkpoint_segments = $db_checkpoint_segments in /etc/postgresql/13/main/postgresql.conf
I, [2023-01-09T01:24:11.961288 #1]  INFO -- : Replacing (?-mix:#?logging_collector *=.*) with logging_collector = $db_logging_collector in /etc/postgresql/13/main/postgresql.conf
I, [2023-01-09T01:24:11.961749 #1]  INFO -- : Replacing (?-mix:#?log_min_duration_statement *=.*) with log_min_duration_statement = $db_log_min_duration_statement in /etc/postgresql/13/main/postgresql.conf
I, [2023-01-09T01:24:11.962361 #1]  INFO -- : Replacing (?-mix:^#local +replication +postgres +peer$) with local replication postgres  peer in /etc/postgresql/13/main/pg_hba.conf
I, [2023-01-09T01:24:11.962631 #1]  INFO -- : Replacing (?-mix:^host.*all.*all.*127.*$) with host all all 0.0.0.0/0 md5 in /etc/postgresql/13/main/pg_hba.conf
I, [2023-01-09T01:24:11.962854 #1]  INFO -- : Replacing (?-mix:^host.*all.*all.*::1\/128.*$) with host all all ::/0 md5 in /etc/postgresql/13/main/pg_hba.conf
I, [2023-01-09T01:24:11.963079 #1]  INFO -- : > HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main
I, [2023-01-09T01:24:11.964769 #1]  INFO -- : > sleep 5
2023-01-09 01:24:12.155 UTC [42] LOG:  starting PostgreSQL 13.9 (Debian 13.9-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
2023-01-09 01:24:12.155 UTC [42] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2023-01-09 01:24:12.155 UTC [42] LOG:  listening on IPv6 address "::", port 5432
2023-01-09 01:24:12.159 UTC [42] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2023-01-09 01:24:12.180 UTC [45] LOG:  database system was interrupted; last known up at 2023-01-09 01:19:52 UTC
2023-01-09 01:24:12.328 UTC [45] LOG:  database system was not properly shut down; automatic recovery in progress
2023-01-09 01:24:12.332 UTC [45] LOG:  redo starts at 104B/AF57BD60
2023-01-09 01:24:12.399 UTC [45] LOG:  redo done at 104B/B014B070
2023-01-09 01:24:12.510 UTC [42] LOG:  database system is ready to accept connections
I, [2023-01-09T01:24:16.968330 #1]  INFO -- : 
I, [2023-01-09T01:24:16.969333 #1]  INFO -- : > su postgres -c 'createdb discourse' || true
2023-01-09 01:24:17.073 UTC [55] postgres@postgres ERROR:  database "discourse" already exists
2023-01-09 01:24:17.073 UTC [55] postgres@postgres STATEMENT:  CREATE DATABASE discourse;
createdb: error: database creation failed: ERROR:  database "discourse" already exists
I, [2023-01-09T01:24:17.075782 #1]  INFO -- : 
I, [2023-01-09T01:24:17.076143 #1]  INFO -- : > su postgres -c 'psql discourse -c "create user discourse;"' || true
2023-01-09 01:24:17.158 UTC [59] postgres@discourse ERROR:  role "discourse" already exists
2023-01-09 01:24:17.158 UTC [59] postgres@discourse STATEMENT:  create user discourse;
ERROR:  role "discourse" already exists
I, [2023-01-09T01:24:17.161838 #1]  INFO -- : 
I, [2023-01-09T01:24:17.162221 #1]  INFO -- : > su postgres -c 'psql discourse -c "grant all privileges on database discourse to discourse;"' || true
I, [2023-01-09T01:24:17.225658 #1]  INFO -- : GRANT

I, [2023-01-09T01:24:17.225947 #1]  INFO -- : > su postgres -c 'psql discourse -c "alter schema public owner to discourse;"'
I, [2023-01-09T01:24:17.287774 #1]  INFO -- : ALTER SCHEMA

I, [2023-01-09T01:24:17.288208 #1]  INFO -- : > su postgres -c 'psql template1 -c "create extension if not exists hstore;"'
NOTICE:  extension "hstore" already exists, skipping
I, [2023-01-09T01:24:17.355297 #1]  INFO -- : CREATE EXTENSION

I, [2023-01-09T01:24:17.355532 #1]  INFO -- : > su postgres -c 'psql template1 -c "create extension if not exists pg_trgm;"'
NOTICE:  extension "pg_trgm" already exists, skipping
I, [2023-01-09T01:24:17.414737 #1]  INFO -- : CREATE EXTENSION

I, [2023-01-09T01:24:17.414981 #1]  INFO -- : > su postgres -c 'psql discourse -c "create extension if not exists hstore;"'
NOTICE:  extension "hstore" already exists, skipping
I, [2023-01-09T01:24:17.480559 #1]  INFO -- : CREATE EXTENSION

I, [2023-01-09T01:24:17.480906 #1]  INFO -- : > su postgres -c 'psql discourse -c "create extension if not exists pg_trgm;"'
NOTICE:  extension "pg_trgm" already exists, skipping
I, [2023-01-09T01:24:17.543581 #1]  INFO -- : CREATE EXTENSION

I, [2023-01-09T01:24:17.543962 #1]  INFO -- : > sudo -u postgres psql discourse
I, [2023-01-09T01:24:17.548022 #1]  INFO -- : update pg_database set encoding = pg_char_to_encoding('UTF8') where datname = 'discourse' AND encoding = pg_char_to_encoding('SQL_ASCII');

I, [2023-01-09T01:24:17.629332 #1]  INFO -- : File > /var/lib/postgresql/take-database-backup  chmod: +x  chown: postgres:postgres
I, [2023-01-09T01:24:17.633132 #1]  INFO -- : File > /var/spool/cron/crontabs/postgres  chmod:   chown: 
I, [2023-01-09T01:24:17.633279 #1]  INFO -- : > echo postgres installed!
I, [2023-01-09T01:24:17.635614 #1]  INFO -- : postgres installed!

I, [2023-01-09T01:24:17.641703 #1]  INFO -- : File > /etc/service/redis/run  chmod: +x  chown: 
I, [2023-01-09T01:24:17.647615 #1]  INFO -- : File > /etc/service/redis/log/run  chmod: +x  chown: 
I, [2023-01-09T01:24:17.653516 #1]  INFO -- : File > /etc/runit/3.d/10-redis  chmod: +x  chown: 
I, [2023-01-09T01:24:17.654485 #1]  INFO -- : Replacing daemonize yes with  in /etc/redis/redis.conf
I, [2023-01-09T01:24:17.655248 #1]  INFO -- : Replacing (?-mix:^pidfile.*$) with  in /etc/redis/redis.conf
I, [2023-01-09T01:24:17.656560 #1]  INFO -- : > install -d -m 0755 -o redis -g redis /shared/redis_data
I, [2023-01-09T01:24:17.659732 #1]  INFO -- : 
I, [2023-01-09T01:24:17.660109 #1]  INFO -- : Replacing (?-mix:^logfile.*$) with logfile "" in /etc/redis/redis.conf
I, [2023-01-09T01:24:17.660734 #1]  INFO -- : Replacing (?-mix:^bind .*$) with  in /etc/redis/redis.conf
I, [2023-01-09T01:24:17.661486 #1]  INFO -- : Replacing (?-mix:^dir .*$) with dir /shared/redis_data in /etc/redis/redis.conf
I, [2023-01-09T01:24:17.662997 #1]  INFO -- : Replacing (?-mix:^protected-mode yes) with protected-mode no in /etc/redis/redis.conf
I, [2023-01-09T01:24:17.663728 #1]  INFO -- : Replacing # io-threads 4 with io-threads $redis_io_threads in /etc/redis/redis.conf
I, [2023-01-09T01:24:17.664352 #1]  INFO -- : > echo redis installed
I, [2023-01-09T01:24:17.667002 #1]  INFO -- : redis installed

I, [2023-01-09T01:24:17.667147 #1]  INFO -- : > cat /etc/redis/redis.conf | grep logfile
I, [2023-01-09T01:24:17.673285 #1]  INFO -- : logfile ""

I, [2023-01-09T01:24:17.673495 #1]  INFO -- : > exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf
I, [2023-01-09T01:24:17.675679 #1]  INFO -- : > sleep 10
103:C 09 Jan 2023 01:24:17.694 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
103:C 09 Jan 2023 01:24:17.694 # Redis version=7.0.7, bits=64, commit=00000000, modified=0, pid=103, just started
103:C 09 Jan 2023 01:24:17.694 # Configuration loaded
103:M 09 Jan 2023 01:24:17.694 * monotonic clock: POSIX clock_gettime
103:M 09 Jan 2023 01:24:17.696 * Running mode=standalone, port=6379.
103:M 09 Jan 2023 01:24:17.696 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
103:M 09 Jan 2023 01:24:17.696 # Server initialized
103:M 09 Jan 2023 01:24:17.696 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
103:M 09 Jan 2023 01:24:17.696 # WARNING You have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo madvise > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled (set to 'madvise' or 'never').
103:M 09 Jan 2023 01:24:17.697 * Loading RDB produced by version 6.2.6
103:M 09 Jan 2023 01:24:17.697 * RDB age 24 seconds
103:M 09 Jan 2023 01:24:17.697 * RDB memory usage when created 223.12 Mb
103:M 09 Jan 2023 01:24:18.488 * Done loading RDB, keys loaded: 20712, keys expired: 2.
103:M 09 Jan 2023 01:24:18.488 * DB loaded from disk: 0.792 seconds
103:M 09 Jan 2023 01:24:18.488 * Ready to accept connections
I, [2023-01-09T01:24:27.678833 #1]  INFO -- : 
I, [2023-01-09T01:24:27.679564 #1]  INFO -- : > thpoff echo "thpoff is installed!"
I, [2023-01-09T01:24:27.685101 #1]  INFO -- : thpoff is installed!

I, [2023-01-09T01:24:27.685300 #1]  INFO -- : > /usr/local/bin/ruby -e 'if ENV["DISCOURSE_SMTP_ADDRESS"] == "smtp.example.com"; puts "Aborting! Mail is not configured!"; exit 1; end'
I, [2023-01-09T01:24:27.821400 #1]  INFO -- : 
I, [2023-01-09T01:24:27.821650 #1]  INFO -- : > /usr/local/bin/ruby -e 'if ENV["DISCOURSE_HOSTNAME"] == "discourse.example.com"; puts "Aborting! Domain is not configured!"; exit 1; end'
I, [2023-01-09T01:24:27.922447 #1]  INFO -- : 
I, [2023-01-09T01:24:27.922729 #1]  INFO -- : > /usr/local/bin/ruby -e 'if (ENV["DISCOURSE_CDN_URL"] || "")[0..1] == "//"; puts "Aborting! CDN must have a protocol specified. Once fixed you should rebake your posts now to correct all posts."; exit 1; end'
I, [2023-01-09T01:24:28.027881 #1]  INFO -- : 
I, [2023-01-09T01:24:28.028111 #1]  INFO -- : > rm -f /etc/cron.d/anacron
I, [2023-01-09T01:24:28.031718 #1]  INFO -- : 
I, [2023-01-09T01:24:28.035410 #1]  INFO -- : File > /etc/cron.d/anacron  chmod:   chown: 
I, [2023-01-09T01:24:28.041535 #1]  INFO -- : File > /etc/runit/1.d/copy-env  chmod: +x  chown: 
I, [2023-01-09T01:24:28.047538 #1]  INFO -- : File > /etc/service/unicorn/run  chmod: +x  chown: 
I, [2023-01-09T01:24:28.053666 #1]  INFO -- : File > /etc/service/nginx/run  chmod: +x  chown: 
I, [2023-01-09T01:24:28.059855 #1]  INFO -- : File > /etc/runit/3.d/01-nginx  chmod: +x  chown: 
I, [2023-01-09T01:24:28.065788 #1]  INFO -- : File > /etc/runit/3.d/02-unicorn  chmod: +x  chown: 
I, [2023-01-09T01:24:28.065958 #1]  INFO -- : Replacing # postgres with sv start postgres || exit 1 in /etc/service/unicorn/run
I, [2023-01-09T01:24:28.066269 #1]  INFO -- : > exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf
I, [2023-01-09T01:24:28.069486 #1]  INFO -- : > cd /var/www/discourse && git reset --hard
130:C 09 Jan 2023 01:24:28.076 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
130:C 09 Jan 2023 01:24:28.076 # Redis version=7.0.7, bits=64, commit=00000000, modified=0, pid=130, just started
130:C 09 Jan 2023 01:24:28.076 # Configuration loaded
130:M 09 Jan 2023 01:24:28.077 * monotonic clock: POSIX clock_gettime
130:M 09 Jan 2023 01:24:28.077 # Warning: Could not create server TCP listening socket *:6379: bind: Address already in use
130:M 09 Jan 2023 01:24:28.077 # Failed listening on port 6379 (TCP), aborting.
Updating files: 100% (32637/32637), done.
I, [2023-01-09T01:24:30.874219 #1]  INFO -- : HEAD is now at e775ed1c DEV: Properly include spec examples (#19532)

I, [2023-01-09T01:24:30.874398 #1]  INFO -- : > cd /var/www/discourse && git clean -f
I, [2023-01-09T01:24:31.062146 #1]  INFO -- : 
I, [2023-01-09T01:24:31.062530 #1]  INFO -- : > cd /var/www/discourse && git remote set-branches --add origin main
I, [2023-01-09T01:24:31.069712 #1]  INFO -- : 
I, [2023-01-09T01:24:31.069909 #1]  INFO -- : > cd /var/www/discourse && git remote set-branches origin tests-passed
I, [2023-01-09T01:24:31.076412 #1]  INFO -- : 
I, [2023-01-09T01:24:31.076586 #1]  INFO -- : > cd /var/www/discourse && git fetch --depth 1 origin tests-passed
From https://github.com/discourse/discourse
 * branch              tests-passed -> FETCH_HEAD
 * [new branch]        tests-passed -> origin/tests-passed
I, [2023-01-09T01:24:33.068037 #1]  INFO -- : 
I, [2023-01-09T01:24:33.068288 #1]  INFO -- : > cd /var/www/discourse && git checkout tests-passed
Switched to a new branch 'tests-passed'
I, [2023-01-09T01:24:34.269503 #1]  INFO -- : Branch 'tests-passed' set up to track remote branch 'tests-passed' from 'origin'.

I, [2023-01-09T01:24:34.269940 #1]  INFO -- : > cd /var/www/discourse && mkdir -p tmp
I, [2023-01-09T01:24:34.275253 #1]  INFO -- : 
I, [2023-01-09T01:24:34.275637 #1]  INFO -- : > cd /var/www/discourse && chown discourse:www-data tmp
I, [2023-01-09T01:24:34.280161 #1]  INFO -- : 
I, [2023-01-09T01:24:34.280520 #1]  INFO -- : > cd /var/www/discourse && mkdir -p tmp/pids
I, [2023-01-09T01:24:34.284884 #1]  INFO -- : 
I, [2023-01-09T01:24:34.285261 #1]  INFO -- : > cd /var/www/discourse && mkdir -p tmp/sockets
I, [2023-01-09T01:24:34.289641 #1]  INFO -- : 
I, [2023-01-09T01:24:34.289776 #1]  INFO -- : > cd /var/www/discourse && touch tmp/.gitkeep
I, [2023-01-09T01:24:34.294382 #1]  INFO -- : 
I, [2023-01-09T01:24:34.294585 #1]  INFO -- : > cd /var/www/discourse && mkdir -p                    /shared/log/rails
I, [2023-01-09T01:24:34.298440 #1]  INFO -- : 
I, [2023-01-09T01:24:34.298630 #1]  INFO -- : > cd /var/www/discourse && bash -c "touch -a           /shared/log/rails/{production,production_errors,unicorn.stdout,unicorn.stderr,sidekiq}.log"
I, [2023-01-09T01:24:34.305505 #1]  INFO -- : 
I, [2023-01-09T01:24:34.305635 #1]  INFO -- : > cd /var/www/discourse && bash -c "ln    -s           /shared/log/rails/{production,production_errors,unicorn.stdout,unicorn.stderr,sidekiq}.log /var/www/discourse/log"
I, [2023-01-09T01:24:34.311597 #1]  INFO -- : 
I, [2023-01-09T01:24:34.311732 #1]  INFO -- : > cd /var/www/discourse && bash -c "mkdir -p           /shared/{uploads,backups}"
I, [2023-01-09T01:24:34.317886 #1]  INFO -- : 
I, [2023-01-09T01:24:34.318011 #1]  INFO -- : > cd /var/www/discourse && bash -c "ln    -s           /shared/{uploads,backups} /var/www/discourse/public"
I, [2023-01-09T01:24:34.324093 #1]  INFO -- : 
I, [2023-01-09T01:24:34.324477 #1]  INFO -- : > cd /var/www/discourse && bash -c "mkdir -p           /shared/tmp/{backups,restores}"
I, [2023-01-09T01:24:34.330970 #1]  INFO -- : 
I, [2023-01-09T01:24:34.331318 #1]  INFO -- : > cd /var/www/discourse && bash -c "ln    -s           /shared/tmp/{backups,restores} /var/www/discourse/tmp"
I, [2023-01-09T01:24:34.338719 #1]  INFO -- : 
I, [2023-01-09T01:24:34.338924 #1]  INFO -- : > cd /var/www/discourse && chown -R discourse:www-data /shared/log/rails /shared/uploads /shared/backups /shared/tmp
I, [2023-01-09T01:24:35.406006 #1]  INFO -- : 
I, [2023-01-09T01:24:35.406244 #1]  INFO -- : > cd /var/www/discourse && [ ! -d public/plugins ] || find public/plugins/ -maxdepth 1 -xtype l -delete
I, [2023-01-09T01:24:35.415072 #1]  INFO -- : 
I, [2023-01-09T01:24:35.415458 #1]  INFO -- : Replacing # redis with sv start redis || exit 1 in /etc/service/unicorn/run
I, [2023-01-09T01:24:35.416573 #1]  INFO -- : > cd /var/www/discourse/plugins && mkdir -p plugins
I, [2023-01-09T01:24:35.422220 #1]  INFO -- : 
I, [2023-01-09T01:24:35.422355 #1]  INFO -- : > cd /var/www/discourse/plugins && git clone https://github.com/discourse/docker_manager.git
Cloning into 'docker_manager'...
I, [2023-01-09T01:24:36.585720 #1]  INFO -- : 
I, [2023-01-09T01:24:36.586299 #1]  INFO -- : > cd /var/www/discourse/plugins && git clone https://github.com/discourse/discourse-solved.git
Cloning into 'discourse-solved'...
I, [2023-01-09T01:24:37.349634 #1]  INFO -- : 
I, [2023-01-09T01:24:37.350162 #1]  INFO -- : > cd /var/www/discourse/plugins && sudo -E -u discourse git clone https://github.com/discourse/discourse-canned-replies.git
Cloning into 'discourse-canned-replies'...
warning: unable to access '/root/.config/git/attributes': Permission denied
I, [2023-01-09T01:24:38.160428 #1]  INFO -- : 
I, [2023-01-09T01:24:38.160771 #1]  INFO -- : > cd /var/www/discourse/plugins && sudo -E -u discourse git clone https://github.com/discourse/discourse-saved-searches.git
Cloning into 'discourse-saved-searches'...
warning: unable to access '/root/.config/git/attributes': Permission denied
I, [2023-01-09T01:24:38.831562 #1]  INFO -- : 
I, [2023-01-09T01:24:38.831772 #1]  INFO -- : > cd /var/www/discourse/plugins && sudo -E -u discourse git clone https://github.com/discourse/discourse-cakeday.git
Cloning into 'discourse-cakeday'...
warning: unable to access '/root/.config/git/attributes': Permission denied
I, [2023-01-09T01:24:39.532439 #1]  INFO -- : 
I, [2023-01-09T01:24:39.532805 #1]  INFO -- : > cd /var/www/discourse/plugins && sudo -E -u discourse git clone https://github.com/discourse/discourse-bcc.git
Cloning into 'discourse-bcc'...
warning: unable to access '/root/.config/git/attributes': Permission denied
I, [2023-01-09T01:24:40.154314 #1]  INFO -- : 
I, [2023-01-09T01:24:40.154550 #1]  INFO -- : > cd /var/www/discourse/plugins && sudo -E -u discourse git clone https://github.com/discourse/discourse-footnote.git
Cloning into 'discourse-footnote'...
warning: unable to access '/root/.config/git/attributes': Permission denied
I, [2023-01-09T01:24:40.815415 #1]  INFO -- : 
I, [2023-01-09T01:24:40.815710 #1]  INFO -- : > cd /var/www/discourse/plugins && sudo -E -u discourse git clone https://github.com/discourse/discourse-no-bump.git
Cloning into 'discourse-no-bump'...
warning: unable to access '/root/.config/git/attributes': Permission denied
I, [2023-01-09T01:24:41.457886 #1]  INFO -- : 
I, [2023-01-09T01:24:41.458104 #1]  INFO -- : > cd /var/www/discourse/plugins && sudo -E -u discourse git clone https://github.com/discourse/discourse-user-notes.git
Cloning into 'discourse-user-notes'...
warning: unable to access '/root/.config/git/attributes': Permission denied
I, [2023-01-09T01:24:42.132928 #1]  INFO -- : 
I, [2023-01-09T01:24:42.133386 #1]  INFO -- : > cd /var/www/discourse/plugins && sudo -E -u discourse git clone https://github.com/discourse/discourse-tooltips.git
Cloning into 'discourse-tooltips'...
warning: unable to access '/root/.config/git/attributes': Permission denied
I, [2023-01-09T01:24:42.837714 #1]  INFO -- : 
I, [2023-01-09T01:24:42.837921 #1]  INFO -- : > cd /var/www/discourse/plugins && sudo -E -u discourse git clone https://github.com/discourse/discourse-adplugin.git
Cloning into 'discourse-adplugin'...
warning: unable to access '/root/.config/git/attributes': Permission denied
I, [2023-01-09T01:24:43.578307 #1]  INFO -- : 
I, [2023-01-09T01:24:43.578602 #1]  INFO -- : > cd /var/www/discourse/plugins && sudo -E -u discourse git clone https://github.com/discourse/discourse-yearly-review.git
Cloning into 'discourse-yearly-review'...
warning: unable to access '/root/.config/git/attributes': Permission denied
I, [2023-01-09T01:24:44.295040 #1]  INFO -- : 
I, [2023-01-09T01:24:44.295608 #1]  INFO -- : > cp /var/www/discourse/config/nginx.sample.conf /etc/nginx/conf.d/discourse.conf
I, [2023-01-09T01:24:44.300575 #1]  INFO -- : 
I, [2023-01-09T01:24:44.300991 #1]  INFO -- : > rm /etc/nginx/sites-enabled/default
I, [2023-01-09T01:24:44.304367 #1]  INFO -- : 
I, [2023-01-09T01:24:44.304686 #1]  INFO -- : > mkdir -p /var/nginx/cache
I, [2023-01-09T01:24:44.308212 #1]  INFO -- : 
I, [2023-01-09T01:24:44.308941 #1]  INFO -- : Replacing pid /run/nginx.pid; with daemon off; in /etc/nginx/nginx.conf
I, [2023-01-09T01:24:44.309341 #1]  INFO -- : Replacing (?m-ix:upstream[^\}]+\}) with upstream discourse { server 127.0.0.1:3000; } in /etc/nginx/conf.d/discourse.conf
I, [2023-01-09T01:24:44.309679 #1]  INFO -- : Replacing (?-mix:server_name.+$) with server_name _ ; in /etc/nginx/conf.d/discourse.conf
I, [2023-01-09T01:24:44.310410 #1]  INFO -- : Replacing (?-mix:client_max_body_size.+$) with client_max_body_size $upload_size ; in /etc/nginx/conf.d/discourse.conf
I, [2023-01-09T01:24:44.310844 #1]  INFO -- : Replacing (?-mix:worker_connections.+$) with worker_connections $nginx_worker_connections ; in /etc/nginx/nginx.conf
I, [2023-01-09T01:24:44.311086 #1]  INFO -- : > echo "done configuring web"
I, [2023-01-09T01:24:44.313609 #1]  INFO -- : done configuring web

I, [2023-01-09T01:24:44.313969 #1]  INFO -- : > cd /var/www/discourse && gem install bundler --conservative -v $(awk '/BUNDLED WITH/ { getline; gsub(/ /,""); print $0 }' Gemfile.lock)
ERROR:  Loading command: install (RuntimeError)
        failed to get urandom
        /usr/local/lib/ruby/3.1.0/securerandom.rb:75:in `urandom'
        /usr/local/lib/ruby/3.1.0/securerandom.rb:75:in `singleton class'
        /usr/local/lib/ruby/3.1.0/securerandom.rb:42:in `<module:SecureRandom>'
        /usr/local/lib/ruby/3.1.0/securerandom.rb:41:in `<top (required)>'
        <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
        <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
        /usr/local/lib/ruby/3.1.0/resolv.rb:8:in `<top (required)>'
        <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
        <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
        /usr/local/lib/ruby/3.1.0/net/http.rb:25:in `<top (required)>'
        <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
        <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
        /usr/local/lib/ruby/3.1.0/rubygems/request.rb:2:in `<top (required)>'
        /usr/local/lib/ruby/3.1.0/rubygems/remote_fetcher.rb:3:in `require_relative'
        /usr/local/lib/ruby/3.1.0/rubygems/remote_fetcher.rb:3:in `<top (required)>'
        /usr/local/lib/ruby/3.1.0/rubygems/spec_fetcher.rb:2:in `require_relative'
        /usr/local/lib/ruby/3.1.0/rubygems/spec_fetcher.rb:2:in `<top (required)>'
        /usr/local/lib/ruby/3.1.0/rubygems/dependency_installer.rb:6:in `require_relative'
        /usr/local/lib/ruby/3.1.0/rubygems/dependency_installer.rb:6:in `<top (required)>'
        /usr/local/lib/ruby/3.1.0/rubygems/commands/install_command.rb:4:in `require_relative'
        /usr/local/lib/ruby/3.1.0/rubygems/commands/install_command.rb:4:in `<top (required)>'
        <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
        <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
        /usr/local/lib/ruby/3.1.0/rubygems/command_manager.rb:228:in `load_and_instantiate'
        /usr/local/lib/ruby/3.1.0/rubygems/command_manager.rb:135:in `[]'
        /usr/local/lib/ruby/3.1.0/rubygems/command_manager.rb:201:in `find_command'
        /usr/local/lib/ruby/3.1.0/rubygems/command_manager.rb:183:in `process_args'
        /usr/local/lib/ruby/3.1.0/rubygems/command_manager.rb:149:in `run'
        /usr/local/lib/ruby/3.1.0/rubygems/gem_runner.rb:51:in `run'
        /usr/local/bin/gem:21:in `<main>'
ERROR:  While executing gem ... (NoMethodError)
    undefined method `deprecated?' for nil:NilClass

      cmd.deprecation_warning if cmd.deprecated?
                                    ^^^^^^^^^^^^
        /usr/local/lib/ruby/3.1.0/rubygems/command_manager.rb:184:in `process_args'
        /usr/local/lib/ruby/3.1.0/rubygems/command_manager.rb:149:in `run'
        /usr/local/lib/ruby/3.1.0/rubygems/gem_runner.rb:51:in `run'
        /usr/local/bin/gem:21:in `<main>'
I, [2023-01-09T01:24:44.603237 #1]  INFO -- : 
I, [2023-01-09T01:24:44.603710 #1]  INFO -- : Terminating async processes
I, [2023-01-09T01:24:44.603794 #1]  INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main pid: 42
I, [2023-01-09T01:24:44.603862 #1]  INFO -- : Sending TERM to exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf pid: 103
2023-01-09 01:24:44.603 UTC [42] LOG:  received fast shutdown request
103:signal-handler (1673227484) Received SIGTERM scheduling shutdown...
2023-01-09 01:24:44.608 UTC [42] LOG:  aborting any active transactions
2023-01-09 01:24:44.610 UTC [42] LOG:  background worker "logical replication launcher" (PID 51) exited with exit code 1
2023-01-09 01:24:44.613 UTC [46] LOG:  shutting down
103:M 09 Jan 2023 01:24:44.669 # User requested shutdown...
103:M 09 Jan 2023 01:24:44.669 * Saving the final RDB snapshot before exiting.
2023-01-09 01:24:44.697 UTC [42] LOG:  database system is shut down
103:M 09 Jan 2023 01:24:45.821 * DB saved on disk
103:M 09 Jan 2023 01:24:45.821 # Redis is now ready to exit, bye bye...


FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && gem install bundler --conservative -v $(awk '/BUNDLED WITH/ { getline; gsub(/ /,""); print $0 }' Gemfile.lock) failed with return #<Process::Status: pid 383 exit 1>
Location of failure: /usr/local/lib/ruby/gems/3.1.0/gems/pups-1.1.1/lib/pups/exec_command.rb:117:in `spawn'
exec failed with the params {"cd"=>"$home", "hook"=>"web", "cmd"=>["gem install bundler --conservative -v $(awk '/BUNDLED WITH/ { getline; gsub(/ /,\"\"); print $0 }' Gemfile.lock)", "find $home ! -user discourse -exec chown discourse {} \\+"]}
bootstrap failed with exit code 1
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.```

hiddenseal · January 9, 2023, 2:07am

Did you try to search for ways to resolve

unable to access '/root/.config/git/attributes': Permission denied

? There are some posts on Stackoverflow related to this.

I understand what you feel. While you wait for someone to reply to your marketplace request, maybe you can try to create a new droplet, install Docker, install Discourse & restore it from backup (the discourse backup, not the whole server image)? That wouldn’t solve your issue per se and will not give you an understanding of why things broke, but at least that may give you a working instance sooner that you’ll get help. I’m suggesting this as I was initially hesitant and was genuinely surprised at how easy it is to migrate.

pacharanero · January 9, 2023, 1:12pm

I would agree, definitely worth trying a full reinstall from a backup. You desperately need to upgrade that server to at least 22.04LTS, it is significantly out of date and running a kernel 2 major versions behind current. It’s likely to have numerous incompatibilities with newer versions of Docker, Redis, Postgres possibly even., as well as being likely to have known security vulnerabilities.

A new Droplet would be the ideal way, this will of course change your IP address so you would need to be able to make changes to DNS settings to wire this up to your domain name.

If you don’t have external backups on AWS S3 then they will be on your existing droplet in the directory /var/discourse/shared/standalone/backups/default. SFTP can be used to get the backup off your server and onto the new one.

littleviolette · January 10, 2023, 5:43pm

Update: We fixed the issue by copying the Discourse instance to a new droplet with @pfaffman’s help. Thank you all for your advice!

system · February 9, 2023, 5:44pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Issue when installing new Discourse instance installation	7	964	April 2, 2023
Trouble installing discourse bug arm	3	501	February 6, 2023
Unable to boostrap Discourse since last upgrade installation	5	513	January 22, 2023
Problem after Discourse update installation	17	1450	March 8, 2021
Build failed for specific commit hash installation	9	507	April 12, 2023

HELP: Discourse Update broke app -- fails to rebuild

Related Topics