Upgrade from 2.8.0.beta10 to 3.1.0.beta1 failed on Ubuntu 16.04

My self-hosted discourse instance is running on a digital ocean droplet on an EOL’d version of Ubuntu 16.04 so I’m ok if there’s no way to come back from this.

However, I only have a database backup from a week ago so I’d at least like to figure out whether there’s a way to start postgres within the container so I can take a manual backup. Or even downgrade to 2.8.0.beta10.

Confirmation of Discourse versions

Hooray, a new version of Discourse is available!

Your version: 2.8.0.beta10
New version: 3.1.0.beta1

Confirmation of Ubuntu version

root@discourse:/var/discourse# lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 16.04.7 LTS
Release:	16.04
Codename:	xenial

During the upgrade/rebuild process, I encountered a few warnings but I’m not sure what’s actually causing the bootstrap failure.

  1. Redis is already listening on port 6379 (which definitely is not possible)

    129:M 25 Jan 2023 21:43:58.413 # Warning: Could not create server TCP listening socket *:6379: bind: Address already in use
    129:M 25 Jan 2023 21:43:58.413 # Failed listening on port 6379 (TCP), aborting.
    
  2. Weird name resolution issue

    I, [2023-01-25T21:44:02.427467 #1]  INFO -- : > cd /var/www/discourse && git fetch --depth 1 origin tests-passed
    fatal: unable to access 'https://github.com/discourse/discourse.git/': Could not resolve host: github.com
    I, [2023-01-25T21:44:22.470200 #1]  INFO -- : 
    I, [2023-01-25T21:44:22.471174 #1]  INFO -- : Terminating async processes
    

The container is running

root@discourse:/var/discourse# docker ps
CONTAINER ID        IMAGE                 COMMAND             CREATED             STATUS              PORTS                                                            NAMES
ac9571b7a51d        local_discourse/app   "/sbin/boot"        12 months ago       Up 13 minutes       0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:2222->22/tcp   app

I can “enter” the container and it seems like most of the services are running.

root@discourse-app:/var/www/discourse# ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 22:09 pts/0    00:00:00 /bin/bash /sbin/boot
root       532     1  0 22:09 pts/0    00:00:00 /usr/bin/runsvdir -P /etc/service
root       533   532  0 22:09 ?        00:00:00 runsv cron
root       534   532  0 22:09 ?        00:00:00 runsv nginx
root       535   532  0 22:09 ?        00:00:01 runsv postgres
root       536   532  0 22:09 ?        00:00:00 runsv unicorn
root       537   532  0 22:09 ?        00:00:01 runsv redis
root       538   532  0 22:09 ?        00:00:00 runsv rsyslog
root       539   533  0 22:09 ?        00:00:00 cron -f
root       540   538  0 22:09 ?        00:00:00 rsyslogd -n
root       541   537  0 22:09 ?        00:00:00 svlogd /var/log/redis
root       544   534  0 22:09 ?        00:00:00 nginx: master process /usr/sbin/nginx
root       546   535  0 22:09 ?        00:00:00 svlogd /var/log/postgres
www-data   554   544  0 22:09 ?        00:00:00 nginx: worker process
www-data   555   544  0 22:09 ?        00:00:00 nginx: worker process
www-data   556   544  0 22:09 ?        00:00:00 nginx: cache manager process
root      8350     0  0 22:30 pts/1    00:00:00 /bin/bash --login
root      8894   536  0 22:31 ?        00:00:00 /bin/bash ./run
root      8895  8894  0 22:31 ?        00:00:00 sv start redis
root      8913  8350  0 22:31 pts/1    00:00:00 ps -ef

Tailing the Redis logs

root@discourse-app:/var/www/discourse# tail -f /var/log/redis/current 
8992:M 25 Jan 2023 22:31:53.807 # Fatal error loading the DB: Invalid argument. Exiting.
8998:C 25 Jan 2023 22:31:54.815 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
8998:C 25 Jan 2023 22:31:54.815 # Redis version=6.2.6, bits=64, commit=00000000, modified=0, pid=8998, just started
8998:C 25 Jan 2023 22:31:54.815 # Configuration loaded
8998:M 25 Jan 2023 22:31:54.816 * monotonic clock: POSIX clock_gettime
8998:M 25 Jan 2023 22:31:54.817 * Running mode=standalone, port=6379.
8998:M 25 Jan 2023 22:31:54.817 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
8998:M 25 Jan 2023 22:31:54.817 # Server initialized
8998:M 25 Jan 2023 22:31:54.817 # Can't handle RDB format version 10

Tailing the postgres logs

root@discourse-app:/var/www/discourse# tail -f /var/log/postgres/current 
2023-01-25 22:32:58.712 UTC [9409] HINT:  The server must be started by the user that owns the data directory.
2023-01-25 22:32:59.727 GMT [9416] LOG:  skipping missing configuration file "/shared/postgres_data/postgresql.auto.conf"
2023-01-25 22:32:59.728 UTC [9416] FATAL:  data directory "/shared/postgres_data" has wrong ownership

The front page of the Discourse instance is being returned but all other pages fail to load.

And here’s the full log from the discourse-doctor script.

root@discourse:/var/discourse# more /tmp/discourse-debug.txt 
DISCOURSE DOCTOR Wed Jan 25 17:08:44 EST 2023
OS: Linux discourse 3.13.0-43-generic #72-Ubuntu SMP Mon Dec 8 19:35:06 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux


Found containers/app.yml

==================== YML SETTINGS ====================
DISCOURSE_HOSTNAME=discuss.openboxes.com
SMTP_ADDRESS=REDACTED
DEVELOPER_EMAILS=REDACTED 
SMTP_PASSWORD=REDACTED
SMTP_PORT=587 
SMTP_USER_NAME=REDACTED
LETSENCRYPT_ACCOUNT_EMAIL=REDACTED 

==================== DOCKER INFO ====================
DOCKER VERSION: Docker version 19.03.13, build 4484c46d9d

DOCKER PROCESSES (docker ps -a)

CONTAINER ID        IMAGE                 COMMAND             CREATED             STATUS                      PORTS               NAMES
ac9571b7a51d        local_discourse/app   "/sbin/boot"        12 months ago       Exited (8) 19 minutes ago                       app

==================== SERIOUS PROBLEM!!!! ====================
app not running!
Attempting to rebuild
==================== REBUILD LOG ====================
x86_64 arch detected.
Ensuring launcher is up to date
Fetching origin
Launcher is up-to-date
Stopping old container
+ /usr/bin/docker stop -t 60 app
app
2.0.20230116-0051: Pulling from discourse/base
Digest: sha256:9e75876adb52427902a6d76dab731c4888af811d7e5ad9ee6efcec4ce6db986f
Status: Image is up to date for discourse/base:2.0.20230116-0051
docker.io/discourse/base:2.0.20230116-0051
/usr/local/lib/ruby/gems/3.1.0/gems/pups-1.1.1/lib/pups.rb
/usr/local/bin/pups --stdin
I, [2023-01-25T22:08:55.819088 #1]  INFO -- : Reading from stdin
I, [2023-01-25T22:08:55.829379 #1]  INFO -- : > locale-gen $LANG && update-locale
I, [2023-01-25T22:08:55.866770 #1]  INFO -- : Generating locales (this might take a while)...
Generation complete.

I, [2023-01-25T22:08:55.867033 #1]  INFO -- : > mkdir -p /shared/postgres_run
I, [2023-01-25T22:08:55.871591 #1]  INFO -- : 
I, [2023-01-25T22:08:55.872103 #1]  INFO -- : > chown postgres:postgres /shared/postgres_run
I, [2023-01-25T22:08:55.875971 #1]  INFO -- : 
I, [2023-01-25T22:08:55.876362 #1]  INFO -- : > chmod 775 /shared/postgres_run
I, [2023-01-25T22:08:55.880234 #1]  INFO -- : 
I, [2023-01-25T22:08:55.880510 #1]  INFO -- : > rm -fr /var/run/postgresql
I, [2023-01-25T22:08:55.883570 #1]  INFO -- : 
I, [2023-01-25T22:08:55.883881 #1]  INFO -- : > ln -s /shared/postgres_run /var/run/postgresql
I, [2023-01-25T22:08:55.886827 #1]  INFO -- : 
I, [2023-01-25T22:08:55.887174 #1]  INFO -- : > socat /dev/null UNIX-CONNECT:/shared/postgres_run/.s.PGSQL.5432 || exit 0 && echo postgres already running stop container ; exit 1
2023/01/25 22:08:55 socat[18] E connect(6, AF=1 "/shared/postgres_run/.s.PGSQL.5432", 36): No such file or directory
I, [2023-01-25T22:08:55.893057 #1]  INFO -- : 
I, [2023-01-25T22:08:55.893329 #1]  INFO -- : > rm -fr /shared/postgres_run/.s*
I, [2023-01-25T22:08:55.897513 #1]  INFO -- : 
I, [2023-01-25T22:08:55.897762 #1]  INFO -- : > rm -fr /shared/postgres_run/*.pid
I, [2023-01-25T22:08:55.901563 #1]  INFO -- : 
I, [2023-01-25T22:08:55.901861 #1]  INFO -- : > mkdir -p /shared/postgres_run/13-main.pg_stat_tmp
I, [2023-01-25T22:08:55.905376 #1]  INFO -- : 
I, [2023-01-25T22:08:55.905731 #1]  INFO -- : > chown postgres:postgres /shared/postgres_run/13-main.pg_stat_tmp
I, [2023-01-25T22:08:55.908318 #1]  INFO -- : 
I, [2023-01-25T22:08:55.914950 #1]  INFO -- : File > /etc/service/postgres/run  chmod: +x  chown: 
I, [2023-01-25T22:08:55.920932 #1]  INFO -- : File > /etc/service/postgres/log/run  chmod: +x  chown: 
I, [2023-01-25T22:08:55.926958 #1]  INFO -- : File > /etc/runit/3.d/99-postgres  chmod: +x  chown: 
I, [2023-01-25T22:08:55.933701 #1]  INFO -- : File > /root/upgrade_postgres  chmod: +x  chown: 
I, [2023-01-25T22:08:55.934071 #1]  INFO -- : > chown -R root /var/lib/postgresql/13/main
I, [2023-01-25T22:08:56.043908 #1]  INFO -- : 
I, [2023-01-25T22:08:56.044192 #1]  INFO -- : > [ ! -e /shared/postgres_data ] && install -d -m 0755 -o postgres -g postgres /shared/postgres_data && sudo -E -u postgres /usr/lib/postgresql/13/bin/initdb -D /sha
red/postgres_data || exit 0
I, [2023-01-25T22:08:56.047007 #1]  INFO -- : 
I, [2023-01-25T22:08:56.047154 #1]  INFO -- : > chown -R postgres:postgres /shared/postgres_data
I, [2023-01-25T22:08:56.062900 #1]  INFO -- : 
I, [2023-01-25T22:08:56.063289 #1]  INFO -- : > chown -R postgres:postgres /var/run/postgresql
I, [2023-01-25T22:08:56.066785 #1]  INFO -- : 
I, [2023-01-25T22:08:56.067108 #1]  INFO -- : > /root/upgrade_postgres
I, [2023-01-25T22:08:56.072305 #1]  INFO -- : 
I, [2023-01-25T22:08:56.072560 #1]  INFO -- : > rm /root/upgrade_postgres
I, [2023-01-25T22:08:56.075197 #1]  INFO -- : 
I, [2023-01-25T22:08:56.075590 #1]  INFO -- : Replacing data_directory = '/var/lib/postgresql/13/main' with data_directory = '/shared/postgres_data' in /etc/postgresql/13/main/postgresql.conf
I, [2023-01-25T22:08:56.076117 #1]  INFO -- : Replacing (?-mix:#?listen_addresses *=.*) with listen_addresses = '*' in /etc/postgresql/13/main/postgresql.conf
I, [2023-01-25T22:08:56.076929 #1]  INFO -- : Replacing (?-mix:#?synchronous_commit *=.*) with synchronous_commit = $db_synchronous_commit in /etc/postgresql/13/main/postgresql.conf
I, [2023-01-25T22:08:56.077585 #1]  INFO -- : Replacing (?-mix:#?shared_buffers *=.*) with shared_buffers = $db_shared_buffers in /etc/postgresql/13/main/postgresql.conf
I, [2023-01-25T22:08:56.078367 #1]  INFO -- : Replacing (?-mix:#?work_mem *=.*) with work_mem = $db_work_mem in /etc/postgresql/13/main/postgresql.conf
I, [2023-01-25T22:08:56.078805 #1]  INFO -- : Replacing (?-mix:#?default_text_search_config *=.*) with default_text_search_config = '$db_default_text_search_config' in /etc/postgresql/13/main/postgresql.conf
I, [2023-01-25T22:08:56.079298 #1]  INFO -- : > install -d -m 0755 -o postgres -g postgres /shared/postgres_backup
I, [2023-01-25T22:08:56.083823 #1]  INFO -- : 
I, [2023-01-25T22:08:56.084723 #1]  INFO -- : Replacing (?-mix:#?checkpoint_segments *=.*) with checkpoint_segments = $db_checkpoint_segments in /etc/postgresql/13/main/postgresql.conf
I, [2023-01-25T22:08:56.085367 #1]  INFO -- : Replacing (?-mix:#?logging_collector *=.*) with logging_collector = $db_logging_collector in /etc/postgresql/13/main/postgresql.conf
I, [2023-01-25T22:08:56.086094 #1]  INFO -- : Replacing (?-mix:#?log_min_duration_statement *=.*) with log_min_duration_statement = $db_log_min_duration_statement in /etc/postgresql/13/main/postgresql.conf
I, [2023-01-25T22:08:56.086763 #1]  INFO -- : Replacing (?-mix:^#local +replication +postgres +peer$) with local replication postgres  peer in /etc/postgresql/13/main/pg_hba.conf
I, [2023-01-25T22:08:56.087355 #1]  INFO -- : Replacing (?-mix:^host.*all.*all.*127.*$) with host all all 0.0.0.0/0 md5 in /etc/postgresql/13/main/pg_hba.conf
I, [2023-01-25T22:08:56.087874 #1]  INFO -- : Replacing (?-mix:^host.*all.*all.*::1\/128.*$) with host all all ::/0 md5 in /etc/postgresql/13/main/pg_hba.conf
I, [2023-01-25T22:08:56.088260 #1]  INFO -- : > HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgre
sql/13/main
I, [2023-01-25T22:08:56.091044 #1]  INFO -- : > sleep 5
2023-01-25 22:08:56.123 UTC [41] LOG:  starting PostgreSQL 13.9 (Debian 13.9-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
2023-01-25 22:08:56.123 UTC [41] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2023-01-25 22:08:56.124 UTC [41] LOG:  listening on IPv6 address "::", port 5432
2023-01-25 22:08:56.126 UTC [41] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2023-01-25 22:08:56.133 UTC [44] LOG:  database system was shut down at 2023-01-25 21:44:22 UTC
2023-01-25 22:08:56.138 UTC [41] LOG:  database system is ready to accept connections
I, [2023-01-25T22:09:01.095066 #1]  INFO -- : 
I, [2023-01-25T22:09:01.096182 #1]  INFO -- : > su postgres -c 'createdb discourse' || true
2023-01-25 22:09:01.202 UTC [54] postgres@postgres ERROR:  database "discourse" already exists
2023-01-25 22:09:01.202 UTC [54] postgres@postgres STATEMENT:  CREATE DATABASE discourse;
createdb: error: database creation failed: ERROR:  database "discourse" already exists
I, [2023-01-25T22:09:01.212185 #1]  INFO -- : 
I, [2023-01-25T22:09:01.213124 #1]  INFO -- : > su postgres -c 'psql discourse -c "create user discourse;"' || true
2023-01-25 22:09:01.294 UTC [58] postgres@discourse ERROR:  role "discourse" already exists
2023-01-25 22:09:01.294 UTC [58] postgres@discourse STATEMENT:  create user discourse;
ERROR:  role "discourse" already exists
I, [2023-01-25T22:09:01.297444 #1]  INFO -- : 
I, [2023-01-25T22:09:01.298068 #1]  INFO -- : > su postgres -c 'psql discourse -c "grant all privileges on database discourse to discourse;"' || true
I, [2023-01-25T22:09:01.425549 #1]  INFO -- : GRANT

I, [2023-01-25T22:09:01.427958 #1]  INFO -- : > su postgres -c 'psql discourse -c "alter schema public owner to discourse;"'
I, [2023-01-25T22:09:01.534241 #1]  INFO -- : ALTER SCHEMA

I, [2023-01-25T22:09:01.537260 #1]  INFO -- : > su postgres -c 'psql template1 -c "create extension if not exists hstore;"'
NOTICE:  extension "hstore" already exists, skipping
I, [2023-01-25T22:09:01.641126 #1]  INFO -- : CREATE EXTENSION

I, [2023-01-25T22:09:01.641850 #1]  INFO -- : > su postgres -c 'psql template1 -c "create extension if not exists pg_trgm;"'
NOTICE:  extension "pg_trgm" already exists, skipping
I, [2023-01-25T22:09:01.738202 #1]  INFO -- : CREATE EXTENSION

I, [2023-01-25T22:09:01.738449 #1]  INFO -- : > su postgres -c 'psql discourse -c "create extension if not exists hstore;"'
NOTICE:  extension "hstore" already exists, skipping
I, [2023-01-25T22:09:01.847287 #1]  INFO -- : CREATE EXTENSION

I, [2023-01-25T22:09:01.847838 #1]  INFO -- : > su postgres -c 'psql discourse -c "create extension if not exists pg_trgm;"'
NOTICE:  extension "pg_trgm" already exists, skipping
I, [2023-01-25T22:09:01.944897 #1]  INFO -- : CREATE EXTENSION

I, [2023-01-25T22:09:01.945538 #1]  INFO -- : > sudo -u postgres psql discourse
I, [2023-01-25T22:09:01.950519 #1]  INFO -- : update pg_database set encoding = pg_char_to_encoding('UTF8') where datname = 'discourse' AND encoding = pg_char_to_encoding('SQL_ASCII');

I, [2023-01-25T22:09:02.097379 #1]  INFO -- : File > /var/lib/postgresql/take-database-backup  chmod: +x  chown: postgres:postgres
I, [2023-01-25T22:09:02.104341 #1]  INFO -- : File > /var/spool/cron/crontabs/postgres  chmod:   chown: 
I, [2023-01-25T22:09:02.104747 #1]  INFO -- : > echo postgres installed!
I, [2023-01-25T22:09:02.109338 #1]  INFO -- : postgres installed!

I, [2023-01-25T22:09:02.121260 #1]  INFO -- : File > /etc/service/redis/run  chmod: +x  chown: 
I, [2023-01-25T22:09:02.140878 #1]  INFO -- : File > /etc/service/redis/log/run  chmod: +x  chown: 
I, [2023-01-25T22:09:02.149864 #1]  INFO -- : File > /etc/runit/3.d/10-redis  chmod: +x  chown: 
I, [2023-01-25T22:09:02.150246 #1]  INFO -- : Replacing daemonize yes with  in /etc/redis/redis.conf
I, [2023-01-25T22:09:02.151024 #1]  INFO -- : Replacing (?-mix:^pidfile.*$) with  in /etc/redis/redis.conf
I, [2023-01-25T22:09:02.152301 #1]  INFO -- : > install -d -m 0755 -o redis -g redis /shared/redis_data
I, [2023-01-25T22:09:02.161829 #1]  INFO -- : 
I, [2023-01-25T22:09:02.162849 #1]  INFO -- : Replacing (?-mix:^logfile.*$) with logfile "" in /etc/redis/redis.conf
I, [2023-01-25T22:09:02.163527 #1]  INFO -- : Replacing (?-mix:^bind .*$) with  in /etc/redis/redis.conf
I, [2023-01-25T22:09:02.165384 #1]  INFO -- : Replacing (?-mix:^dir .*$) with dir /shared/redis_data in /etc/redis/redis.conf
I, [2023-01-25T22:09:02.167458 #1]  INFO -- : Replacing (?-mix:^protected-mode yes) with protected-mode no in /etc/redis/redis.conf
I, [2023-01-25T22:09:02.169383 #1]  INFO -- : Replacing # io-threads 4 with io-threads $redis_io_threads in /etc/redis/redis.conf
I, [2023-01-25T22:09:02.171253 #1]  INFO -- : > echo redis installed
I, [2023-01-25T22:09:02.184461 #1]  INFO -- : redis installed

I, [2023-01-25T22:09:02.184796 #1]  INFO -- : > cat /etc/redis/redis.conf | grep logfile
I, [2023-01-25T22:09:02.200319 #1]  INFO -- : logfile ""

I, [2023-01-25T22:09:02.200861 #1]  INFO -- : > exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf
I, [2023-01-25T22:09:02.204295 #1]  INFO -- : > sleep 10
102:C 25 Jan 2023 22:09:02.220 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
102:C 25 Jan 2023 22:09:02.220 # Redis version=7.0.7, bits=64, commit=00000000, modified=0, pid=102, just started
102:C 25 Jan 2023 22:09:02.220 # Configuration loaded
102:M 25 Jan 2023 22:09:02.221 * monotonic clock: POSIX clock_gettime
102:M 25 Jan 2023 22:09:02.226 * Running mode=standalone, port=6379.
102:M 25 Jan 2023 22:09:02.226 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
102:M 25 Jan 2023 22:09:02.226 # Server initialized
102:M 25 Jan 2023 22:09:02.226 * Loading RDB produced by version 7.0.7
102:M 25 Jan 2023 22:09:02.227 * RDB age 1480 seconds
102:M 25 Jan 2023 22:09:02.227 * RDB memory usage when created 13.94 Mb
102:M 25 Jan 2023 22:09:02.296 * Done loading RDB, keys loaded: 7733, keys expired: 0.
102:M 25 Jan 2023 22:09:02.296 * DB loaded from disk: 0.070 seconds
102:M 25 Jan 2023 22:09:02.296 * Ready to accept connections
I, [2023-01-25T22:09:12.212150 #1]  INFO -- : 
I, [2023-01-25T22:09:12.213234 #1]  INFO -- : > thpoff echo "thpoff is installed!"
I, [2023-01-25T22:09:12.218785 #1]  INFO -- : thpoff is installed!

I, [2023-01-25T22:09:12.219549 #1]  INFO -- : > /usr/local/bin/ruby -e 'if ENV["DISCOURSE_SMTP_ADDRESS"] == "smtp.example.com"; puts "Aborting! Mail is not configured!"; exit 1; end'
I, [2023-01-25T22:09:12.334141 #1]  INFO -- : 
I, [2023-01-25T22:09:12.334618 #1]  INFO -- : > /usr/local/bin/ruby -e 'if ENV["DISCOURSE_HOSTNAME"] == "discourse.example.com"; puts "Aborting! Domain is not configured!"; exit 1; end'
I, [2023-01-25T22:09:12.453531 #1]  INFO -- : 
I, [2023-01-25T22:09:12.453796 #1]  INFO -- : > /usr/local/bin/ruby -e 'if (ENV["DISCOURSE_CDN_URL"] || "")[0..1] == "//"; puts "Aborting! CDN must have a protocol specified. Once fixed you should rebake your po
sts now to correct all posts."; exit 1; end'
I, [2023-01-25T22:09:12.558890 #1]  INFO -- : 
I, [2023-01-25T22:09:12.559328 #1]  INFO -- : > rm -f /etc/cron.d/anacron
I, [2023-01-25T22:09:12.564335 #1]  INFO -- : 
I, [2023-01-25T22:09:12.570649 #1]  INFO -- : File > /etc/cron.d/anacron  chmod:   chown: 
I, [2023-01-25T22:09:12.584977 #1]  INFO -- : File > /etc/runit/1.d/copy-env  chmod: +x  chown: 
I, [2023-01-25T22:09:12.594923 #1]  INFO -- : File > /etc/service/unicorn/run  chmod: +x  chown: 
I, [2023-01-25T22:09:12.603871 #1]  INFO -- : File > /etc/service/nginx/run  chmod: +x  chown: 
I, [2023-01-25T22:09:12.613138 #1]  INFO -- : File > /etc/runit/3.d/01-nginx  chmod: +x  chown: 
I, [2023-01-25T22:09:12.622887 #1]  INFO -- : File > /etc/runit/3.d/02-unicorn  chmod: +x  chown: 
I, [2023-01-25T22:09:12.623078 #1]  INFO -- : Replacing # postgres with sv start postgres || exit 1 in /etc/service/unicorn/run
I, [2023-01-25T22:09:12.623409 #1]  INFO -- : > exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf
I, [2023-01-25T22:09:12.627629 #1]  INFO -- : > cd /var/www/discourse && git reset --hard
129:C 25 Jan 2023 22:09:12.641 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
129:C 25 Jan 2023 22:09:12.641 # Redis version=7.0.7, bits=64, commit=00000000, modified=0, pid=129, just started
129:C 25 Jan 2023 22:09:12.641 # Configuration loaded
129:M 25 Jan 2023 22:09:12.646 * monotonic clock: POSIX clock_gettime
129:M 25 Jan 2023 22:09:12.647 # Warning: Could not create server TCP listening socket *:6379: bind: Address already in use
129:M 25 Jan 2023 22:09:12.647 # Failed listening on port 6379 (TCP), aborting.
Updating files:  54% (17646/32677)
Updating files:  77% (25162/32677)
Updating files: 100% (32677/32677)
7), done.
I, [2023-01-25T22:09:15.887253 #1]  INFO -- : HEAD is now at 2eb0a300 FIX: Use hashtags in channel archive PMs if available (#19859)

I, [2023-01-25T22:09:15.887815 #1]  INFO -- : > cd /var/www/discourse && git clean -f
I, [2023-01-25T22:09:16.049162 #1]  INFO -- : 
I, [2023-01-25T22:09:16.049650 #1]  INFO -- : > cd /var/www/discourse && git remote set-branches --add origin main
I, [2023-01-25T22:09:16.054514 #1]  INFO -- : 
I, [2023-01-25T22:09:16.055235 #1]  INFO -- : > cd /var/www/discourse && git remote set-branches origin tests-passed
I, [2023-01-25T22:09:16.060855 #1]  INFO -- : 
I, [2023-01-25T22:09:16.061437 #1]  INFO -- : > cd /var/www/discourse && git fetch --depth 1 origin tests-passed
fatal: unable to access 'https://github.com/discourse/discourse.git/': Could not resolve host: github.com
I, [2023-01-25T22:09:36.102699 #1]  INFO -- : 
I, [2023-01-25T22:09:36.103318 #1]  INFO -- : Terminating async processes
I, [2023-01-25T22:09:36.103652 #1]  INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D
 /etc/postgresql/13/main pid: 41
I, [2023-01-25T22:09:36.103917 #1]  INFO -- : Sending TERM to exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf pid: 102
2023-01-25 22:09:36.104 UTC [41] LOG:  received fast shutdown request
102:signal-handler (1674684576) Received SIGTERM scheduling shutdown...
2023-01-25 22:09:36.106 UTC [41] LOG:  aborting any active transactions
2023-01-25 22:09:36.109 UTC [41] LOG:  background worker "logical replication launcher" (PID 50) exited with exit code 1
2023-01-25 22:09:36.109 UTC [45] LOG:  shutting down
102:M 25 Jan 2023 22:09:36.117 # User requested shutdown...
102:M 25 Jan 2023 22:09:36.117 * Saving the final RDB snapshot before exiting.
2023-01-25 22:09:36.148 UTC [41] LOG:  database system is shut down
102:M 25 Jan 2023 22:09:36.214 * DB saved on disk
102:M 25 Jan 2023 22:09:36.215 # Redis is now ready to exit, bye bye...


FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && git fetch --depth 1 origin tests-passed failed with return #<Process::Status: pid 139 exit 128>
Location of failure: /usr/local/lib/ruby/gems/3.1.0/gems/pups-1.1.1/lib/pups/exec_command.rb:117:in `spawn'
exec failed with the params {"cd"=>"$home", "hook"=>"code", "cmd"=>["git reset --hard", "git clean -f", "git remote set-branches --add origin main", "git remote set-branches origin $version", "git fetch --depth 
1 origin $version", "git checkout $version", "mkdir -p tmp", "chown discourse:www-data tmp", "mkdir -p tmp/pids", "mkdir -p tmp/sockets", "touch tmp/.gitkeep", "mkdir -p                    /shared/log/rails", "b
ash -c \"touch -a           /shared/log/rails/{production,production_errors,unicorn.stdout,unicorn.stderr,sidekiq}.log\"", "bash -c \"ln    -s           /shared/log/rails/{production,production_errors,unicorn.st
dout,unicorn.stderr,sidekiq}.log $home/log\"", "bash -c \"mkdir -p           /shared/{uploads,backups}\"", "bash -c \"ln    -s           /shared/{uploads,backups} $home/public\"", "bash -c \"mkdir -p           /
shared/tmp/{backups,restores}\"", "bash -c \"ln    -s           /shared/tmp/{backups,restores} $home/tmp\"", "chown -R discourse:www-data /shared/log/rails /shared/uploads /shared/backups /shared/tmp", "[ ! -d p
ublic/plugins ] || find public/plugins/ -maxdepth 1 -xtype l -delete"]}
bootstrap failed with exit code 128
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.
616144b500d35ec963bb2bac5db3da545a2d2a28334f4cf139f9051d527176e8
==================== END REBUILD LOG ====================
Failed to rebuild app.

Checking your domain name . . .

Connection to discuss.openboxes.com succeeded.
You should probably remove any non-standard plugins and rebuild.
Attempting to restart existing container. . . 
x86_64 arch detected.

starting up existing container
+ /usr/bin/docker start app
app
Restarted the container.


==================== PLUGINS ====================
          - git clone https://github.com/discourse/docker_manager.git
          - git clone https://github.com/discourse/discourse-solved.git
          - git clone https://github.com/discourse/discourse-adplugin.git

No non-official plugins detected.

See https://github.com/discourse/discourse/blob/main/lib/plugin/metadata.rb for the official list.

========================================
Discourse version at discuss.openboxes.com: NOT FOUND
Discourse version at localhost: NOT FOUND


==================== MEMORY INFORMATION ====================
OS: Linux
RAM (MB): 2049

              total        used        free      shared  buff/cache   available
Mem:           2001         146         497          20        1358        1646
Swap:          1023           3        1020

==================== DISK SPACE CHECK ====================
---------- OS Disk Space ----------
Filesystem                 Size  Used Avail Use% Mounted on
/dev/disk/by-label/DOROOT   30G   19G  9.8G  66% /

==================== DISK INFORMATION ====================
Disk /dev/vda: 30 GiB, 32212254720 bytes, 62914560 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 29B528BA-16C4-402E-BEE9-53555C8B6F10

Device     Start      End  Sectors Size Type
/dev/vda1   2048 62912511 62910464  30G Linux filesystem

==================== END DISK INFORMATION ====================

==================== MAIL TEST ====================
For a robust test, get an address from http://www.mail-tester.com/
Mail test skipped.

==================== DONE! ====================
The output of this program may be available at http://discuss.openboxes.com/OTk4MGE0YWUzM2JmZTdj.txt
You should inspect that file carefully before sharing the URL.

You can examine the output of this script with 
LESS=-Ri less /tmp/discourse-debug.txt

BUT FIRST, make sure that you know the first three commands below!!!

Commands to know when viewing the file with the above command (called 'less'): 
q              -- quit
/error<ENTER>  -- search for the word 'error'
n              -- search for the next occurrence
g              -- go to the beginning of the file
f              -- go forward a page
b              -- go back a page
G              -- go to the end of the file
1 Like

If you can’t start the old container then copying your shared directory to the new server is likely a way to move to a new server without losing anything. There is a topic about that but I can’t find it quickly enough.

1 Like

Thank you, @pfaffman. That worked like a charm!

1 Like