3.1.x to 3.2.0 upgrade hangs/fails on 1GB instance

RBoy · February 1, 2024, 6:18am

Any assistance would be appreciated. Been trying to upgrade 3.1.5 to 3.2.0 for the last 8 hours.

It just seems to hang here:

Ensuring launcher is up to date
Fetching origin
Launcher is up-to-date
Stopping old container
+ /usr/bin/docker stop -t 600 app
app
2.0.20231218-0429: Pulling from discourse/base
Digest: sha256:468f70b9bb4c6d0c6c2bbb3efc1a5e12d145eae57bdb6946b7fe5558beb52dc1
Status: Image is up to date for discourse/base:2.0.20231218-0429
docker.io/discourse/base:2.0.20231218-0429
/usr/local/lib/ruby/gems/3.2.0/gems/pups-1.2.1/lib/pups.rb
/usr/local/bin/pups --stdin
I, [2024-02-01T05:20:13.663978 #1]  INFO -- : Reading from stdin
I, [2024-02-01T05:20:13.670645 #1]  INFO -- : > locale-gen $LANG && update-locale
I, [2024-02-01T05:20:13.712975 #1]  INFO -- : Generating locales (this might take a while)...
Generation complete.

I, [2024-02-01T05:20:13.713587 #1]  INFO -- : > mkdir -p /shared/postgres_run
I, [2024-02-01T05:20:13.717850 #1]  INFO -- :
I, [2024-02-01T05:20:13.718463 #1]  INFO -- : > chown postgres:postgres /shared/postgres_run
I, [2024-02-01T05:20:13.722273 #1]  INFO -- :
I, [2024-02-01T05:20:13.722901 #1]  INFO -- : > chmod 775 /shared/postgres_run
I, [2024-02-01T05:20:13.726117 #1]  INFO -- :
I, [2024-02-01T05:20:13.726724 #1]  INFO -- : > rm -fr /var/run/postgresql
I, [2024-02-01T05:20:13.730093 #1]  INFO -- :
I, [2024-02-01T05:20:13.730719 #1]  INFO -- : > ln -s /shared/postgres_run /var/run/postgresql
I, [2024-02-01T05:20:13.734550 #1]  INFO -- :
I, [2024-02-01T05:20:13.735116 #1]  INFO -- : > socat /dev/null UNIX-CONNECT:/shared/postgres_run/.s.PGSQL.5432 || exit 0 && echo postgres already running stop container ; exit 1
2024/02/01 05:20:13 socat[18] E connect(6, AF=1 "/shared/postgres_run/.s.PGSQL.5432", 36): No such file or directory
I, [2024-02-01T05:20:13.757786 #1]  INFO -- :
I, [2024-02-01T05:20:13.758259 #1]  INFO -- : > rm -fr /shared/postgres_run/.s*
I, [2024-02-01T05:20:13.761231 #1]  INFO -- :
I, [2024-02-01T05:20:13.761705 #1]  INFO -- : > rm -fr /shared/postgres_run/*.pid
I, [2024-02-01T05:20:13.764603 #1]  INFO -- :
I, [2024-02-01T05:20:13.765030 #1]  INFO -- : > mkdir -p /shared/postgres_run/13-main.pg_stat_tmp
I, [2024-02-01T05:20:13.767619 #1]  INFO -- :
I, [2024-02-01T05:20:13.768243 #1]  INFO -- : > chown postgres:postgres /shared/postgres_run/13-main.pg_stat_tmp
I, [2024-02-01T05:20:13.770687 #1]  INFO -- :
I, [2024-02-01T05:20:13.775750 #1]  INFO -- : File > /etc/service/postgres/run  chmod: +x  chown:
I, [2024-02-01T05:20:13.780586 #1]  INFO -- : File > /etc/service/postgres/log/run  chmod: +x  chown:
I, [2024-02-01T05:20:13.785496 #1]  INFO -- : File > /etc/runit/3.d/99-postgres  chmod: +x  chown:
I, [2024-02-01T05:20:13.790412 #1]  INFO -- : File > /root/upgrade_postgres  chmod: +x  chown:
I, [2024-02-01T05:20:13.791046 #1]  INFO -- : > chown -R root /var/lib/postgresql/13/main
I, [2024-02-01T05:20:16.705999 #1]  INFO -- :
I, [2024-02-01T05:20:16.706819 #1]  INFO -- : > [ ! -e /shared/postgres_data ] && install -d -m 0755 -o postgres -g postgres /shared/postgres_data && sudo -E -u postgres /usr/lib/postgresql/13/bin/initdb -D /shared/postgres_data || exit 0
I, [2024-02-01T05:20:16.709471 #1]  INFO -- :
I, [2024-02-01T05:20:16.709873 #1]  INFO -- : > chown -R postgres:postgres /shared/postgres_data
I, [2024-02-01T05:20:16.748408 #1]  INFO -- :
I, [2024-02-01T05:20:16.749205 #1]  INFO -- : > chown -R postgres:postgres /var/run/postgresql
I, [2024-02-01T05:20:16.752138 #1]  INFO -- :
I, [2024-02-01T05:20:16.752728 #1]  INFO -- : > /root/upgrade_postgres
I, [2024-02-01T05:20:16.759412 #1]  INFO -- :
I, [2024-02-01T05:20:16.759890 #1]  INFO -- : > rm /root/upgrade_postgres
I, [2024-02-01T05:20:16.762107 #1]  INFO -- :
I, [2024-02-01T05:20:16.764889 #1]  INFO -- : Replacing data_directory = '/var/lib/postgresql/13/main' with data_directory = '/shared/postgres_data' in /etc/postgresql/13/main/postgresql.conf
I, [2024-02-01T05:20:16.765803 #1]  INFO -- : Replacing (?-mix:#?listen_addresses *=.*) with listen_addresses = '*' in /etc/postgresql/13/main/postgresql.conf
I, [2024-02-01T05:20:16.766486 #1]  INFO -- : Replacing (?-mix:#?synchronous_commit *=.*) with synchronous_commit = $db_synchronous_commit in /etc/postgresql/13/main/postgresql.conf
I, [2024-02-01T05:20:16.767571 #1]  INFO -- : Replacing (?-mix:#?shared_buffers *=.*) with shared_buffers = $db_shared_buffers in /etc/postgresql/13/main/postgresql.conf
I, [2024-02-01T05:20:16.768666 #1]  INFO -- : Replacing (?-mix:#?work_mem *=.*) with work_mem = $db_work_mem in /etc/postgresql/13/main/postgresql.conf
I, [2024-02-01T05:20:16.769702 #1]  INFO -- : Replacing (?-mix:#?default_text_search_config *=.*) with default_text_search_config = '$db_default_text_search_config' in /etc/postgresql/13/main/postgresql.conf
I, [2024-02-01T05:20:16.770784 #1]  INFO -- : > install -d -m 0755 -o postgres -g postgres /shared/postgres_backup
I, [2024-02-01T05:20:16.777960 #1]  INFO -- :
I, [2024-02-01T05:20:16.778711 #1]  INFO -- : Replacing (?-mix:#?checkpoint_segments *=.*) with checkpoint_segments = $db_checkpoint_segments in /etc/postgresql/13/main/postgresql.conf
I, [2024-02-01T05:20:16.779400 #1]  INFO -- : Replacing (?-mix:#?logging_collector *=.*) with logging_collector = $db_logging_collector in /etc/postgresql/13/main/postgresql.conf
I, [2024-02-01T05:20:16.780719 #1]  INFO -- : Replacing (?-mix:#?log_min_duration_statement *=.*) with log_min_duration_statement = $db_log_min_duration_statement in /etc/postgresql/13/main/postgresql.conf
I, [2024-02-01T05:20:16.782507 #1]  INFO -- : Replacing (?-mix:^#local +replication +postgres +peer$) with local replication postgres  peer in /etc/postgresql/13/main/pg_hba.conf
I, [2024-02-01T05:20:16.783245 #1]  INFO -- : Replacing (?-mix:^host.*all.*all.*127.*$) with host all all 0.0.0.0/0 md5 in /etc/postgresql/13/main/pg_hba.conf
I, [2024-02-01T05:20:16.783845 #1]  INFO -- : Replacing (?-mix:^host.*all.*all.*::1\/128.*$) with host all all ::/0 md5 in /etc/postgresql/13/main/pg_hba.conf
I, [2024-02-01T05:20:16.784749 #1]  INFO -- : > HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main
I, [2024-02-01T05:20:16.786361 #1]  INFO -- : > sleep 5
2024-02-01 05:20:16.885 UTC [41] LOG:  starting PostgreSQL 13.13 (Debian 13.13-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
2024-02-01 05:20:16.887 UTC [41] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2024-02-01 05:20:16.887 UTC [41] LOG:  listening on IPv6 address "::", port 5432
2024-02-01 05:20:16.893 UTC [41] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2024-02-01 05:20:16.902 UTC [44] LOG:  database system was shut down at 2024-02-01 05:20:02 UTC
2024-02-01 05:20:16.911 UTC [41] LOG:  database system is ready to accept connections
I, [2024-02-01T05:20:21.790669 #1]  INFO -- :
I, [2024-02-01T05:20:21.790891 #1]  INFO -- : > su postgres -c 'createdb discourse' || true
2024-02-01 05:20:21.897 UTC [54] postgres@postgres ERROR:  database "discourse" already exists
2024-02-01 05:20:21.897 UTC [54] postgres@postgres STATEMENT:  CREATE DATABASE discourse;
createdb: error: database creation failed: ERROR:  database "discourse" already exists
I, [2024-02-01T05:20:21.900341 #1]  INFO -- :
I, [2024-02-01T05:20:21.900879 #1]  INFO -- : > su postgres -c 'psql discourse -c "create user discourse;"' || true
2024-02-01 05:20:21.987 UTC [58] postgres@discourse ERROR:  role "discourse" already exists
2024-02-01 05:20:21.987 UTC [58] postgres@discourse STATEMENT:  create user discourse;
ERROR:  role "discourse" already exists
I, [2024-02-01T05:20:21.990907 #1]  INFO -- :
I, [2024-02-01T05:20:21.991456 #1]  INFO -- : > su postgres -c 'psql discourse -c "grant all privileges on database discourse to discourse;"' || true
I, [2024-02-01T05:20:22.048297 #1]  INFO -- : GRANT

I, [2024-02-01T05:20:22.048988 #1]  INFO -- : > su postgres -c 'psql discourse -c "alter schema public owner to discourse;"'
I, [2024-02-01T05:20:22.106460 #1]  INFO -- : ALTER SCHEMA

I, [2024-02-01T05:20:22.107187 #1]  INFO -- : > su postgres -c 'psql template1 -c "create extension if not exists hstore;"'
NOTICE:  extension "hstore" already exists, skipping
I, [2024-02-01T05:20:22.183367 #1]  INFO -- : CREATE EXTENSION

I, [2024-02-01T05:20:22.183889 #1]  INFO -- : > su postgres -c 'psql template1 -c "create extension if not exists pg_trgm;"'
NOTICE:  extension "pg_trgm" already exists, skipping
I, [2024-02-01T05:20:22.240624 #1]  INFO -- : CREATE EXTENSION

I, [2024-02-01T05:20:22.241181 #1]  INFO -- : > su postgres -c 'psql template1 -c "create extension if not exists vector;"'
NOTICE:  extension "vector" already exists, skipping
I, [2024-02-01T05:20:22.297658 #1]  INFO -- : CREATE EXTENSION

I, [2024-02-01T05:20:22.298255 #1]  INFO -- : > su postgres -c 'psql discourse -c "create extension if not exists hstore;"'
NOTICE:  extension "hstore" already exists, skipping
I, [2024-02-01T05:20:22.356472 #1]  INFO -- : CREATE EXTENSION

I, [2024-02-01T05:20:22.356991 #1]  INFO -- : > su postgres -c 'psql discourse -c "create extension if not exists pg_trgm;"'
NOTICE:  extension "pg_trgm" already exists, skipping
I, [2024-02-01T05:20:22.414056 #1]  INFO -- : CREATE EXTENSION

I, [2024-02-01T05:20:22.414618 #1]  INFO -- : > su postgres -c 'psql discourse -c "create extension if not exists vector;"'
NOTICE:  extension "vector" already exists, skipping
I, [2024-02-01T05:20:22.470035 #1]  INFO -- : CREATE EXTENSION

I, [2024-02-01T05:20:22.470658 #1]  INFO -- : > sudo -u postgres psql discourse
I, [2024-02-01T05:20:22.475733 #1]  INFO -- : update pg_database set encoding = pg_char_to_encoding('UTF8') where datname = 'discourse' AND encoding = pg_char_to_encoding('SQL_ASCII');

I, [2024-02-01T05:20:22.567511 #1]  INFO -- : File > /var/lib/postgresql/take-database-backup  chmod: +x  chown: postgres:postgres
I, [2024-02-01T05:20:22.572125 #1]  INFO -- : File > /var/spool/cron/crontabs/postgres  chmod:   chown:
I, [2024-02-01T05:20:22.572526 #1]  INFO -- : > echo postgres installed!
I, [2024-02-01T05:20:22.575793 #1]  INFO -- : postgres installed!

I, [2024-02-01T05:20:22.580912 #1]  INFO -- : File > /etc/service/redis/run  chmod: +x  chown:
I, [2024-02-01T05:20:22.585785 #1]  INFO -- : File > /etc/service/redis/log/run  chmod: +x  chown:
I, [2024-02-01T05:20:22.590830 #1]  INFO -- : File > /etc/runit/3.d/10-redis  chmod: +x  chown:
I, [2024-02-01T05:20:22.593112 #1]  INFO -- : Replacing daemonize yes with  in /etc/redis/redis.conf
I, [2024-02-01T05:20:22.594012 #1]  INFO -- : Replacing (?-mix:^pidfile.*$) with  in /etc/redis/redis.conf
I, [2024-02-01T05:20:22.594826 #1]  INFO -- : > install -d -m 0755 -o redis -g redis /shared/redis_data
I, [2024-02-01T05:20:22.597926 #1]  INFO -- :
I, [2024-02-01T05:20:22.598727 #1]  INFO -- : Replacing (?-mix:^logfile.*$) with logfile "" in /etc/redis/redis.conf
I, [2024-02-01T05:20:22.599588 #1]  INFO -- : Replacing (?-mix:^bind .*$) with  in /etc/redis/redis.conf
I, [2024-02-01T05:20:22.601300 #1]  INFO -- : Replacing (?-mix:^dir .*$) with dir /shared/redis_data in /etc/redis/redis.conf
I, [2024-02-01T05:20:22.603062 #1]  INFO -- : Replacing (?-mix:^protected-mode yes) with protected-mode no in /etc/redis/redis.conf
I, [2024-02-01T05:20:22.604759 #1]  INFO -- : Replacing # io-threads 4 with io-threads $redis_io_threads in /etc/redis/redis.conf
I, [2024-02-01T05:20:22.606503 #1]  INFO -- : > echo redis installed
I, [2024-02-01T05:20:22.609035 #1]  INFO -- : redis installed

I, [2024-02-01T05:20:22.609661 #1]  INFO -- : > cat /etc/redis/redis.conf | grep logfile
I, [2024-02-01T05:20:22.614668 #1]  INFO -- : logfile ""

I, [2024-02-01T05:20:22.615160 #1]  INFO -- : > exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf
I, [2024-02-01T05:20:22.616826 #1]  INFO -- : > sleep 10
110:C 01 Feb 2024 05:20:22.631 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
110:C 01 Feb 2024 05:20:22.631 # Redis version=7.0.7, bits=64, commit=00000000, modified=0, pid=110, just started
110:C 01 Feb 2024 05:20:22.631 # Configuration loaded
110:M 01 Feb 2024 05:20:22.632 * monotonic clock: POSIX clock_gettime
110:M 01 Feb 2024 05:20:22.633 * Running mode=standalone, port=6379.
110:M 01 Feb 2024 05:20:22.633 # Server initialized
110:M 01 Feb 2024 05:20:22.634 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
110:M 01 Feb 2024 05:20:22.634 # WARNING Your system is configured to use the 'xen' clocksource which might lead to degraded performance. Check the result of the [slow-clocksource] system check: run 'redis-server --check-system' to check if the system's clocksource isn't degrading performance.
110:M 01 Feb 2024 05:20:22.635 * Loading RDB produced by version 7.0.7
110:M 01 Feb 2024 05:20:22.635 * RDB age 28 seconds
110:M 01 Feb 2024 05:20:22.635 * RDB memory usage when created 12.85 Mb
110:M 01 Feb 2024 05:20:22.670 * Done loading RDB, keys loaded: 11479, keys expired: 6.
110:M 01 Feb 2024 05:20:22.670 * DB loaded from disk: 0.035 seconds
110:M 01 Feb 2024 05:20:22.670 * Ready to accept connections
I, [2024-02-01T05:20:32.621081 #1]  INFO -- :
I, [2024-02-01T05:20:32.621808 #1]  INFO -- : > thpoff echo "thpoff is installed!"
I, [2024-02-01T05:20:32.626389 #1]  INFO -- : thpoff is installed!

I, [2024-02-01T05:20:32.626901 #1]  INFO -- : > /usr/local/bin/ruby -e 'if ENV["DISCOURSE_SMTP_ADDRESS"] == "smtp.example.com"; puts "Aborting! Mail is not configured!"; exit 1; end'
I, [2024-02-01T05:20:32.707287 #1]  INFO -- :
I, [2024-02-01T05:20:32.707923 #1]  INFO -- : > /usr/local/bin/ruby -e 'if ENV["DISCOURSE_HOSTNAME"] == "discourse.example.com"; puts "Aborting! Domain is not configured!"; exit 1; end'
I, [2024-02-01T05:20:32.788125 #1]  INFO -- :
I, [2024-02-01T05:20:32.788819 #1]  INFO -- : > /usr/local/bin/ruby -e 'if (ENV["DISCOURSE_CDN_URL"] || "")[0..1] == "//"; puts "Aborting! CDN must have a protocol specified. Once fixed you should rebake your posts now to correct all posts."; exit 1; end'
I, [2024-02-01T05:20:32.868839 #1]  INFO -- :
I, [2024-02-01T05:20:32.869502 #1]  INFO -- : > rm -f /etc/cron.d/anacron
I, [2024-02-01T05:20:32.872477 #1]  INFO -- :
I, [2024-02-01T05:20:32.875792 #1]  INFO -- : File > /etc/cron.d/anacron  chmod:   chown:
I, [2024-02-01T05:20:32.882091 #1]  INFO -- : File > /etc/runit/1.d/copy-env  chmod: +x  chown:
I, [2024-02-01T05:20:32.887278 #1]  INFO -- : File > /etc/service/unicorn/run  chmod: +x  chown:
I, [2024-02-01T05:20:32.892443 #1]  INFO -- : File > /etc/service/nginx/run  chmod: +x  chown:
I, [2024-02-01T05:20:32.897678 #1]  INFO -- : File > /etc/runit/3.d/01-nginx  chmod: +x  chown:
I, [2024-02-01T05:20:32.902952 #1]  INFO -- : File > /etc/runit/3.d/02-unicorn  chmod: +x  chown:
I, [2024-02-01T05:20:32.903394 #1]  INFO -- : Replacing # postgres with sv start postgres || exit 1 in /etc/service/unicorn/run
I, [2024-02-01T05:20:32.903939 #1]  INFO -- : > exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf
I, [2024-02-01T05:20:32.906369 #1]  INFO -- : > cd /var/www/discourse && sudo -H -E -u discourse git reset --hard
137:C 01 Feb 2024 05:20:32.915 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
137:C 01 Feb 2024 05:20:32.918 # Redis version=7.0.7, bits=64, commit=00000000, modified=0, pid=137, just started
137:C 01 Feb 2024 05:20:32.918 # Configuration loaded
137:M 01 Feb 2024 05:20:32.919 * monotonic clock: POSIX clock_gettime
137:M 01 Feb 2024 05:20:32.920 # Warning: Could not create server TCP listening socket *:6379: bind: Address already in use
137:M 01 Feb 2024 05:20:32.920 # Failed listening on port 6379 (TCP), aborting.
Updating files: 100% (33715/33715), done.
I, [2024-02-01T05:20:39.803997 #1]  INFO -- : HEAD is now at 6ab1a19e9 DEV: Convert min_trust_level_to_allow_invite to groups (#24893)

I, [2024-02-01T05:20:39.804645 #1]  INFO -- : > cd /var/www/discourse && sudo -H -E -u discourse git clean -f
I, [2024-02-01T05:20:40.031462 #1]  INFO -- :
I, [2024-02-01T05:20:40.032125 #1]  INFO -- : > cd /var/www/discourse && sudo -H -E -u discourse bash -c '
  set -o errexit
  if [ $(git rev-parse --is-shallow-repository) == "true" ]; then
      git remote set-branches --add origin main
      git remote set-branches origin stable
      git fetch --depth 1 origin stable
  else
      git fetch --tags --prune-tags --prune --force origin
  fi
'
From https://github.com/discourse/discourse
 - [deleted]             (none)     -> origin/0-drop-code
 - [deleted]             (none)     -> origin/bulk-button-class
 - [deleted]             (none)     -> origin/dependabot/bundler/excon-0.108.0
 - [deleted]             (none)     -> origin/dependabot/bundler/hashdiff-1.1.0
 - [deleted]             (none)     -> origin/dependabot/bundler/parallel-1.24.0
 - [deleted]             (none)     -> origin/dependabot/bundler/rubocop-discourse-3.6.0
 - [deleted]             (none)     -> origin/dependabot/npm_and_yarn/app/assets/javascripts/uppy/drop-target-2.0.2
 - [deleted]             (none)     -> origin/dev/add-new-chat-metrics
 - [deleted]             (none)     -> origin/dev/allow-chat-mentions-to-have-several-notifications
 - [deleted]             (none)     -> origin/dev/convert-min-trust-level-to-create-tag-to-group
 - [deleted]             (none)     -> origin/dev/convert-min-trust-level-to-ignore-to-groups
 - [deleted]             (none)     -> origin/dev/rethink-mass-mentions-polymorphic
 - [deleted]             (none)     -> origin/feature/migrations-schema-generator
 - [deleted]             (none)     -> origin/fix-join-filter-upload
 - [deleted]             (none)     -> origin/fix-modal-flash-minor
 - [deleted]             (none)     -> origin/global-notice-reactive
 - [deleted]             (none)     -> origin/labeler
 - [deleted]             (none)     -> origin/low-ram-help
 - [deleted]             (none)     -> origin/multi-invite
 - [deleted]             (none)     -> origin/table-builder-modal-modifier
 - [deleted]             (none)     -> origin/wip_categories
 - [deleted]             (none)     -> origin/wizard-split
   6ab1a19e9..969ab0fd6  main                    -> origin/main
 * [new branch]          0-category-reorder      -> origin/0-category-reorder
 * [new branch]          0-ember-bump            -> origin/0-ember-bump
 * [new branch]          0-event-helper          -> origin/0-event-helper
 * [new branch]          0-extend-native         -> origin/0-extend-native
 * [new branch]          0-rev                   -> origin/0-rev
 * [new branch]          add-api-for-forced-auth -> origin/add-api-for-forced-auth
 * [new branch]          amend-visit-copy        -> origin/amend-visit-copy
 * [new branch]          asciinema-iframe        -> origin/asciinema-iframe
 * [new branch]          chat/original-message-link -> origin/chat/original-message-link
 * [new branch]          convert-header-to-glimmer -> origin/convert-header-to-glimmer
 * [new branch]          decorator-transforms    -> origin/decorator-transforms
 * [new branch]          dependabot/bundler/omniauth-2.0.0 -> origin/dependabot/bundler/omniauth-2.0.0
 * [new branch]          dependabot/bundler/parallel_tests-4.4.0 -> origin/dependabot/bundler/parallel_tests-4.4.0
 * [new branch]          dev-convert-topic-map-expanded-widget-to-component -> origin/dev-convert-topic-map-expanded-widget-to-component
 * [new branch]          dev/check-whether-online-user-ids-may-be-memoized-between-job-runs -> origin/dev/check-whether-online-user-ids-may-be-memoized-between-job-runs
 * [new branch]          dev/drop-the-user-id-column-from-chat-mentions -> origin/dev/drop-the-user-id-column-from-chat-mentions
 + e0c0191bb...feb00e04c dev/dry-up-user-status-serializers -> origin/dev/dry-up-user-status-serializers  (forced update)
 * [new branch]          dev/experiment-theme-settings-free-ui -> origin/dev/experiment-theme-settings-free-ui
 * [new branch]          dev/make-admins-tl4-in-tests -> origin/dev/make-admins-tl4-in-tests
 * [new branch]          dev/promote-plugin-post-deploy-migrations -> origin/dev/promote-plugin-post-deploy-migrations
 * [new branch]          dev/proof-of-concept-rewriting-mentions-notification-logic -> origin/dev/proof-of-concept-rewriting-mentions-notification-logic
 * [new branch]          dev_category_order_database -> origin/dev_category_order_database
 * [new branch]          dev_update_category_position -> origin/dev_update_category_position
 * [new branch]          ellaestigoy-patch-2     -> origin/ellaestigoy-patch-2
 * [new branch]          fast-edit-expand-regex  -> origin/fast-edit-expand-regex
 * [new branch]          feature/add-new-illegal-flag-reason -> origin/feature/add-new-illegal-flag-reason
 * [new branch]          feature/custom-date-range -> origin/feature/custom-date-range
 * [new branch]          feature/paste-upload-on-first-upload-input-templates -> origin/feature/paste-upload-on-first-upload-input-templates
 * [new branch]          feature_async_load_colors -> origin/feature_async_load_colors
 * [new branch]          fix-email-category-badge -> origin/fix-email-category-badge
 * [new branch]          fix_category_chooser    -> origin/fix_category_chooser
 * [new branch]          fix_in_any_groups       -> origin/fix_in_any_groups
 * [new branch]          fix_preload_sidebar_categories -> origin/fix_preload_sidebar_categories
 * [new branch]          fix_reviewable_translation -> origin/fix_reviewable_translation
   7f2bdca65..b851b879c  generic-import-latest   -> origin/generic-import-latest
 * [new branch]          hijacks                 -> origin/hijacks
 * [new branch]          issue/avatar-upload-permission-serialize -> origin/issue/avatar-upload-permission-serialize
 * [new branch]          issue/run-admin-sidebar-init-after-logs-notice -> origin/issue/run-admin-sidebar-init-after-logs-notice
 + 90a3535b7...e6e64cfd5 migrations-convert      -> origin/migrations-convert  (forced update)
 + 0fa691496...b0f74a188 plugin-extra-js         -> origin/plugin-extra-js  (forced update)
 * [new branch]          promote-migration-tweak -> origin/promote-migration-tweak
 * [new branch]          revert-25009-dev/convert-min-trust-to-allow-self-wiki-to-groups -> origin/revert-25009-dev/convert-min-trust-to-allow-self-wiki-to-groups
 * [new branch]          revert-bulk-topic-actions -> origin/revert-bulk-topic-actions
 * [new branch]          sidebar-above-outlet    -> origin/sidebar-above-outlet
 * [new branch]          site-logo-height        -> origin/site-logo-height
 * [new branch]          soft-load-topic-results -> origin/soft-load-topic-results
   d7f7f79b6..968b1a57e  stable                  -> origin/stable
 * [new branch]          support-reset-bump-date-post-id -> origin/support-reset-bump-date-post-id
   6ab1a19e9..969ab0fd6  tests-passed            -> origin/tests-passed
 * [new branch]          unhide_lazy_load_categories -> origin/unhide_lazy_load_categories
 * [new branch]          universal-nonce         -> origin/universal-nonce
 * [new branch]          unread-mobile-indicator -> origin/unread-mobile-indicator
 * [new branch]          virtual-scroll-list     -> origin/virtual-scroll-list
 t [tag update]          beta                    -> beta
 t [tag update]          latest-release          -> latest-release
 * [new tag]             v3.1.4                  -> v3.1.4
 * [new tag]             v3.1.5                  -> v3.1.5
 * [new tag]             v3.2.0                  -> v3.2.0
 * [new tag]             v3.2.0.beta4            -> v3.2.0.beta4
 * [new tag]             v3.2.0.beta5            -> v3.2.0.beta5
I, [2024-02-01T05:20:40.935200 #1]  INFO -- :
I, [2024-02-01T05:20:40.935686 #1]  INFO -- : > cd /var/www/discourse && sudo -H -E -u discourse bash -c '
  set -o errexit
  if [[ $(git symbolic-ref --short HEAD) == stable ]] ; then
      git pull
  else
      git -c advice.detachedHead=false checkout stable
  fi
'
Switched to a new branch 'stable'
I, [2024-02-01T05:20:44.064865 #1]  INFO -- : Branch 'stable' set up to track remote branch 'stable' from 'origin'.

I, [2024-02-01T05:20:44.065354 #1]  INFO -- : > cd /var/www/discourse && sudo -H -E -u discourse git config user.discourse-version stable
I, [2024-02-01T05:20:44.077044 #1]  INFO -- :
I, [2024-02-01T05:20:44.077436 #1]  INFO -- : > cd /var/www/discourse && mkdir -p tmp
I, [2024-02-01T05:20:44.085161 #1]  INFO -- :
I, [2024-02-01T05:20:44.085555 #1]  INFO -- : > cd /var/www/discourse && chown discourse:www-data tmp
I, [2024-02-01T05:20:44.090473 #1]  INFO -- :
I, [2024-02-01T05:20:44.090945 #1]  INFO -- : > cd /var/www/discourse && mkdir -p tmp/pids
I, [2024-02-01T05:20:44.094272 #1]  INFO -- :
I, [2024-02-01T05:20:44.094646 #1]  INFO -- : > cd /var/www/discourse && mkdir -p tmp/sockets
I, [2024-02-01T05:20:44.097962 #1]  INFO -- :
I, [2024-02-01T05:20:44.098333 #1]  INFO -- : > cd /var/www/discourse && touch tmp/.gitkeep
I, [2024-02-01T05:20:44.102920 #1]  INFO -- :
I, [2024-02-01T05:20:44.103294 #1]  INFO -- : > cd /var/www/discourse && mkdir -p                    /shared/log/rails
I, [2024-02-01T05:20:44.106489 #1]  INFO -- :
I, [2024-02-01T05:20:44.106964 #1]  INFO -- : > cd /var/www/discourse && bash -c "touch -a           /shared/log/rails/{production,production_errors,unicorn.stdout,unicorn.stderr,sidekiq}.log"
I, [2024-02-01T05:20:44.112554 #1]  INFO -- :
I, [2024-02-01T05:20:44.112953 #1]  INFO -- : > cd /var/www/discourse && bash -c "ln    -s           /shared/log/rails/{production,production_errors,unicorn.stdout,unicorn.stderr,sidekiq}.log /var/www/discourse/log"
I, [2024-02-01T05:20:44.119868 #1]  INFO -- :
I, [2024-02-01T05:20:44.120247 #1]  INFO -- : > cd /var/www/discourse && bash -c "mkdir -p           /shared/{uploads,backups}"
I, [2024-02-01T05:20:44.125723 #1]  INFO -- :
I, [2024-02-01T05:20:44.126103 #1]  INFO -- : > cd /var/www/discourse && bash -c "ln    -s           /shared/{uploads,backups} /var/www/discourse/public"
I, [2024-02-01T05:20:44.130710 #1]  INFO -- :
I, [2024-02-01T05:20:44.131085 #1]  INFO -- : > cd /var/www/discourse && bash -c "mkdir -p           /shared/tmp/{backups,restores}"
I, [2024-02-01T05:20:44.136397 #1]  INFO -- :
I, [2024-02-01T05:20:44.136900 #1]  INFO -- : > cd /var/www/discourse && bash -c "ln    -s           /shared/tmp/{backups,restores} /var/www/discourse/tmp"
I, [2024-02-01T05:20:44.141456 #1]  INFO -- :
I, [2024-02-01T05:20:44.141832 #1]  INFO -- : > cd /var/www/discourse && chown -R discourse:www-data /shared/log/rails /shared/uploads /shared/backups /shared/tmp
I, [2024-02-01T05:20:44.267143 #1]  INFO -- :
I, [2024-02-01T05:20:44.267732 #1]  INFO -- : > cd /var/www/discourse && [ ! -d public/plugins ] || find public/plugins/ -maxdepth 1 -xtype l -delete
I, [2024-02-01T05:20:44.270260 #1]  INFO -- :
I, [2024-02-01T05:20:44.271569 #1]  INFO -- : Replacing # redis with sv start redis || exit 1 in /etc/service/unicorn/run
I, [2024-02-01T05:20:44.273289 #1]  INFO -- : > cd /var/www/discourse/plugins && git clone https://github.com/discourse/docker_manager.git
Cloning into 'docker_manager'...
I, [2024-02-01T05:20:45.047892 #1]  INFO -- :
I, [2024-02-01T05:20:45.048485 #1]  INFO -- : > cd /var/www/discourse/plugins && git clone https://github.com/discourse/discourse-voting.git
Cloning into 'discourse-voting'...
I, [2024-02-01T05:20:45.349517 #1]  INFO -- :
I, [2024-02-01T05:20:45.350128 #1]  INFO -- : > cd /var/www/discourse/plugins && git clone https://github.com/discourse/discourse-solved.git
Cloning into 'discourse-solved'...
I, [2024-02-01T05:20:45.696361 #1]  INFO -- :
I, [2024-02-01T05:20:45.696958 #1]  INFO -- : > cd /var/www/discourse/plugins && git clone https://github.com/discourse/discourse-policy.git
Cloning into 'discourse-policy'...
I, [2024-02-01T05:20:45.985549 #1]  INFO -- :
I, [2024-02-01T05:20:45.986116 #1]  INFO -- : > cd /var/www/discourse/plugins && git clone https://github.com/discourse/discourse-adplugin.git
Cloning into 'discourse-adplugin'...
I, [2024-02-01T05:20:46.333531 #1]  INFO -- :
I, [2024-02-01T05:20:46.334121 #1]  INFO -- : > cd /var/www/discourse/plugins && git clone https://github.com/discourse/discourse-data-explorer.git
Cloning into 'discourse-data-explorer'...
I, [2024-02-01T05:20:46.684891 #1]  INFO -- :
I, [2024-02-01T05:20:46.685512 #1]  INFO -- : > cd /var/www/discourse/plugins && git clone https://github.com/communiteq/discourse-suppress-category-from-latest.git
Cloning into 'discourse-suppress-category-from-latest'...
I, [2024-02-01T05:20:46.860067 #1]  INFO -- :
I, [2024-02-01T05:20:46.860646 #1]  INFO -- : > cd /var/www/discourse/plugins && git clone https://github.com/discourse/discourse-bbcode-color
Cloning into 'discourse-bbcode-color'...
I, [2024-02-01T05:20:47.089640 #1]  INFO -- :
I, [2024-02-01T05:20:47.090378 #1]  INFO -- : > cp /var/www/discourse/config/nginx.sample.conf /etc/nginx/conf.d/discourse.conf
I, [2024-02-01T05:20:47.099746 #1]  INFO -- :
I, [2024-02-01T05:20:47.100326 #1]  INFO -- : > rm /etc/nginx/sites-enabled/default
I, [2024-02-01T05:20:47.103575 #1]  INFO -- :
I, [2024-02-01T05:20:47.104073 #1]  INFO -- : > mkdir -p /var/nginx/cache
I, [2024-02-01T05:20:47.107202 #1]  INFO -- :
I, [2024-02-01T05:20:47.108186 #1]  INFO -- : Replacing pid /run/nginx.pid; with daemon off; in /etc/nginx/nginx.conf
I, [2024-02-01T05:20:47.109528 #1]  INFO -- : Replacing (?m-ix:upstream[^\}]+\}) with upstream discourse { server 127.0.0.1:3000; } in /etc/nginx/conf.d/discourse.conf
I, [2024-02-01T05:20:47.110130 #1]  INFO -- : Replacing (?-mix:server_name.+$) with server_name _ ; in /etc/nginx/conf.d/discourse.conf
I, [2024-02-01T05:20:47.110992 #1]  INFO -- : Replacing (?-mix:client_max_body_size.+$) with client_max_body_size $upload_size ; in /etc/nginx/conf.d/discourse.conf
I, [2024-02-01T05:20:47.111810 #1]  INFO -- : Replacing (?-mix:worker_connections.+$) with worker_connections $nginx_worker_connections ; in /etc/nginx/nginx.conf
I, [2024-02-01T05:20:47.112296 #1]  INFO -- : > echo "done configuring web"
I, [2024-02-01T05:20:47.114305 #1]  INFO -- : done configuring web

I, [2024-02-01T05:20:47.114863 #1]  INFO -- : > cd /var/www/discourse && gem install bundler --conservative -v $(awk '/BUNDLED WITH/ { getline; gsub(/ /,""); print $0 }' Gemfile.lock)
I, [2024-02-01T05:20:48.019548 #1]  INFO -- : Successfully installed bundler-2.5.3
1 gem installed

I, [2024-02-01T05:20:48.020208 #1]  INFO -- : > cd /var/www/discourse && find /var/www/discourse ! -user discourse -exec chown discourse {} \+
I, [2024-02-01T05:20:56.712671 #1]  INFO -- :
I, [2024-02-01T05:20:56.723363 #1]  INFO -- : > cd /var/www/discourse && if [ "stable" != "tests-passed" ]; then
  rm -rf app/assets/javascripts/node_modules
fi
I, [2024-02-01T05:20:59.502523 #1]  INFO -- :
I, [2024-02-01T05:20:59.503242 #1]  INFO -- : > cd /var/www/discourse && su discourse -c 'yarn install --frozen-lockfile && yarn cache clean'
^[[D^[[D^[[Dwarning "@discourse/lint-configs > eslint-plugin-ember > ember-eslint-parser@0.2.5" has unmet peer dependency "@typescript-eslint/parser@^6.15.0".
warning "@discourse/lint-configs > eslint-plugin-ember > ember-eslint-parser@0.2.5" has incorrect peer dependency "typescript@^5.3.3".
warning " > @glint/environment-ember-loose@1.3.0" has unmet peer dependency "@glimmer/component@^1.1.2".
warning Resolution field "unset-value@2.0.1" is incompatible with requested version "unset-value@^1.0.0"
warning Pattern ["wrap-ansi@^7.0.0"] is trying to unpack in the same destination "/home/discourse/.cache/yarn/v6/npm-wrap-ansi-cjs-7.0.0-67e145cff510a6a6984bdf1152911d69d2eb9e43-integrity/node_modules/wrap-ansi-cjs" as pattern ["wrap-ansi-cjs@npm:wrap-ansi@^7.0.0"]. This could result in non-deterministic behavior, skipping.
warning " > discourse-markdown-it@1.0.0" has unmet peer dependency "xss@*".
warning "workspace-aggregator-ca265ca6-43f3-4f9e-bc1f-e2610715a50c > discourse > @uppy/aws-s3@3.0.6" has incorrect peer dependency "@uppy/core@^3.1.2".
warning "workspace-aggregator-ca265ca6-43f3-4f9e-bc1f-e2610715a50c > discourse > @uppy/aws-s3-multipart@3.1.3" has incorrect peer dependency "@uppy/core@^3.1.2".
warning "workspace-aggregator-ca265ca6-43f3-4f9e-bc1f-e2610715a50c > discourse > @uppy/xhr-upload@3.1.1" has incorrect peer dependency "@uppy/core@^3.1.2".
warning "workspace-aggregator-ca265ca6-43f3-4f9e-bc1f-e2610715a50c > discourse > ember-router-service-refresh-polyfill@1.1.0" has incorrect peer dependency "ember-source@^3.4 || 4".
warning "workspace-aggregator-ca265ca6-43f3-4f9e-bc1f-e2610715a50c > discourse-plugins > ember-this-fallback@0.4.0" has unmet peer dependency "ember-source@^3.28.11 || ^4.0.0".
warning "workspace-aggregator-ca265ca6-43f3-4f9e-bc1f-e2610715a50c > admin > ember-source > router_js@8.0.3" has unmet peer dependency "rsvp@^4.8.5".
warning "workspace-aggregator-ca265ca6-43f3-4f9e-bc1f-e2610715a50c > discourse > @uppy/aws-s3 > @uppy/xhr-upload@3.3.0" has incorrect peer dependency "@uppy/core@^3.2.1".
I, [2024-02-01T05:23:59.990567 #1]  INFO -- : yarn install v1.22.19
[1/5] Validating package.json...
[2/5] Resolving packages...
[3/5] Fetching packages...
[4/5] Linking dependencies...
[5/5] Building fresh packages...
$ yarn --cwd app/assets/javascripts $(node -e 'const argv = JSON.parse(process.env.npm_config_argv).original; const passthrough = [`--frozen-lockfile`, `-s`].filter(arg => argv.includes(arg)); console.log(passthrough.join(` `));')
yarn install v1.22.19
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
$ ./run-patch-package
patch-package 8.0.0
Applying patches...
@ember/test-helpers@2.9.4 ✔
babel-plugin-debug-macros@0.3.4 ✔
content-tag@1.2.2 ✔
ember-cli@5.0.0 ✔
ember-this-fallback@0.4.0 (1 deprecation-name) ✔
ember-this-fallback@0.4.0 (2 themes) ✔
ember-this-fallback@0.4.0 (3 exclude-strict-mode) ✔
virtual-dom@2.1.1 ✔
Done in 87.51s.
Done in 164.47s.
yarn cache v1.22.19
success Cleared cache.
Done in 14.71s.

I, [2024-02-01T05:23:59.999517 #1]  INFO -- : > cd /var/www/discourse && su discourse -c 'bundle config --local deployment true'
I, [2024-02-01T05:24:00.428628 #1]  INFO -- :
I, [2024-02-01T05:24:00.429260 #1]  INFO -- : > cd /var/www/discourse && su discourse -c 'bundle config --local without "development test"'
I, [2024-02-01T05:24:00.632752 #1]  INFO -- : You are replacing the current local value of without, which is currently "test:development"

I, [2024-02-01T05:24:00.633369 #1]  INFO -- : > cd /var/www/discourse && su discourse -c 'bundle install --retry 3 --jobs 4'
I, [2024-02-01T05:24:51.790059 #1]  INFO -- : Fetching gem metadata from https://rubygems.org/.........
Fetching concurrent-ruby 1.2.3
Fetching minitest 5.21.2
Fetching bigdecimal 3.1.6
Fetching cbor 0.5.9.8
Installing minitest 5.21.2
Installing bigdecimal 3.1.6 with native extensions
Installing cbor 0.5.9.8 with native extensions
Installing concurrent-ruby 1.2.3
Fetching excon 0.109.0
Installing excon 0.109.0
Fetching fastimage 2.3.0
Installing fastimage 2.3.0
Fetching highline 3.0.1
Fetching google-protobuf 3.25.2 (x86_64-linux)
Installing highline 3.0.1
Fetching image_size 3.4.0
Installing image_size 3.4.0
Installing google-protobuf 3.25.2 (x86_64-linux)
Fetching regexp_parser 2.9.0
Installing regexp_parser 2.9.0
Fetching logster 2.16.0
Fetching stackprof 0.2.26
Installing stackprof 0.2.26 with native extensions
Installing logster 2.16.0
Fetching nokogiri 1.16.0 (x86_64-linux)
Installing nokogiri 1.16.0 (x86_64-linux)
Fetching rack-protection 3.2.0
Installing rack-protection 3.2.0
Fetching bootsnap 1.17.1
Installing bootsnap 1.17.1 with native extensions
Fetching net-http 0.4.1
Installing net-http 0.4.1
Fetching sass-embedded 1.70.0 (x86_64-linux-gnu)
Installing sass-embedded 1.70.0 (x86_64-linux-gnu)
Fetching puma 6.4.2
Installing puma 6.4.2 with native extensions
Fetching rbtrace 0.5.1
Installing rbtrace 0.5.1 with native extensions
Fetching net-imap 0.4.9.1
Installing net-imap 0.4.9.1
Fetching net-smtp 0.4.0.1
Installing net-smtp 0.4.0.1
Fetching faraday-net_http 3.1.0
Installing faraday-net_http 3.1.0
Fetching activesupport 7.0.8
Installing activesupport 7.0.8
Fetching tzinfo-data 1.2023.4
Installing tzinfo-data 1.2023.4
Fetching sassc-embedded 1.70.0
Installing sassc-embedded 1.70.0
Fetching faraday 2.9.0
Installing faraday 2.9.0
Fetching activemodel 7.0.8
Installing activemodel 7.0.8
Fetching actionview 7.0.8
Installing actionview 7.0.8
Fetching activejob 7.0.8
Installing activejob 7.0.8
Fetching activerecord 7.0.8
Installing activerecord 7.0.8
Fetching actionpack 7.0.8
Installing actionpack 7.0.8
Fetching actionmailer 7.0.8
Installing actionmailer 7.0.8
Fetching railties 7.0.8
Installing railties 7.0.8
Bundle complete! 138 Gemfile dependencies, 170 gems now installed.
Gems in the groups 'development' and 'test' were not installed.
Bundled gems are installed into `./vendor/bundle`

I, [2024-02-01T05:24:51.794263 #1]  INFO -- : > cd /var/www/discourse && su discourse -c 'LOAD_PLUGINS=0 bundle exec rake plugin:pull_compatible_all'
I, [2024-02-01T05:24:57.793104 #1]  INFO -- : discourse-adplugin is already at latest compatible version
discourse-bbcode-color is already at latest compatible version
discourse-data-explorer is already at latest compatible version
discourse-policy is already at latest compatible version
discourse-solved is already at latest compatible version
discourse-suppress-category-from-latest is already at latest compatible version
discourse-voting is already at latest compatible version
docker_manager is already at latest compatible version

I, [2024-02-01T05:24:57.796053 #1]  INFO -- : > cd /var/www/discourse && su discourse -c 'bundle exec rake db:migrate'
Plugin name is 'discourse-topic-voting', but plugin directory is named 'discourse-voting'
I, [2024-02-01T05:25:07.163248 #1]  INFO -- :
I, [2024-02-01T05:25:07.166699 #1]  INFO -- : > cd /var/www/discourse && su discourse -c 'bundle exec rake themes:update assets:precompile'
Plugin name is 'discourse-topic-voting', but plugin directory is named 'discourse-voting'
Node.js heap_size_limit (487.0) is less than 2048MB. Setting --max-old-space-size=2048.
110:M 01 Feb 2024 05:25:23.049 * 100 changes in 300 seconds. Saving...
110:M 01 Feb 2024 05:25:23.053 * Background saving started by pid 3562
3562:C 01 Feb 2024 05:25:23.932 * DB saved on disk
3562:C 01 Feb 2024 05:25:23.933 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 0 MB
110:M 01 Feb 2024 05:25:23.955 * Background saving terminated with success

110:M 01 Feb 2024 06:20:24.614 * Replication backlog freed after 3600 seconds without connected replicas.

The CPU utilization shows under 10% and it gets stuck here for hours on end. I’ve tried to reboot the instance and restart the upgrade, it gets stuck at the same point again.

Ed_S · February 1, 2024, 2:02pm

There’s a chance you have not enough RAM, causing lots of swapping, which means the CPU won’t be pegged because the machine is always paging things in and out.

You can measure things from the command line:

cat /etc/lsb-release
df -h /
free
vmstat 5 5

either with the machine in a quiet state, or at the time that it’s stuck. (Connect from a second window to get a second command line. Or similar.)

RBoy · February 1, 2024, 5:59pm

Thanks, on this rebuild, it’s moving really really slowly at yarn, the CPU’s barely being used and it doesn’t look like there’s much I/O either. I’m a little lost as why it’s crawling.

ubuntu@ip-172-26-6-122:~$ cat /etc/lsb-release
df -h /
free
vmstat 5 5
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.3 LTS"
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        39G   19G   21G  47% /
               total        used        free      shared  buff/cache   available
Mem:          972524      399360       75956       14532      497208      365668
Swap:        2097148       51296     2045852
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  4  51296  75956  56504 440780  951  936  7442  1197  620 1597  2  4  2 91  0
 0  5  51296  69668  56632 444976    0    0   418   585  389  893  2  2  0 96  0
 0  4  51296  65384  57280 448368    0    0   422   599  382  906  1  1  0 97  0
 0  4  51296  74460  57564 439444    0    0   506   546  399  946  1  3  0 96  0
 0  4  51296  64588  57828 444900    0    0   545   614  407  983  2  2  0 96  0

Prior to 3.2.0, the rebuilds would generally fly along and be done in about 10-20 minutes.

Ed_S · February 1, 2024, 6:20pm

Thanks. So, no swapping, not a RAM problem. You do have constant block I/O. Could that be remote storage?? Is everything local or do you have cloud storage involved?

I notice both snapd and amazon-ssm-agent have accumulated quite a bit of CPU time, even though their percentage is zero. I don’t know what’s going on there - I don’t have either of those.

If I/O is really slow, it might that you are sharing a machine (and a disk) with some very hungry neighbour. Perhaps you can ask your hosting provider to move you, or check?

Sorry, no good answers at present.

RBoy · February 1, 2024, 6:36pm

Running on EC2/Lightsail. I thought it might an instance issue so I took a snapshot and a spun up a new instance in a new region all together and it’s still getting stuck at

3536:C 01 Feb 2024 18:24:10.670 * DB saved on disk
3536:C 01 Feb 2024 18:24:10.672 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB
110:M 01 Feb 2024 18:24:10.761 * Background saving terminated with success
Plugin name is 'discourse-topic-voting', but plugin directory is named 'discourse-voting'
Node.js heap_size_limit (487.0) is less than 2048MB. Setting --max-old-space-size=2048.
110:M 01 Feb 2024 18:29:11.092 * 100 changes in 300 seconds. Saving...
110:M 01 Feb 2024 18:29:11.509 * Background saving started by pid 3721
3721:C 01 Feb 2024 18:29:59.839 * DB saved on disk
3721:C 01 Feb 2024 18:29:59.840 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB
110:M 01 Feb 2024 18:29:59.876 * Background saving terminated with success

This is the point where it just “hangs”:

ubuntu@ip-172-26-6-122:~$ cat /etc/lsb-release
df -h /
free
vmstat 5 5
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.3 LTS"
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        39G   19G   21G  47% /
               total        used        free      shared  buff/cache   available
Mem:          972524      674416      155656       15932      142452      143064
Swap:        2097148      201824     1895324
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  5 201824 155408   3220 140844  870  860  6959  1176  601 1627  3  4  2 91  0
 1  0 201824 154824   5588 169332    3    0  5503    11  365 1708 39  8  0 53  0
 3  0 202336  67104   4896 159076   48  150   986   194  321 3046 96  4  0  0  0
 0  6 209760  74732    296 114488  283 1738  8385  1934  499 2265 46  5  0 49  0
 0  4 212576  76968    200 115316    0  569  7678   652  387  524  1  1  0 97  0

and htop shows it’s stuck spinning CPU cycles on node, ember and snapd, as it will suddenly burst and take a 100% CPU and then sit at around <5% for a while and then again burst and so on

RBoy · February 1, 2024, 6:44pm

I suspect it’s an issue with the discourse upgrade itself. Is there a way to “reset” the discourse upgrade and have it restart over clean?

RBoy · February 1, 2024, 8:16pm

It appears to me that something about the discourse build is broken when it’s trying to precompile the assets. It’s just plain hung here with no I/O or CPU consumption.

I can’t find anything on the forum search on how to reset a rebuild from the console and am out of ideas here. @JammyDodger looking for some help from the experts who may throw light on why the rebuild has hung and how to reset it.

JammyDodger · February 1, 2024, 8:35pm

I think every new ./launcher rebuild app starts a fresh one, so if you’ve rebooted and tried another rebuild it should be starting from scratch each time.

Could this be connected to the changes you made to the number of unicorns?

RBoy · February 1, 2024, 9:05pm

That’s a great question. I don’t know and that’s what I was hoping the dev team could comment on. I will try to rebuild it again using 2 unicorns’ instead of 1.

Falco · February 1, 2024, 9:49pm

Yes, as that the default storage in AWS most common instances

I think he is swapping over the network, so it’s as fast as we see here.

While doing the JS build step is also quite slow on DO small instance types, at least it’s local storage AFAIK.

As for what to do next, looks like the only way out is moving to a more expensive instance type.

RBoy · February 1, 2024, 10:02pm

I’ll try that next. What is surprising is that I’ve had this instance running for many years now. Never have an issue with upgrading right up to 3.1.5. Suddenly with 3.2.0 it’s having issues. Was there a big change in the system requirements? I’ve tried multiple instances from snapshots of 3.1.1 to .3.1.5 and none of them complete the upgrade to 3.2.0

Falco · February 1, 2024, 10:07pm

It’s a little weird because that server can run Discourse just fine, the whole problem is “just” building the JS files. That process got a lot heavier over the last year’s, and that even accounting multiple efforts we made to keep it running in small servers

Ed_S · February 1, 2024, 10:07pm

Ah, OK, so with a 1G RAM instance and swapping over the network, you might be fine serving your forum day to day but upgrading - especially recently - is very RAM hungry. I recommend you take the few minutes of downtime, and temporary resize to an 8G instance, and then do your upgrade and then resize down again. I did this, and it cost me less than a dollar extra. It also ran faster and saved me a lot of stress.

The important thing is that you don’t need to double or quadruple your monthly spend - you make a temporary change.

RBoy · February 1, 2024, 10:26pm

Yeah I’m well over 20 hours of downtime now. So you took a snapshot, sized up a new instance, upgraded discourse, then took another snapshot and then sized down a new instance. Did I get that right?

Edit: Can I safely restore a 3.1.1 backup with a 3.2.0 instance? Basically I keep the last working snapshot running to reduce downtime, then backup Discourse from the 3.1.1 instance and once the new 3.2.0 instance is ready restore that 3.1.1 discourse backup?

RBoy · February 1, 2024, 10:32pm

When you say swapping over the network, what are you referring to? The instance is self contained, the only thing on S3 are the backups. I’ve kept all the files local.

Ed_S · February 1, 2024, 10:53pm

Falco reckoned that’s what EC2/Lightsail is doing - I don’t have that knowledge.

I think so. But take a second opinion!

That’s not what I did, but it might be equivalent. With my hosting providers (Digital Ocean and Hetzner) I’m offered an action called “rescale” or similar, whereby my instance goes down, a few minutes pass, and then a new instance appears with the right IP address and all the right disk contents, but with a larger or smaller amount of RAM and CPUs. In both cases, the interface makes a solid distinction between reversible rescaling, and irreversible rescaling. I’ve never taken a provider-level snapshot or backup, and I’ve never used Amazon, so it might be different over there. Sorry, no direct experience!

Edit: might be worth adding EC2/Lightsail into the topic title??

RBoy · February 1, 2024, 11:00pm

I set it back to 2 from 1, didn’t make any difference. Still hanging.

Looking at the stats, it’s not taking CPU, it’s not using all the available RAM/Swap either. I just don’t know why it’s crawling to a point where it went from taking 20 minutes to build 3.1.5 to now hours into 3.2.0 it’s just stuck after the background saving terminates.

What am I missing here Falco? What could it be doing without taking up CPU or RAM or any significant disk I/O?

RBoy · February 2, 2024, 12:05am

I didn’t realize that everytime I setup a new instance from a snapshot the server would contact letsencrypt for a new certificate renewal and now it’s exceeded it’s limit for the next 1 week.

Someway to force discourse to generate new certification using the workaround provided here: Duplicate Certificate Limit - Let's Encrypt

EDIT: So I was able to get the certificates working. I copied the entire letsencrypt directory from /var/discourse/shared/standalone from the valid certification instance to the new instance. I also copied the ssl folder in addition to the letsencrypt folder - but the question is - is that required?

What’s the difference between the letsencrypt folder and ssl folder? I ask because I noticed that the certificate files on the two folders are different in size and timestamp? @JammyDodger would you know why the certificate files are stored in 2 different folders?

RBoy · February 2, 2024, 4:18am

@JammyDodger @Falco @Ed_S thank you all for your help in working through this.

So I was finally able to get it to build on a 2GB RAM instance. It appears that while trying to build 3.2.0 discourse it was asking for 64GB of VM for ember. That’s looks pretty wild.

So it looks like 3.1.5 is the last version that will run on 1GB of RAM. Starting 3.2.0, the minimum appears to be 2GB. @falco @sam can we expect the hardware requirements to keep doubling to upgrade discourse? Discourse can run pretty happily on 1GB, it’s the upgrades that are killing it. Is there no way to prebuild the assets or images and keep them on GitHub and have them downloaded while upgrading to reduce the memory load? The concern is there that for smaller sites, we don’t need 2GB, 4GB, 8GB etc instances.

Ed, just FYI - my strategy does not work with Lightsail. Once you upgrade to a larger instance there’s no way to downgrade. So I can take a snapshot of the 1GB server, then spin up a new 2GB server (and upgrade discourse), but then when I take a snapshot of the 2GB server there’s no way to spin that back up on a 1GB server (because the filesystem goes from 40GB to 60GB and that can’t be reversed). Having said that it’s still not a happy situation that one needs to “upscale” the server just to upgrade the software.

So basically if you’re using Lightsail, you need to get atleast a 2GB instance (bye bye 1GB) and if the 2GB instance for whatever reason (like needing 64GB VM for a small site) fails to build/upgrade discourse, you’re SOL
The only solution would be likely switch over to EC2 or something like Digital Ocean where you can dynamically scale up your RAM/CPU to complete the discourse upgrade and then scale it back down after the upgrade completes. I still have a hard time fathoming that we need more resources to upgrade the software than to run the software.

Finally, it would be nice if the upgrade process gave some sort of error if it needs more RAM but couldn’t get it. Failing silently and causing the entire system to slow down isn’t right (BTW, for reference, I tried overcommit_memory with 1 and 0 on my 1GB instance - but like I said, no errors, even the RAM/swap wasn’t maxed out at the time of hanging from the screenshots above).

sam · February 2, 2024, 5:53am

At some point I don’t mind shipping a bootstrapped image: Can Discourse ship frequent Docker images that do not need to be bootstrapped?

Maybe if this whole series of constraints was simplified:

You run a pg container
You run a redis container
You run a pre-bootstrapped container, with zero support for custom plugins, we would just bundle with official

This could work and be handy for people, the downside is that we would diverge our install story which effectively doubles our support for free install

Running a cache for compiled assets, I am not sure we can pull that off it is going to be too complicated with too many moving parts.

Topic		Replies	Views
Problems building a new server installation	21	1480	July 29, 2023
Upgrading 2.7 to 3.1 failing: "hotlinked_media_status" already exists installation	30	848	October 1, 2023
Error upgrading from 3.0.1 to 3.1.0 installation	14	647	March 28, 2023
Rebuild failure: "extconf failed, exit code 1" -- due to old Docker? installation	11	1091	February 21, 2024
Rebuild failed - FAILED TO BOOTSTRAP installation	20	1304	December 28, 2023

3.1.x to 3.2.0 upgrade hangs/fails on 1GB instance

Related Topics