# 3.1.x to 3.2.0 upgrade hangs/fails on 1GB instance **URL:** https://meta.discourse.org/t/3-1-x-to-3-2-0-upgrade-hangs-fails-on-1gb-instance/293743 **Category:** Self-hosting **Created:** [February 1, 2024, 6:18am UTC](https://meta.discourse.org/t/3-1-x-to-3-2-0-upgrade-hangs-fails-on-1gb-instance/293743 "2024-02-01T06:18:18Z") **Posts on this page:** 20 **Page:** 1
### Author: ![RBoy](https://avatars.discourse-cdn.com/v4/letter/r/2bfe46/32.png) [@RBoy](https://meta.discourse.org/u/RBoy) #### Post date: [February 1, 2024, 6:18am UTC](https://meta.discourse.org/t/3-1-x-to-3-2-0-upgrade-hangs-fails-on-1gb-instance/293743/1 "2024-02-01T06:18:18Z")
Any assistance would be appreciated. Been trying to upgrade 3.1.5 to 3.2.0 for the last 8 hours. It just seems to hang here: ```plaintext Ensuring launcher is up to date Fetching origin Launcher is up-to-date Stopping old container + /usr/bin/docker stop -t 600 app app 2.0.20231218-0429: Pulling from discourse/base Digest: sha256:468f70b9bb4c6d0c6c2bbb3efc1a5e12d145eae57bdb6946b7fe5558beb52dc1 Status: Image is up to date for discourse/base:2.0.20231218-0429 docker.io/discourse/base:2.0.20231218-0429 /usr/local/lib/ruby/gems/3.2.0/gems/pups-1.2.1/lib/pups.rb /usr/local/bin/pups --stdin I, [2024-02-01T05:20:13.663978 #1] INFO -- : Reading from stdin I, [2024-02-01T05:20:13.670645 #1] INFO -- : > locale-gen $LANG && update-locale I, [2024-02-01T05:20:13.712975 #1] INFO -- : Generating locales (this might take a while)... Generation complete. I, [2024-02-01T05:20:13.713587 #1] INFO -- : > mkdir -p /shared/postgres_run I, [2024-02-01T05:20:13.717850 #1] INFO -- : I, [2024-02-01T05:20:13.718463 #1] INFO -- : > chown postgres:postgres /shared/postgres_run I, [2024-02-01T05:20:13.722273 #1] INFO -- : I, [2024-02-01T05:20:13.722901 #1] INFO -- : > chmod 775 /shared/postgres_run I, [2024-02-01T05:20:13.726117 #1] INFO -- : I, [2024-02-01T05:20:13.726724 #1] INFO -- : > rm -fr /var/run/postgresql I, [2024-02-01T05:20:13.730093 #1] INFO -- : I, [2024-02-01T05:20:13.730719 #1] INFO -- : > ln -s /shared/postgres_run /var/run/postgresql I, [2024-02-01T05:20:13.734550 #1] INFO -- : I, [2024-02-01T05:20:13.735116 #1] INFO -- : > socat /dev/null UNIX-CONNECT:/shared/postgres_run/.s.PGSQL.5432 || exit 0 && echo postgres already running stop container ; exit 1 2024/02/01 05:20:13 socat[18] E connect(6, AF=1 "/shared/postgres_run/.s.PGSQL.5432", 36): No such file or directory I, [2024-02-01T05:20:13.757786 #1] INFO -- : I, [2024-02-01T05:20:13.758259 #1] INFO -- : > rm -fr /shared/postgres_run/.s* I, [2024-02-01T05:20:13.761231 #1] INFO -- : I, [2024-02-01T05:20:13.761705 #1] INFO -- : > rm -fr /shared/postgres_run/*.pid I, [2024-02-01T05:20:13.764603 #1] INFO -- : I, [2024-02-01T05:20:13.765030 #1] INFO -- : > mkdir -p /shared/postgres_run/13-main.pg_stat_tmp I, [2024-02-01T05:20:13.767619 #1] INFO -- : I, [2024-02-01T05:20:13.768243 #1] INFO -- : > chown postgres:postgres /shared/postgres_run/13-main.pg_stat_tmp I, [2024-02-01T05:20:13.770687 #1] INFO -- : I, [2024-02-01T05:20:13.775750 #1] INFO -- : File > /etc/service/postgres/run chmod: +x chown: I, [2024-02-01T05:20:13.780586 #1] INFO -- : File > /etc/service/postgres/log/run chmod: +x chown: I, [2024-02-01T05:20:13.785496 #1] INFO -- : File > /etc/runit/3.d/99-postgres chmod: +x chown: I, [2024-02-01T05:20:13.790412 #1] INFO -- : File > /root/upgrade_postgres chmod: +x chown: I, [2024-02-01T05:20:13.791046 #1] INFO -- : > chown -R root /var/lib/postgresql/13/main I, [2024-02-01T05:20:16.705999 #1] INFO -- : I, [2024-02-01T05:20:16.706819 #1] INFO -- : > [! -e /shared/postgres_data] && install -d -m 0755 -o postgres -g postgres /shared/postgres_data && sudo -E -u postgres /usr/lib/postgresql/13/bin/initdb -D /shared/postgres_data || exit 0 I, [2024-02-01T05:20:16.709471 #1] INFO -- : I, [2024-02-01T05:20:16.709873 #1] INFO -- : > chown -R postgres:postgres /shared/postgres_data I, [2024-02-01T05:20:16.748408 #1] INFO -- : I, [2024-02-01T05:20:16.749205 #1] INFO -- : > chown -R postgres:postgres /var/run/postgresql I, [2024-02-01T05:20:16.752138 #1] INFO -- : I, [2024-02-01T05:20:16.752728 #1] INFO -- : > /root/upgrade_postgres I, [2024-02-01T05:20:16.759412 #1] INFO -- : I, [2024-02-01T05:20:16.759890 #1] INFO -- : > rm /root/upgrade_postgres I, [2024-02-01T05:20:16.762107 #1] INFO -- : I, [2024-02-01T05:20:16.764889 #1] INFO -- : Replacing data_directory = '/var/lib/postgresql/13/main' with data_directory = '/shared/postgres_data' in /etc/postgresql/13/main/postgresql.conf I, [2024-02-01T05:20:16.765803 #1] INFO -- : Replacing (?-mix:#?listen_addresses *=.*) with listen_addresses = '*' in /etc/postgresql/13/main/postgresql.conf I, [2024-02-01T05:20:16.766486 #1] INFO -- : Replacing (?-mix:#?synchronous_commit *=.*) with synchronous_commit = $db_synchronous_commit in /etc/postgresql/13/main/postgresql.conf I, [2024-02-01T05:20:16.767571 #1] INFO -- : Replacing (?-mix:#?shared_buffers *=.*) with shared_buffers = $db_shared_buffers in /etc/postgresql/13/main/postgresql.conf I, [2024-02-01T05:20:16.768666 #1] INFO -- : Replacing (?-mix:#?work_mem *=.*) with work_mem = $db_work_mem in /etc/postgresql/13/main/postgresql.conf I, [2024-02-01T05:20:16.769702 #1] INFO -- : Replacing (?-mix:#?default_text_search_config *=.*) with default_text_search_config = '$db_default_text_search_config' in /etc/postgresql/13/main/postgresql.conf I, [2024-02-01T05:20:16.770784 #1] INFO -- : > install -d -m 0755 -o postgres -g postgres /shared/postgres_backup I, [2024-02-01T05:20:16.777960 #1] INFO -- : I, [2024-02-01T05:20:16.778711 #1] INFO -- : Replacing (?-mix:#?checkpoint_segments *=.*) with checkpoint_segments = $db_checkpoint_segments in /etc/postgresql/13/main/postgresql.conf I, [2024-02-01T05:20:16.779400 #1] INFO -- : Replacing (?-mix:#?logging_collector *=.*) with logging_collector = $db_logging_collector in /etc/postgresql/13/main/postgresql.conf I, [2024-02-01T05:20:16.780719 #1] INFO -- : Replacing (?-mix:#?log_min_duration_statement *=.*) with log_min_duration_statement = $db_log_min_duration_statement in /etc/postgresql/13/main/postgresql.conf I, [2024-02-01T05:20:16.782507 #1] INFO -- : Replacing (?-mix:^#local +replication +postgres +peer$) with local replication postgres peer in /etc/postgresql/13/main/pg_hba.conf I, [2024-02-01T05:20:16.783245 #1] INFO -- : Replacing (?-mix:^host.*all.*all.*127.*$) with host all all 0.0.0.0/0 md5 in /etc/postgresql/13/main/pg_hba.conf I, [2024-02-01T05:20:16.783845 #1] INFO -- : Replacing (?-mix:^host.*all.*all.*::1\/128.*$) with host all all ::/0 md5 in /etc/postgresql/13/main/pg_hba.conf I, [2024-02-01T05:20:16.784749 #1] INFO -- : > HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main I, [2024-02-01T05:20:16.786361 #1] INFO -- : > sleep 5 2024-02-01 05:20:16.885 UTC [41] LOG: starting PostgreSQL 13.13 (Debian 13.13-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit 2024-02-01 05:20:16.887 UTC [41] LOG: listening on IPv4 address "0.0.0.0", port 5432 2024-02-01 05:20:16.887 UTC [41] LOG: listening on IPv6 address "::", port 5432 2024-02-01 05:20:16.893 UTC [41] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" 2024-02-01 05:20:16.902 UTC [44] LOG: database system was shut down at 2024-02-01 05:20:02 UTC 2024-02-01 05:20:16.911 UTC [41] LOG: database system is ready to accept connections I, [2024-02-01T05:20:21.790669 #1] INFO -- : I, [2024-02-01T05:20:21.790891 #1] INFO -- : > su postgres -c 'createdb discourse' || true 2024-02-01 05:20:21.897 UTC [54] postgres@postgres ERROR: database "discourse" already exists 2024-02-01 05:20:21.897 UTC [54] postgres@postgres STATEMENT: CREATE DATABASE discourse; createdb: error: database creation failed: ERROR: database "discourse" already exists I, [2024-02-01T05:20:21.900341 #1] INFO -- : I, [2024-02-01T05:20:21.900879 #1] INFO -- : > su postgres -c 'psql discourse -c "create user discourse;"' || true 2024-02-01 05:20:21.987 UTC [58] postgres@discourse ERROR: role "discourse" already exists 2024-02-01 05:20:21.987 UTC [58] postgres@discourse STATEMENT: create user discourse; ERROR: role "discourse" already exists I, [2024-02-01T05:20:21.990907 #1] INFO -- : I, [2024-02-01T05:20:21.991456 #1] INFO -- : > su postgres -c 'psql discourse -c "grant all privileges on database discourse to discourse;"' || true I, [2024-02-01T05:20:22.048297 #1] INFO -- : GRANT I, [2024-02-01T05:20:22.048988 #1] INFO -- : > su postgres -c 'psql discourse -c "alter schema public owner to discourse;"' I, [2024-02-01T05:20:22.106460 #1] INFO -- : ALTER SCHEMA I, [2024-02-01T05:20:22.107187 #1] INFO -- : > su postgres -c 'psql template1 -c "create extension if not exists hstore;"' NOTICE: extension "hstore" already exists, skipping I, [2024-02-01T05:20:22.183367 #1] INFO -- : CREATE EXTENSION I, [2024-02-01T05:20:22.183889 #1] INFO -- : > su postgres -c 'psql template1 -c "create extension if not exists pg_trgm;"' NOTICE: extension "pg_trgm" already exists, skipping I, [2024-02-01T05:20:22.240624 #1] INFO -- : CREATE EXTENSION I, [2024-02-01T05:20:22.241181 #1] INFO -- : > su postgres -c 'psql template1 -c "create extension if not exists vector;"' NOTICE: extension "vector" already exists, skipping I, [2024-02-01T05:20:22.297658 #1] INFO -- : CREATE EXTENSION I, [2024-02-01T05:20:22.298255 #1] INFO -- : > su postgres -c 'psql discourse -c "create extension if not exists hstore;"' NOTICE: extension "hstore" already exists, skipping I, [2024-02-01T05:20:22.356472 #1] INFO -- : CREATE EXTENSION I, [2024-02-01T05:20:22.356991 #1] INFO -- : > su postgres -c 'psql discourse -c "create extension if not exists pg_trgm;"' NOTICE: extension "pg_trgm" already exists, skipping I, [2024-02-01T05:20:22.414056 #1] INFO -- : CREATE EXTENSION I, [2024-02-01T05:20:22.414618 #1] INFO -- : > su postgres -c 'psql discourse -c "create extension if not exists vector;"' NOTICE: extension "vector" already exists, skipping I, [2024-02-01T05:20:22.470035 #1] INFO -- : CREATE EXTENSION I, [2024-02-01T05:20:22.470658 #1] INFO -- : > sudo -u postgres psql discourse I, [2024-02-01T05:20:22.475733 #1] INFO -- : update pg_database set encoding = pg_char_to_encoding('UTF8') where datname = 'discourse' AND encoding = pg_char_to_encoding('SQL_ASCII'); I, [2024-02-01T05:20:22.567511 #1] INFO -- : File > /var/lib/postgresql/take-database-backup chmod: +x chown: postgres:postgres I, [2024-02-01T05:20:22.572125 #1] INFO -- : File > /var/spool/cron/crontabs/postgres chmod: chown: I, [2024-02-01T05:20:22.572526 #1] INFO -- : > echo postgres installed! I, [2024-02-01T05:20:22.575793 #1] INFO -- : postgres installed! I, [2024-02-01T05:20:22.580912 #1] INFO -- : File > /etc/service/redis/run chmod: +x chown: I, [2024-02-01T05:20:22.585785 #1] INFO -- : File > /etc/service/redis/log/run chmod: +x chown: I, [2024-02-01T05:20:22.590830 #1] INFO -- : File > /etc/runit/3.d/10-redis chmod: +x chown: I, [2024-02-01T05:20:22.593112 #1] INFO -- : Replacing daemonize yes with in /etc/redis/redis.conf I, [2024-02-01T05:20:22.594012 #1] INFO -- : Replacing (?-mix:^pidfile.*$) with in /etc/redis/redis.conf I, [2024-02-01T05:20:22.594826 #1] INFO -- : > install -d -m 0755 -o redis -g redis /shared/redis_data I, [2024-02-01T05:20:22.597926 #1] INFO -- : I, [2024-02-01T05:20:22.598727 #1] INFO -- : Replacing (?-mix:^logfile.*$) with logfile "" in /etc/redis/redis.conf I, [2024-02-01T05:20:22.599588 #1] INFO -- : Replacing (?-mix:^bind .*$) with in /etc/redis/redis.conf I, [2024-02-01T05:20:22.601300 #1] INFO -- : Replacing (?-mix:^dir .*$) with dir /shared/redis_data in /etc/redis/redis.conf I, [2024-02-01T05:20:22.603062 #1] INFO -- : Replacing (?-mix:^protected-mode yes) with protected-mode no in /etc/redis/redis.conf I, [2024-02-01T05:20:22.604759 #1] INFO -- : Replacing # io-threads 4 with io-threads $redis_io_threads in /etc/redis/redis.conf I, [2024-02-01T05:20:22.606503 #1] INFO -- : > echo redis installed I, [2024-02-01T05:20:22.609035 #1] INFO -- : redis installed I, [2024-02-01T05:20:22.609661 #1] INFO -- : > cat /etc/redis/redis.conf | grep logfile I, [2024-02-01T05:20:22.614668 #1] INFO -- : logfile "" I, [2024-02-01T05:20:22.615160 #1] INFO -- : > exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf I, [2024-02-01T05:20:22.616826 #1] INFO -- : > sleep 10 110:C 01 Feb 2024 05:20:22.631 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 110:C 01 Feb 2024 05:20:22.631 # Redis version=7.0.7, bits=64, commit=00000000, modified=0, pid=110, just started 110:C 01 Feb 2024 05:20:22.631 # Configuration loaded 110:M 01 Feb 2024 05:20:22.632 * monotonic clock: POSIX clock_gettime 110:M 01 Feb 2024 05:20:22.633 * Running mode=standalone, port=6379. 110:M 01 Feb 2024 05:20:22.633 # Server initialized 110:M 01 Feb 2024 05:20:22.634 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. 110:M 01 Feb 2024 05:20:22.634 # WARNING Your system is configured to use the 'xen' clocksource which might lead to degraded performance. Check the result of the [slow-clocksource] system check: run 'redis-server --check-system' to check if the system's clocksource isn't degrading performance. 110:M 01 Feb 2024 05:20:22.635 * Loading RDB produced by version 7.0.7 110:M 01 Feb 2024 05:20:22.635 * RDB age 28 seconds 110:M 01 Feb 2024 05:20:22.635 * RDB memory usage when created 12.85 Mb 110:M 01 Feb 2024 05:20:22.670 * Done loading RDB, keys loaded: 11479, keys expired: 6. 110:M 01 Feb 2024 05:20:22.670 * DB loaded from disk: 0.035 seconds 110:M 01 Feb 2024 05:20:22.670 * Ready to accept connections I, [2024-02-01T05:20:32.621081 #1] INFO -- : I, [2024-02-01T05:20:32.621808 #1] INFO -- : > thpoff echo "thpoff is installed!" I, [2024-02-01T05:20:32.626389 #1] INFO -- : thpoff is installed! I, [2024-02-01T05:20:32.626901 #1] INFO -- : > /usr/local/bin/ruby -e 'if ENV["DISCOURSE_SMTP_ADDRESS"] == "smtp.example.com"; puts "Aborting! Mail is not configured!"; exit 1; end' I, [2024-02-01T05:20:32.707287 #1] INFO -- : I, [2024-02-01T05:20:32.707923 #1] INFO -- : > /usr/local/bin/ruby -e 'if ENV["DISCOURSE_HOSTNAME"] == "discourse.example.com"; puts "Aborting! Domain is not configured!"; exit 1; end' I, [2024-02-01T05:20:32.788125 #1] INFO -- : I, [2024-02-01T05:20:32.788819 #1] INFO -- : > /usr/local/bin/ruby -e 'if (ENV["DISCOURSE_CDN_URL"] || "")[0..1] == "//"; puts "Aborting! CDN must have a protocol specified. Once fixed you should rebake your posts now to correct all posts."; exit 1; end' I, [2024-02-01T05:20:32.868839 #1] INFO -- : I, [2024-02-01T05:20:32.869502 #1] INFO -- : > rm -f /etc/cron.d/anacron I, [2024-02-01T05:20:32.872477 #1] INFO -- : I, [2024-02-01T05:20:32.875792 #1] INFO -- : File > /etc/cron.d/anacron chmod: chown: I, [2024-02-01T05:20:32.882091 #1] INFO -- : File > /etc/runit/1.d/copy-env chmod: +x chown: I, [2024-02-01T05:20:32.887278 #1] INFO -- : File > /etc/service/unicorn/run chmod: +x chown: I, [2024-02-01T05:20:32.892443 #1] INFO -- : File > /etc/service/nginx/run chmod: +x chown: I, [2024-02-01T05:20:32.897678 #1] INFO -- : File > /etc/runit/3.d/01-nginx chmod: +x chown: I, [2024-02-01T05:20:32.902952 #1] INFO -- : File > /etc/runit/3.d/02-unicorn chmod: +x chown: I, [2024-02-01T05:20:32.903394 #1] INFO -- : Replacing # postgres with sv start postgres || exit 1 in /etc/service/unicorn/run I, [2024-02-01T05:20:32.903939 #1] INFO -- : > exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf I, [2024-02-01T05:20:32.906369 #1] INFO -- : > cd /var/www/discourse && sudo -H -E -u discourse git reset --hard 137:C 01 Feb 2024 05:20:32.915 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 137:C 01 Feb 2024 05:20:32.918 # Redis version=7.0.7, bits=64, commit=00000000, modified=0, pid=137, just started 137:C 01 Feb 2024 05:20:32.918 # Configuration loaded 137:M 01 Feb 2024 05:20:32.919 * monotonic clock: POSIX clock_gettime 137:M 01 Feb 2024 05:20:32.920 # Warning: Could not create server TCP listening socket *:6379: bind: Address already in use 137:M 01 Feb 2024 05:20:32.920 # Failed listening on port 6379 (TCP), aborting. Updating files: 100% (33715/33715), done. I, [2024-02-01T05:20:39.803997 #1] INFO -- : HEAD is now at 6ab1a19e9 DEV: Convert min_trust_level_to_allow_invite to groups (#24893) I, [2024-02-01T05:20:39.804645 #1] INFO -- : > cd /var/www/discourse && sudo -H -E -u discourse git clean -f I, [2024-02-01T05:20:40.031462 #1] INFO -- : I, [2024-02-01T05:20:40.032125 #1] INFO -- : > cd /var/www/discourse && sudo -H -E -u discourse bash -c ' set -o errexit if [$(git rev-parse --is-shallow-repository) == "true"]; then git remote set-branches --add origin main git remote set-branches origin stable git fetch --depth 1 origin stable else git fetch --tags --prune-tags --prune --force origin fi ' From https://github.com/discourse/discourse - [deleted] (none) -> origin/0-drop-code - [deleted] (none) -> origin/bulk-button-class - [deleted] (none) -> origin/dependabot/bundler/excon-0.108.0 - [deleted] (none) -> origin/dependabot/bundler/hashdiff-1.1.0 - [deleted] (none) -> origin/dependabot/bundler/parallel-1.24.0 - [deleted] (none) -> origin/dependabot/bundler/rubocop-discourse-3.6.0 - [deleted] (none) -> origin/dependabot/npm_and_yarn/app/assets/javascripts/uppy/drop-target-2.0.2 - [deleted] (none) -> origin/dev/add-new-chat-metrics - [deleted] (none) -> origin/dev/allow-chat-mentions-to-have-several-notifications - [deleted] (none) -> origin/dev/convert-min-trust-level-to-create-tag-to-group - [deleted] (none) -> origin/dev/convert-min-trust-level-to-ignore-to-groups - [deleted] (none) -> origin/dev/rethink-mass-mentions-polymorphic - [deleted] (none) -> origin/feature/migrations-schema-generator - [deleted] (none) -> origin/fix-join-filter-upload - [deleted] (none) -> origin/fix-modal-flash-minor - [deleted] (none) -> origin/global-notice-reactive - [deleted] (none) -> origin/labeler - [deleted] (none) -> origin/low-ram-help - [deleted] (none) -> origin/multi-invite - [deleted] (none) -> origin/table-builder-modal-modifier - [deleted] (none) -> origin/wip_categories - [deleted] (none) -> origin/wizard-split 6ab1a19e9..969ab0fd6 main -> origin/main * [new branch] 0-category-reorder -> origin/0-category-reorder * [new branch] 0-ember-bump -> origin/0-ember-bump * [new branch] 0-event-helper -> origin/0-event-helper * [new branch] 0-extend-native -> origin/0-extend-native * [new branch] 0-rev -> origin/0-rev * [new branch] add-api-for-forced-auth -> origin/add-api-for-forced-auth * [new branch] amend-visit-copy -> origin/amend-visit-copy * [new branch] asciinema-iframe -> origin/asciinema-iframe * [new branch] chat/original-message-link -> origin/chat/original-message-link * [new branch] convert-header-to-glimmer -> origin/convert-header-to-glimmer * [new branch] decorator-transforms -> origin/decorator-transforms * [new branch] dependabot/bundler/omniauth-2.0.0 -> origin/dependabot/bundler/omniauth-2.0.0 * [new branch] dependabot/bundler/parallel_tests-4.4.0 -> origin/dependabot/bundler/parallel_tests-4.4.0 * [new branch] dev-convert-topic-map-expanded-widget-to-component -> origin/dev-convert-topic-map-expanded-widget-to-component * [new branch] dev/check-whether-online-user-ids-may-be-memoized-between-job-runs -> origin/dev/check-whether-online-user-ids-may-be-memoized-between-job-runs * [new branch] dev/drop-the-user-id-column-from-chat-mentions -> origin/dev/drop-the-user-id-column-from-chat-mentions + e0c0191bb...feb00e04c dev/dry-up-user-status-serializers -> origin/dev/dry-up-user-status-serializers (forced update) * [new branch] dev/experiment-theme-settings-free-ui -> origin/dev/experiment-theme-settings-free-ui * [new branch] dev/make-admins-tl4-in-tests -> origin/dev/make-admins-tl4-in-tests * [new branch] dev/promote-plugin-post-deploy-migrations -> origin/dev/promote-plugin-post-deploy-migrations * [new branch] dev/proof-of-concept-rewriting-mentions-notification-logic -> origin/dev/proof-of-concept-rewriting-mentions-notification-logic * [new branch] dev_category_order_database -> origin/dev_category_order_database * [new branch] dev_update_category_position -> origin/dev_update_category_position * [new branch] ellaestigoy-patch-2 -> origin/ellaestigoy-patch-2 * [new branch] fast-edit-expand-regex -> origin/fast-edit-expand-regex * [new branch] feature/add-new-illegal-flag-reason -> origin/feature/add-new-illegal-flag-reason * [new branch] feature/custom-date-range -> origin/feature/custom-date-range * [new branch] feature/paste-upload-on-first-upload-input-templates -> origin/feature/paste-upload-on-first-upload-input-templates * [new branch] feature_async_load_colors -> origin/feature_async_load_colors * [new branch] fix-email-category-badge -> origin/fix-email-category-badge * [new branch] fix_category_chooser -> origin/fix_category_chooser * [new branch] fix_in_any_groups -> origin/fix_in_any_groups * [new branch] fix_preload_sidebar_categories -> origin/fix_preload_sidebar_categories * [new branch] fix_reviewable_translation -> origin/fix_reviewable_translation 7f2bdca65..b851b879c generic-import-latest -> origin/generic-import-latest * [new branch] hijacks -> origin/hijacks * [new branch] issue/avatar-upload-permission-serialize -> origin/issue/avatar-upload-permission-serialize * [new branch] issue/run-admin-sidebar-init-after-logs-notice -> origin/issue/run-admin-sidebar-init-after-logs-notice + 90a3535b7...e6e64cfd5 migrations-convert -> origin/migrations-convert (forced update) + 0fa691496...b0f74a188 plugin-extra-js -> origin/plugin-extra-js (forced update) * [new branch] promote-migration-tweak -> origin/promote-migration-tweak * [new branch] revert-25009-dev/convert-min-trust-to-allow-self-wiki-to-groups -> origin/revert-25009-dev/convert-min-trust-to-allow-self-wiki-to-groups * [new branch] revert-bulk-topic-actions -> origin/revert-bulk-topic-actions * [new branch] sidebar-above-outlet -> origin/sidebar-above-outlet * [new branch] site-logo-height -> origin/site-logo-height * [new branch] soft-load-topic-results -> origin/soft-load-topic-results d7f7f79b6..968b1a57e stable -> origin/stable * [new branch] support-reset-bump-date-post-id -> origin/support-reset-bump-date-post-id 6ab1a19e9..969ab0fd6 tests-passed -> origin/tests-passed * [new branch] unhide_lazy_load_categories -> origin/unhide_lazy_load_categories * [new branch] universal-nonce -> origin/universal-nonce * [new branch] unread-mobile-indicator -> origin/unread-mobile-indicator * [new branch] virtual-scroll-list -> origin/virtual-scroll-list t [tag update] beta -> beta t [tag update] latest-release -> latest-release * [new tag] v3.1.4 -> v3.1.4 * [new tag] v3.1.5 -> v3.1.5 * [new tag] v3.2.0 -> v3.2.0 * [new tag] v3.2.0.beta4 -> v3.2.0.beta4 * [new tag] v3.2.0.beta5 -> v3.2.0.beta5 I, [2024-02-01T05:20:40.935200 #1] INFO -- : I, [2024-02-01T05:20:40.935686 #1] INFO -- : > cd /var/www/discourse && sudo -H -E -u discourse bash -c ' set -o errexit if [[$(git symbolic-ref --short HEAD) == stable]] ; then git pull else git -c advice.detachedHead=false checkout stable fi ' Switched to a new branch 'stable' I, [2024-02-01T05:20:44.064865 #1] INFO -- : Branch 'stable' set up to track remote branch 'stable' from 'origin'. I, [2024-02-01T05:20:44.065354 #1] INFO -- : > cd /var/www/discourse && sudo -H -E -u discourse git config user.discourse-version stable I, [2024-02-01T05:20:44.077044 #1] INFO -- : I, [2024-02-01T05:20:44.077436 #1] INFO -- : > cd /var/www/discourse && mkdir -p tmp I, [2024-02-01T05:20:44.085161 #1] INFO -- : I, [2024-02-01T05:20:44.085555 #1] INFO -- : > cd /var/www/discourse && chown discourse:www-data tmp I, [2024-02-01T05:20:44.090473 #1] INFO -- : I, [2024-02-01T05:20:44.090945 #1] INFO -- : > cd /var/www/discourse && mkdir -p tmp/pids I, [2024-02-01T05:20:44.094272 #1] INFO -- : I, [2024-02-01T05:20:44.094646 #1] INFO -- : > cd /var/www/discourse && mkdir -p tmp/sockets I, [2024-02-01T05:20:44.097962 #1] INFO -- : I, [2024-02-01T05:20:44.098333 #1] INFO -- : > cd /var/www/discourse && touch tmp/.gitkeep I, [2024-02-01T05:20:44.102920 #1] INFO -- : I, [2024-02-01T05:20:44.103294 #1] INFO -- : > cd /var/www/discourse && mkdir -p /shared/log/rails I, [2024-02-01T05:20:44.106489 #1] INFO -- : I, [2024-02-01T05:20:44.106964 #1] INFO -- : > cd /var/www/discourse && bash -c "touch -a /shared/log/rails/{production,production_errors,unicorn.stdout,unicorn.stderr,sidekiq}.log" I, [2024-02-01T05:20:44.112554 #1] INFO -- : I, [2024-02-01T05:20:44.112953 #1] INFO -- : > cd /var/www/discourse && bash -c "ln -s /shared/log/rails/{production,production_errors,unicorn.stdout,unicorn.stderr,sidekiq}.log /var/www/discourse/log" I, [2024-02-01T05:20:44.119868 #1] INFO -- : I, [2024-02-01T05:20:44.120247 #1] INFO -- : > cd /var/www/discourse && bash -c "mkdir -p /shared/{uploads,backups}" I, [2024-02-01T05:20:44.125723 #1] INFO -- : I, [2024-02-01T05:20:44.126103 #1] INFO -- : > cd /var/www/discourse && bash -c "ln -s /shared/{uploads,backups} /var/www/discourse/public" I, [2024-02-01T05:20:44.130710 #1] INFO -- : I, [2024-02-01T05:20:44.131085 #1] INFO -- : > cd /var/www/discourse && bash -c "mkdir -p /shared/tmp/{backups,restores}" I, [2024-02-01T05:20:44.136397 #1] INFO -- : I, [2024-02-01T05:20:44.136900 #1] INFO -- : > cd /var/www/discourse && bash -c "ln -s /shared/tmp/{backups,restores} /var/www/discourse/tmp" I, [2024-02-01T05:20:44.141456 #1] INFO -- : I, [2024-02-01T05:20:44.141832 #1] INFO -- : > cd /var/www/discourse && chown -R discourse:www-data /shared/log/rails /shared/uploads /shared/backups /shared/tmp I, [2024-02-01T05:20:44.267143 #1] INFO -- : I, [2024-02-01T05:20:44.267732 #1] INFO -- : > cd /var/www/discourse && [! -d public/plugins] || find public/plugins/ -maxdepth 1 -xtype l -delete I, [2024-02-01T05:20:44.270260 #1] INFO -- : I, [2024-02-01T05:20:44.271569 #1] INFO -- : Replacing # redis with sv start redis || exit 1 in /etc/service/unicorn/run I, [2024-02-01T05:20:44.273289 #1] INFO -- : > cd /var/www/discourse/plugins && git clone https://github.com/discourse/docker_manager.git Cloning into 'docker_manager'... I, [2024-02-01T05:20:45.047892 #1] INFO -- : I, [2024-02-01T05:20:45.048485 #1] INFO -- : > cd /var/www/discourse/plugins && git clone https://github.com/discourse/discourse-voting.git Cloning into 'discourse-voting'... I, [2024-02-01T05:20:45.349517 #1] INFO -- : I, [2024-02-01T05:20:45.350128 #1] INFO -- : > cd /var/www/discourse/plugins && git clone https://github.com/discourse/discourse-solved.git Cloning into 'discourse-solved'... I, [2024-02-01T05:20:45.696361 #1] INFO -- : I, [2024-02-01T05:20:45.696958 #1] INFO -- : > cd /var/www/discourse/plugins && git clone https://github.com/discourse/discourse-policy.git Cloning into 'discourse-policy'... I, [2024-02-01T05:20:45.985549 #1] INFO -- : I, [2024-02-01T05:20:45.986116 #1] INFO -- : > cd /var/www/discourse/plugins && git clone https://github.com/discourse/discourse-adplugin.git Cloning into 'discourse-adplugin'... I, [2024-02-01T05:20:46.333531 #1] INFO -- : I, [2024-02-01T05:20:46.334121 #1] INFO -- : > cd /var/www/discourse/plugins && git clone https://github.com/discourse/discourse-data-explorer.git Cloning into 'discourse-data-explorer'... I, [2024-02-01T05:20:46.684891 #1] INFO -- : I, [2024-02-01T05:20:46.685512 #1] INFO -- : > cd /var/www/discourse/plugins && git clone https://github.com/communiteq/discourse-suppress-category-from-latest.git Cloning into 'discourse-suppress-category-from-latest'... I, [2024-02-01T05:20:46.860067 #1] INFO -- : I, [2024-02-01T05:20:46.860646 #1] INFO -- : > cd /var/www/discourse/plugins && git clone https://github.com/discourse/discourse-bbcode-color Cloning into 'discourse-bbcode-color'... I, [2024-02-01T05:20:47.089640 #1] INFO -- : I, [2024-02-01T05:20:47.090378 #1] INFO -- : > cp /var/www/discourse/config/nginx.sample.conf /etc/nginx/conf.d/discourse.conf I, [2024-02-01T05:20:47.099746 #1] INFO -- : I, [2024-02-01T05:20:47.100326 #1] INFO -- : > rm /etc/nginx/sites-enabled/default I, [2024-02-01T05:20:47.103575 #1] INFO -- : I, [2024-02-01T05:20:47.104073 #1] INFO -- : > mkdir -p /var/nginx/cache I, [2024-02-01T05:20:47.107202 #1] INFO -- : I, [2024-02-01T05:20:47.108186 #1] INFO -- : Replacing pid /run/nginx.pid; with daemon off; in /etc/nginx/nginx.conf I, [2024-02-01T05:20:47.109528 #1] INFO -- : Replacing (?m-ix:upstream[^\}]+\}) with upstream discourse { server 127.0.0.1:3000; } in /etc/nginx/conf.d/discourse.conf I, [2024-02-01T05:20:47.110130 #1] INFO -- : Replacing (?-mix:server_name.+$) with server_name _ ; in /etc/nginx/conf.d/discourse.conf I, [2024-02-01T05:20:47.110992 #1] INFO -- : Replacing (?-mix:client_max_body_size.+$) with client_max_body_size $upload_size ; in /etc/nginx/conf.d/discourse.conf I, [2024-02-01T05:20:47.111810 #1] INFO -- : Replacing (?-mix:worker_connections.+$) with worker_connections $nginx_worker_connections ; in /etc/nginx/nginx.conf I, [2024-02-01T05:20:47.112296 #1] INFO -- : > echo "done configuring web" I, [2024-02-01T05:20:47.114305 #1] INFO -- : done configuring web I, [2024-02-01T05:20:47.114863 #1] INFO -- : > cd /var/www/discourse && gem install bundler --conservative -v $(awk '/BUNDLED WITH/ { getline; gsub(/ /,""); print $0 }' Gemfile.lock) I, [2024-02-01T05:20:48.019548 #1] INFO -- : Successfully installed bundler-2.5.3 1 gem installed I, [2024-02-01T05:20:48.020208 #1] INFO -- : > cd /var/www/discourse && find /var/www/discourse ! -user discourse -exec chown discourse {} \+ I, [2024-02-01T05:20:56.712671 #1] INFO -- : I, [2024-02-01T05:20:56.723363 #1] INFO -- : > cd /var/www/discourse && if ["stable" != "tests-passed"]; then rm -rf app/assets/javascripts/node_modules fi I, [2024-02-01T05:20:59.502523 #1] INFO -- : I, [2024-02-01T05:20:59.503242 #1] INFO -- : > cd /var/www/discourse && su discourse -c 'yarn install --frozen-lockfile && yarn cache clean' ^[[D^[[D^[[Dwarning "@discourse/lint-configs > eslint-plugin-ember > ember-eslint-parser@0.2.5" has unmet peer dependency "@typescript-eslint/parser@^6.15.0". warning "@discourse/lint-configs > eslint-plugin-ember > ember-eslint-parser@0.2.5" has incorrect peer dependency "typescript@^5.3.3". warning " > @glint/environment-ember-loose@1.3.0" has unmet peer dependency "@glimmer/component@^1.1.2". warning Resolution field "unset-value@2.0.1" is incompatible with requested version "unset-value@^1.0.0" warning Pattern ["wrap-ansi@^7.0.0"] is trying to unpack in the same destination "/home/discourse/.cache/yarn/v6/npm-wrap-ansi-cjs-7.0.0-67e145cff510a6a6984bdf1152911d69d2eb9e43-integrity/node_modules/wrap-ansi-cjs" as pattern ["wrap-ansi-cjs@npm:wrap-ansi@^7.0.0"]. This could result in non-deterministic behavior, skipping. warning " > discourse-markdown-it@1.0.0" has unmet peer dependency "xss@*". warning "workspace-aggregator-ca265ca6-43f3-4f9e-bc1f-e2610715a50c > discourse > @uppy/aws-s3@3.0.6" has incorrect peer dependency "@uppy/core@^3.1.2". warning "workspace-aggregator-ca265ca6-43f3-4f9e-bc1f-e2610715a50c > discourse > @uppy/aws-s3-multipart@3.1.3" has incorrect peer dependency "@uppy/core@^3.1.2". warning "workspace-aggregator-ca265ca6-43f3-4f9e-bc1f-e2610715a50c > discourse > @uppy/xhr-upload@3.1.1" has incorrect peer dependency "@uppy/core@^3.1.2". warning "workspace-aggregator-ca265ca6-43f3-4f9e-bc1f-e2610715a50c > discourse > ember-router-service-refresh-polyfill@1.1.0" has incorrect peer dependency "ember-source@^3.4 || 4". warning "workspace-aggregator-ca265ca6-43f3-4f9e-bc1f-e2610715a50c > discourse-plugins > ember-this-fallback@0.4.0" has unmet peer dependency "ember-source@^3.28.11 || ^4.0.0". warning "workspace-aggregator-ca265ca6-43f3-4f9e-bc1f-e2610715a50c > admin > ember-source > router_js@8.0.3" has unmet peer dependency "rsvp@^4.8.5". warning "workspace-aggregator-ca265ca6-43f3-4f9e-bc1f-e2610715a50c > discourse > @uppy/aws-s3 > @uppy/xhr-upload@3.3.0" has incorrect peer dependency "@uppy/core@^3.2.1". I, [2024-02-01T05:23:59.990567 #1] INFO -- : yarn install v1.22.19 [1/5] Validating package.json... [2/5] Resolving packages... [3/5] Fetching packages... [4/5] Linking dependencies... [5/5] Building fresh packages... $ yarn --cwd app/assets/javascripts $(node -e 'const argv = JSON.parse(process.env.npm_config_argv).original; const passthrough = [`--frozen-lockfile`, `-s`].filter(arg => argv.includes(arg)); console.log(passthrough.join(` `));') yarn install v1.22.19 [1/4] Resolving packages... [2/4] Fetching packages... [3/4] Linking dependencies... [4/4] Building fresh packages... $ ./run-patch-package patch-package 8.0.0 Applying patches... @ember/test-helpers@2.9.4 ✔ babel-plugin-debug-macros@0.3.4 ✔ content-tag@1.2.2 ✔ ember-cli@5.0.0 ✔ ember-this-fallback@0.4.0 (1 deprecation-name) ✔ ember-this-fallback@0.4.0 (2 themes) ✔ ember-this-fallback@0.4.0 (3 exclude-strict-mode) ✔ virtual-dom@2.1.1 ✔ Done in 87.51s. Done in 164.47s. yarn cache v1.22.19 success Cleared cache. Done in 14.71s. I, [2024-02-01T05:23:59.999517 #1] INFO -- : > cd /var/www/discourse && su discourse -c 'bundle config --local deployment true' I, [2024-02-01T05:24:00.428628 #1] INFO -- : I, [2024-02-01T05:24:00.429260 #1] INFO -- : > cd /var/www/discourse && su discourse -c 'bundle config --local without "development test"' I, [2024-02-01T05:24:00.632752 #1] INFO -- : You are replacing the current local value of without, which is currently "test:development" I, [2024-02-01T05:24:00.633369 #1] INFO -- : > cd /var/www/discourse && su discourse -c 'bundle install --retry 3 --jobs 4' I, [2024-02-01T05:24:51.790059 #1] INFO -- : Fetching gem metadata from https://rubygems.org/......... Fetching concurrent-ruby 1.2.3 Fetching minitest 5.21.2 Fetching bigdecimal 3.1.6 Fetching cbor 0.5.9.8 Installing minitest 5.21.2 Installing bigdecimal 3.1.6 with native extensions Installing cbor 0.5.9.8 with native extensions Installing concurrent-ruby 1.2.3 Fetching excon 0.109.0 Installing excon 0.109.0 Fetching fastimage 2.3.0 Installing fastimage 2.3.0 Fetching highline 3.0.1 Fetching google-protobuf 3.25.2 (x86_64-linux) Installing highline 3.0.1 Fetching image_size 3.4.0 Installing image_size 3.4.0 Installing google-protobuf 3.25.2 (x86_64-linux) Fetching regexp_parser 2.9.0 Installing regexp_parser 2.9.0 Fetching logster 2.16.0 Fetching stackprof 0.2.26 Installing stackprof 0.2.26 with native extensions Installing logster 2.16.0 Fetching nokogiri 1.16.0 (x86_64-linux) Installing nokogiri 1.16.0 (x86_64-linux) Fetching rack-protection 3.2.0 Installing rack-protection 3.2.0 Fetching bootsnap 1.17.1 Installing bootsnap 1.17.1 with native extensions Fetching net-http 0.4.1 Installing net-http 0.4.1 Fetching sass-embedded 1.70.0 (x86_64-linux-gnu) Installing sass-embedded 1.70.0 (x86_64-linux-gnu) Fetching puma 6.4.2 Installing puma 6.4.2 with native extensions Fetching rbtrace 0.5.1 Installing rbtrace 0.5.1 with native extensions Fetching net-imap 0.4.9.1 Installing net-imap 0.4.9.1 Fetching net-smtp 0.4.0.1 Installing net-smtp 0.4.0.1 Fetching faraday-net_http 3.1.0 Installing faraday-net_http 3.1.0 Fetching activesupport 7.0.8 Installing activesupport 7.0.8 Fetching tzinfo-data 1.2023.4 Installing tzinfo-data 1.2023.4 Fetching sassc-embedded 1.70.0 Installing sassc-embedded 1.70.0 Fetching faraday 2.9.0 Installing faraday 2.9.0 Fetching activemodel 7.0.8 Installing activemodel 7.0.8 Fetching actionview 7.0.8 Installing actionview 7.0.8 Fetching activejob 7.0.8 Installing activejob 7.0.8 Fetching activerecord 7.0.8 Installing activerecord 7.0.8 Fetching actionpack 7.0.8 Installing actionpack 7.0.8 Fetching actionmailer 7.0.8 Installing actionmailer 7.0.8 Fetching railties 7.0.8 Installing railties 7.0.8 Bundle complete! 138 Gemfile dependencies, 170 gems now installed. Gems in the groups 'development' and 'test' were not installed. Bundled gems are installed into `./vendor/bundle` I, [2024-02-01T05:24:51.794263 #1] INFO -- : > cd /var/www/discourse && su discourse -c 'LOAD_PLUGINS=0 bundle exec rake plugin:pull_compatible_all' I, [2024-02-01T05:24:57.793104 #1] INFO -- : discourse-adplugin is already at latest compatible version discourse-bbcode-color is already at latest compatible version discourse-data-explorer is already at latest compatible version discourse-policy is already at latest compatible version discourse-solved is already at latest compatible version discourse-suppress-category-from-latest is already at latest compatible version discourse-voting is already at latest compatible version docker_manager is already at latest compatible version I, [2024-02-01T05:24:57.796053 #1] INFO -- : > cd /var/www/discourse && su discourse -c 'bundle exec rake db:migrate' Plugin name is 'discourse-topic-voting', but plugin directory is named 'discourse-voting' I, [2024-02-01T05:25:07.163248 #1] INFO -- : I, [2024-02-01T05:25:07.166699 #1] INFO -- : > cd /var/www/discourse && su discourse -c 'bundle exec rake themes:update assets:precompile' Plugin name is 'discourse-topic-voting', but plugin directory is named 'discourse-voting' Node.js heap_size_limit (487.0) is less than 2048MB. Setting --max-old-space-size=2048. 110:M 01 Feb 2024 05:25:23.049 * 100 changes in 300 seconds. Saving... 110:M 01 Feb 2024 05:25:23.053 * Background saving started by pid 3562 3562:C 01 Feb 2024 05:25:23.932 * DB saved on disk 3562:C 01 Feb 2024 05:25:23.933 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 0 MB 110:M 01 Feb 2024 05:25:23.955 * Background saving terminated with success 110:M 01 Feb 2024 06:20:24.614 * Replication backlog freed after 3600 seconds without connected replicas. ``` The CPU utilization shows under 10% and it gets stuck here for hours on end. I’ve tried to reboot the instance and restart the upgrade, it gets stuck at the same point again. ---
### Author: ![Ed\_S](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/ed_s/32/134015_2.png) [@Ed\_S](https://meta.discourse.org/u/Ed_S) #### Post date: [February 1, 2024, 2:02pm UTC](https://meta.discourse.org/t/3-1-x-to-3-2-0-upgrade-hangs-fails-on-1gb-instance/293743/2 "2024-02-01T14:02:13Z")
There’s a chance you have not enough RAM, causing lots of swapping, which means the CPU won’t be pegged because the machine is always paging things in and out. You can measure things from the command line: ```plaintext cat /etc/lsb-release df -h / free vmstat 5 5 ``` either with the machine in a quiet state, or at the time that it’s stuck. (Connect from a second window to get a second command line. Or similar.) ---
### Author: ![RBoy](https://avatars.discourse-cdn.com/v4/letter/r/2bfe46/32.png) [@RBoy](https://meta.discourse.org/u/RBoy) #### Post date: [February 1, 2024, 5:59pm UTC](https://meta.discourse.org/t/3-1-x-to-3-2-0-upgrade-hangs-fails-on-1gb-instance/293743/3 "2024-02-01T17:59:38Z")
Thanks, on this rebuild, it’s moving really really slowly at `yarn`, the CPU’s barely being used and it doesn’t look like there’s much I/O either. I’m a little lost as why it’s crawling. ```plaintext ubuntu@ip-172-26-6-122:~$ cat /etc/lsb-release df -h / free vmstat 5 5 DISTRIB_ID=Ubuntu DISTRIB_RELEASE=22.04 DISTRIB_CODENAME=jammy DISTRIB_DESCRIPTION="Ubuntu 22.04.3 LTS" Filesystem Size Used Avail Use% Mounted on /dev/root 39G 19G 21G 47% / total used free shared buff/cache available Mem: 972524 399360 75956 14532 497208 365668 Swap: 2097148 51296 2045852 procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 0 4 51296 75956 56504 440780 951 936 7442 1197 620 1597 2 4 2 91 0 0 5 51296 69668 56632 444976 0 0 418 585 389 893 2 2 0 96 0 0 4 51296 65384 57280 448368 0 0 422 599 382 906 1 1 0 97 0 0 4 51296 74460 57564 439444 0 0 506 546 399 946 1 3 0 96 0 0 4 51296 64588 57828 444900 0 0 545 614 407 983 2 2 0 96 0 ``` ![image](https://global.discourse-cdn.com/meta/original/4X/2/a/8/2a8214c11bd829557fb1a7f3702c710acd5e4183.png) Prior to 3.2.0, the rebuilds would generally fly along and be done in about 10-20 minutes. ---
### Author: ![Ed\_S](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/ed_s/32/134015_2.png) [@Ed\_S](https://meta.discourse.org/u/Ed_S) #### Post date: [February 1, 2024, 6:20pm UTC](https://meta.discourse.org/t/3-1-x-to-3-2-0-upgrade-hangs-fails-on-1gb-instance/293743/4 "2024-02-01T18:20:51Z")
Thanks. So, no swapping, not a RAM problem. You do have constant block I/O. Could that be remote storage?? Is everything local or do you have cloud storage involved? I notice both snapd and amazon-ssm-agent have accumulated quite a bit of CPU time, even though their percentage is zero. I don’t know what’s going on there - I don’t have either of those. If I/O is really slow, it might that you are sharing a machine (and a disk) with some very hungry neighbour. Perhaps you can ask your hosting provider to move you, or check? Sorry, no good answers at present. ---
### Author: ![RBoy](https://avatars.discourse-cdn.com/v4/letter/r/2bfe46/32.png) [@RBoy](https://meta.discourse.org/u/RBoy) #### Post date: [February 1, 2024, 6:36pm UTC](https://meta.discourse.org/t/3-1-x-to-3-2-0-upgrade-hangs-fails-on-1gb-instance/293743/5 "2024-02-01T18:36:19Z")
Running on EC2/Lightsail. I thought it might an instance issue so I took a snapshot and a spun up a new instance in a new region all together and it’s still getting stuck at ```plaintext 3536:C 01 Feb 2024 18:24:10.670 * DB saved on disk 3536:C 01 Feb 2024 18:24:10.672 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB 110:M 01 Feb 2024 18:24:10.761 * Background saving terminated with success Plugin name is 'discourse-topic-voting', but plugin directory is named 'discourse-voting' Node.js heap_size_limit (487.0) is less than 2048MB. Setting --max-old-space-size=2048. 110:M 01 Feb 2024 18:29:11.092 * 100 changes in 300 seconds. Saving... 110:M 01 Feb 2024 18:29:11.509 * Background saving started by pid 3721 3721:C 01 Feb 2024 18:29:59.839 * DB saved on disk 3721:C 01 Feb 2024 18:29:59.840 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB 110:M 01 Feb 2024 18:29:59.876 * Background saving terminated with success ``` This is the point where it just “hangs”: ```plaintext ubuntu@ip-172-26-6-122:~$ cat /etc/lsb-release df -h / free vmstat 5 5 DISTRIB_ID=Ubuntu DISTRIB_RELEASE=22.04 DISTRIB_CODENAME=jammy DISTRIB_DESCRIPTION="Ubuntu 22.04.3 LTS" Filesystem Size Used Avail Use% Mounted on /dev/root 39G 19G 21G 47% / total used free shared buff/cache available Mem: 972524 674416 155656 15932 142452 143064 Swap: 2097148 201824 1895324 procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 1 5 201824 155408 3220 140844 870 860 6959 1176 601 1627 3 4 2 91 0 1 0 201824 154824 5588 169332 3 0 5503 11 365 1708 39 8 0 53 0 3 0 202336 67104 4896 159076 48 150 986 194 321 3046 96 4 0 0 0 0 6 209760 74732 296 114488 283 1738 8385 1934 499 2265 46 5 0 49 0 0 4 212576 76968 200 115316 0 569 7678 652 387 524 1 1 0 97 0 ``` and htop shows it’s stuck spinning CPU cycles on `node`, `ember` and `snapd`, as it will suddenly burst and take a 100% CPU and then sit at around \<5% for a while and then again burst and so on ![image](https://global.discourse-cdn.com/meta/original/4X/f/4/d/f4d1fdb514eb278724dc92c48a54d7777374f6db.png) ![image](https://global.discourse-cdn.com/meta/original/4X/f/f/6/ff66716ff2bb6462aa00ef6a039ba171f110681a.png) ![image](https://global.discourse-cdn.com/meta/original/4X/4/3/5/435a325160cad728ce9bc5005eb1f6f72e7c7cb5.png) ---
### Author: ![RBoy](https://avatars.discourse-cdn.com/v4/letter/r/2bfe46/32.png) [@RBoy](https://meta.discourse.org/u/RBoy) #### Post date: [February 1, 2024, 6:44pm UTC](https://meta.discourse.org/t/3-1-x-to-3-2-0-upgrade-hangs-fails-on-1gb-instance/293743/6 "2024-02-01T18:44:57Z")
I suspect it’s an issue with the discourse upgrade itself. Is there a way to “reset” the discourse upgrade and have it restart over clean? ---
### Author: ![RBoy](https://avatars.discourse-cdn.com/v4/letter/r/2bfe46/32.png) [@RBoy](https://meta.discourse.org/u/RBoy) #### Post date: [February 1, 2024, 8:16pm UTC](https://meta.discourse.org/t/3-1-x-to-3-2-0-upgrade-hangs-fails-on-1gb-instance/293743/7 "2024-02-01T20:16:49Z")
It appears to me that something about the discourse build is broken when it’s trying to precompile the assets. It’s just plain hung here with no I/O or CPU consumption. ![image](https://global.discourse-cdn.com/meta/original/4X/7/a/d/7ad13a9d405c58c5f8e9290e3a1299cb45dc230b.png) I can’t find anything on the forum search on how to reset a rebuild from the console and am out of ideas here. @JammyDodger looking for some help from the experts who may throw light on why the rebuild has hung and how to reset it. ---
### Author: ![JammyDodger](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/jammydodger/32/254611_2.png) [@JammyDodger](https://meta.discourse.org/u/JammyDodger) #### Post date: [February 1, 2024, 8:35pm UTC](https://meta.discourse.org/t/3-1-x-to-3-2-0-upgrade-hangs-fails-on-1gb-instance/293743/8 "2024-02-01T20:35:00Z")
I think every new `./launcher rebuild app` starts a fresh one, so if you’ve rebooted and tried another rebuild it should be starting from scratch each time. Could this be connected to the changes you made to the number of unicorns? ---
### Author: ![RBoy](https://avatars.discourse-cdn.com/v4/letter/r/2bfe46/32.png) [@RBoy](https://meta.discourse.org/u/RBoy) #### Post date: [February 1, 2024, 9:05pm UTC](https://meta.discourse.org/t/3-1-x-to-3-2-0-upgrade-hangs-fails-on-1gb-instance/293743/9 "2024-02-01T21:05:32Z")
> [@JammyDodger](#): > > Could this be connected to the changes you made to the number of unicorns? That’s a great question. I don’t know and that’s what I was hoping the dev team could comment on. I will try to rebuild it again using 2 unicorns’ instead of 1. ---
### Author: ![Falco](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/falco/32/179432_2.png) [@Falco](https://meta.discourse.org/u/Falco) #### Post date: [February 1, 2024, 9:49pm UTC](https://meta.discourse.org/t/3-1-x-to-3-2-0-upgrade-hangs-fails-on-1gb-instance/293743/11 "2024-02-01T21:49:41Z")
> [@Ed\_S](#): > > Could that be remote storage? Yes, as that the default storage in AWS most common instances > [@RBoy](#): > > Running on EC2/Lightsail I think he is swapping over the network, so it’s as fast as we see here. While doing the JS build step is also quite slow on DO small instance types, at least it’s local storage AFAIK. As for what to do next, looks like the only way out is moving to a more expensive instance type. ---
### Author: ![RBoy](https://avatars.discourse-cdn.com/v4/letter/r/2bfe46/32.png) [@RBoy](https://meta.discourse.org/u/RBoy) #### Post date: [February 1, 2024, 10:02pm UTC](https://meta.discourse.org/t/3-1-x-to-3-2-0-upgrade-hangs-fails-on-1gb-instance/293743/12 "2024-02-01T22:02:35Z")
I’ll try that next. What is surprising is that I’ve had this instance running for many years now. Never have an issue with upgrading right up to 3.1.5. Suddenly with 3.2.0 it’s having issues. Was there a big change in the system requirements? I’ve tried multiple instances from snapshots of 3.1.1 to .3.1.5 and none of them complete the upgrade to 3.2.0 ---
### Author: ![Falco](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/falco/32/179432_2.png) [@Falco](https://meta.discourse.org/u/Falco) #### Post date: [February 1, 2024, 10:07pm UTC](https://meta.discourse.org/t/3-1-x-to-3-2-0-upgrade-hangs-fails-on-1gb-instance/293743/13 "2024-02-01T22:07:20Z")
It’s a little weird because that server can run Discourse just fine, the whole problem is “just” building the JS files. That process got a lot heavier over the last year’s, and that even accounting multiple efforts we made to keep it running in small servers ---
### Author: ![Ed\_S](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/ed_s/32/134015_2.png) [@Ed\_S](https://meta.discourse.org/u/Ed_S) #### Post date: [February 1, 2024, 10:07pm UTC](https://meta.discourse.org/t/3-1-x-to-3-2-0-upgrade-hangs-fails-on-1gb-instance/293743/14 "2024-02-01T22:07:28Z")
Ah, OK, so with a 1G RAM instance and swapping over the network, you might be fine serving your forum day to day but upgrading - especially recently - is very RAM hungry. I recommend you take the few minutes of downtime, and temporary resize to an 8G instance, and then do your upgrade and then resize down again. I did this, and it cost me less than a dollar extra. It also ran faster and saved me a lot of stress. The important thing is that you don’t need to double or quadruple your monthly spend - you make a temporary change. ---
### Author: ![RBoy](https://avatars.discourse-cdn.com/v4/letter/r/2bfe46/32.png) [@RBoy](https://meta.discourse.org/u/RBoy) #### Post date: [February 1, 2024, 10:26pm UTC](https://meta.discourse.org/t/3-1-x-to-3-2-0-upgrade-hangs-fails-on-1gb-instance/293743/15 "2024-02-01T22:26:45Z")
> [@Ed\_S](#): > > I recommend you take the few minutes of downtime, and temporary resize to an 8G instance, and then do your upgrade and then resize down again Yeah I’m well over 20 hours of downtime now. So you took a snapshot, sized up a new instance, upgraded discourse, then took another snapshot and then sized down a new instance. Did I get that right? Edit: Can I safely restore a 3.1.1 backup with a 3.2.0 instance? Basically I keep the last working snapshot running to reduce downtime, then backup Discourse from the 3.1.1 instance and once the new 3.2.0 instance is ready restore that 3.1.1 discourse backup? ---
### Author: ![RBoy](https://avatars.discourse-cdn.com/v4/letter/r/2bfe46/32.png) [@RBoy](https://meta.discourse.org/u/RBoy) #### Post date: [February 1, 2024, 10:32pm UTC](https://meta.discourse.org/t/3-1-x-to-3-2-0-upgrade-hangs-fails-on-1gb-instance/293743/16 "2024-02-01T22:32:29Z")
> [@Ed\_S](#): > > swapping over the network When you say swapping over the network, what are you referring to? The instance is self contained, the only thing on S3 are the backups. I’ve kept all the files local. ---
### Author: ![Ed\_S](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/ed_s/32/134015_2.png) [@Ed\_S](https://meta.discourse.org/u/Ed_S) #### Post date: [February 1, 2024, 10:53pm UTC](https://meta.discourse.org/t/3-1-x-to-3-2-0-upgrade-hangs-fails-on-1gb-instance/293743/17 "2024-02-01T22:53:28Z")
> [@RBoy](#): > > When you say swapping over the network, what are you referring to? Falco reckoned that’s what EC2/Lightsail is doing - I don’t have that knowledge. > [@RBoy](#): > > Can I safely restore a 3.1.1 backup with a 3.2.0 instance? I think so. But take a second opinion! > [@RBoy](#): > > So you took a snapshot, sized up a new instance, upgraded discourse, then took another snapshot and then sized down a new instance That’s not what I did, but it _might_ be equivalent. With my hosting providers (Digital Ocean and Hetzner) I’m offered an action called “rescale” or similar, whereby my instance goes down, a few minutes pass, and then a new instance appears with the right IP address and all the right disk contents, but with a larger or smaller amount of RAM and CPUs. In both cases, the interface makes a solid distinction between reversible rescaling, and irreversible rescaling. I’ve never taken a provider-level snapshot or backup, and I’ve never used Amazon, so it might be different over there. Sorry, no direct experience! Edit: might be worth adding EC2/Lightsail into the topic title?? ---
### Author: ![RBoy](https://avatars.discourse-cdn.com/v4/letter/r/2bfe46/32.png) [@RBoy](https://meta.discourse.org/u/RBoy) #### Post date: [February 1, 2024, 11:00pm UTC](https://meta.discourse.org/t/3-1-x-to-3-2-0-upgrade-hangs-fails-on-1gb-instance/293743/18 "2024-02-01T23:00:25Z")
> [@JammyDodger](#): > > changes you made to the number of unicorns I set it back to 2 from 1, didn’t make any difference. Still hanging. > [@Falco](#): > > the whole problem is “just” building the JS files. That process got a lot heavier over the last year’s, and that even accounting multiple efforts we made to keep it running in small servers Looking at the stats, it’s not taking CPU, it’s not using all the available RAM/Swap either. I just don’t know why it’s crawling to a point where it went from taking 20 minutes to build 3.1.5 to now hours into 3.2.0 it’s just stuck after the background saving terminates. ![image](https://global.discourse-cdn.com/meta/original/4X/6/f/7/6f742507d953c13ecda8e127c54c5806028786b3.png) ![image](https://global.discourse-cdn.com/meta/original/4X/1/6/7/167dfb6f322edfc574416f2d228f992de1e68a41.png) What am I missing here Falco? What could it be doing without taking up CPU or RAM or any significant disk I/O? ---
### Author: ![RBoy](https://avatars.discourse-cdn.com/v4/letter/r/2bfe46/32.png) [@RBoy](https://meta.discourse.org/u/RBoy) #### Post date: [February 2, 2024, 12:05am UTC](https://meta.discourse.org/t/3-1-x-to-3-2-0-upgrade-hangs-fails-on-1gb-instance/293743/19 "2024-02-02T00:05:40Z")
I didn’t realize that everytime I setup a new instance from a snapshot the server would contact letsencrypt for a new certificate renewal and now it’s exceeded it’s limit for the next 1 week. Someway to force discourse to generate new certification using the workaround provided here: [https://letsencrypt.org/docs/duplicate-certificate-limit/](https://letsencrypt.org/docs/duplicate-certificate-limit/) EDIT: So I was able to get the certificates working. I copied the entire `letsencrypt` directory from `/var/discourse/shared/standalone` from the valid certification instance to the new instance. I also copied the `ssl` folder in addition to the `letsencrypt` folder - but the question is - is that required? What’s the difference between the `letsencrypt` folder and `ssl` folder? I ask because I noticed that the certificate files on the two folders are different in size and timestamp? @JammyDodger would you know why the certificate files are stored in 2 different folders? ---
### Author: ![RBoy](https://avatars.discourse-cdn.com/v4/letter/r/2bfe46/32.png) [@RBoy](https://meta.discourse.org/u/RBoy) #### Post date: [February 2, 2024, 4:18am UTC](https://meta.discourse.org/t/3-1-x-to-3-2-0-upgrade-hangs-fails-on-1gb-instance/293743/20 "2024-02-02T04:18:03Z")
@JammyDodger @Falco @Ed_S thank you all for your help in working through this. > [@Falco](#): > > the whole problem is “just” building the JS files. That process got a lot heavier over the last year’s So I was finally able to get it to build on a 2GB RAM instance. It appears that while trying to build 3.2.0 discourse it was asking for **64GB** of VM for ember. That’s looks pretty wild. ![image](https://global.discourse-cdn.com/meta/original/4X/9/7/a/97a47a6e468eed546c9d363348994c45588841e6.png) So it looks like 3.1.5 is the last version that will run on 1GB of RAM. Starting 3.2.0, the minimum appears to be 2GB. @falco @sam can we expect the hardware requirements to keep doubling to upgrade discourse? Discourse can run pretty happily on 1GB, it’s the upgrades that are killing it. Is there no way to prebuild the assets or images and keep them on GitHub and have them downloaded while upgrading to reduce the memory load? The concern is there that for smaller sites, we don’t need 2GB, 4GB, 8GB etc instances. > [@Ed\_S](#): > > > [@RBoy](#): > > > > So you took a snapshot, sized up a new instance, upgraded discourse, then took another snapshot and then sized down a new instance > > I’ve never used Amazon, so it might be different over there. Sorry, no direct experience! Ed, just FYI - my strategy does not work with Lightsail. Once you upgrade to a larger instance there’s no way to downgrade. So I can take a snapshot of the 1GB server, then spin up a new 2GB server (and upgrade discourse), but then when I take a snapshot of the 2GB server there’s no way to spin that back up on a 1GB server (because the filesystem goes from 40GB to 60GB and that can’t be reversed). Having said that it’s still not a happy situation that one needs to “upscale” the server just to upgrade the software. So basically if you’re using Lightsail, you need to get atleast a 2GB instance (bye bye 1GB) and if the 2GB instance for whatever reason (like needing 64GB VM for a small site) fails to build/upgrade discourse, you’re SOL ☹ The only solution would be likely switch over to EC2 or something like Digital Ocean where you can dynamically scale up your RAM/CPU to complete the discourse upgrade and then scale it back down after the upgrade completes. I still have a hard time fathoming that we need more resources to upgrade the software than to run the software. Finally, it would be nice if the upgrade process gave some sort of error if it needs more RAM but couldn’t get it. Failing silently and causing the entire system to slow down isn’t right (BTW, for reference, I tried `overcommit_memory` with 1 and 0 on my 1GB instance - but like I said, no errors, even the RAM/swap wasn’t maxed out at the time of hanging from the screenshots above). ---
### Author: ![sam](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/sam/32/102149_2.png) [@sam](https://meta.discourse.org/u/sam) #### Post date: [February 2, 2024, 5:53am UTC](https://meta.discourse.org/t/3-1-x-to-3-2-0-upgrade-hangs-fails-on-1gb-instance/293743/21 "2024-02-02T05:53:16Z")
At some point I don’t mind shipping a bootstrapped image: [Can Discourse ship frequent Docker images that do not need to be bootstrapped?](https://meta.discourse.org/t/can-discourse-ship-frequent-docker-images-that-do-not-need-to-be-bootstrapped/33205) Maybe if this whole series of constraints was simplified: - You run a pg container - You run a redis container - You run a pre-bootstrapped container, with zero support for custom plugins, we would just bundle with official This could work and be handy for people, the downside is that we would diverge our install story which effectively doubles our support for free install Running a cache for compiled assets, I am not sure we can pull that off it is going to be too complicated with too many moving parts. [Next page](https://meta.discourse.org/t/3-1-x-to-3-2-0-upgrade-hangs-fails-on-1gb-instance/293743.md?page=2)