Facing issue while setting up discourse on my aws ec2 instance ubuntu20, ebs 30gb ram 1gb

2024-01-14 22:15:13.479 UTC [4179] LOG:  using stale statistics instead of current ones because stats collector is not responding
node:events:492
      throw er; // Unhandled 'error' event
      ^

Error: write EPIPE
    at afterWriteDispatched (node:internal/stream_base_commons:160:15)
    at writeGeneric (node:internal/stream_base_commons:151:3)
    at Socket._writeGeneric (node:net:931:11)
    at Socket._write (node:net:943:8)
    at writeOrBuffer (node:internal/streams/writable:392:12)
    at _write (node:internal/streams/writable:333:10)
    at Writable.write (node:internal/streams/writable:337:10)
    at DestroyableTransform.ondata (/var/www/discourse/app/assets/javascripts/node_modules/readable-stream/lib/_stream_readable.js:629:20)
    at DestroyableTransform.emit (node:events:514:28)
    at addChunk (/var/www/discourse/app/assets/javascripts/node_modules/readable-stream/lib/_stream_readable.js:279:12)
    at readableAddChunk (/var/www/discourse/app/assets/javascripts/node_modules/readable-stream/lib/_stream_readable.js:262:11)
    at Readable.push (/var/www/discourse/app/assets/javascripts/node_modules/readable-stream/lib/_stream_readable.js:228:10)
    at Transform.push (/var/www/discourse/app/assets/javascripts/node_modules/readable-stream/lib/_stream_transform.js:132:32)
    at DestroyableTransform.afterTransform (/var/www/discourse/app/assets/javascripts/node_modules/readable-stream/lib/_stream_transform.js:85:10)
    at DestroyableTransform._transform (/var/www/discourse/app/assets/javascripts/node_modules/console-ui/lib/index.js:46:7)
    at Transform._read (/var/www/discourse/app/assets/javascripts/node_modules/readable-stream/lib/_stream_transform.js:166:10)
    at Transform._write (/var/www/discourse/app/assets/javascripts/node_modules/readable-stream/lib/_stream_transform.js:155:83)
    at doWrite (/var/www/discourse/app/assets/javascripts/node_modules/readable-stream/lib/_stream_writable.js:390:139)
    at writeOrBuffer (/var/www/discourse/app/assets/javascripts/node_modules/readable-stream/lib/_stream_writable.js:381:5)
    at Writable.write (/var/www/discourse/app/assets/javascripts/node_modules/readable-stream/lib/_stream_writable.js:302:11)
    at UI.write (/var/www/discourse/app/assets/javascripts/node_modules/console-ui/lib/index.js:73:25)
    at UI.writeLine (/var/www/discourse/app/assets/javascripts/node_modules/console-ui/lib/index.js:85:10)
    at UI.writeInfoLine (/var/www/discourse/app/assets/javascripts/node_modules/console-ui/lib/index.js:111:10)
    at Class._handleProgress (/var/www/discourse/app/assets/javascripts/ember-cli-progress-ci/index.js:45:23)
    at listOnTimeout (node:internal/timers:569:17)
    at process.processTimers (node:internal/timers:512:7)
Emitted 'error' event on Socket instance at:
    at emitErrorNT (node:internal/streams/destroy:151:8)
    at emitErrorCloseNT (node:internal/streams/destroy:116:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  errno: -32,
  code: 'EPIPE',
  syscall: 'write'
}

Node.js v18.17.1
error Command failed with exit code 1.
I, [2024-01-15T01:48:50.161803 #1]  INFO -- : Terminating async processes
I, [2024-01-15T01:48:50.890858 #1]  INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main pid: 42
I, [2024-01-15T01:48:51.051009 #1]  INFO -- : Sending TERM to exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf pid: 111
2024-01-15 01:48:50.991 UTC [42] LOG:  received fast shutdown request
1112024-01-15 01:48:51.261 UTC [42] LOG:  aborting any active transactions
:signal-handler (1705283331) Received SIGTERM scheduling shutdown...
111:M 15 Jan 2024 01:48:51.320 # User requested shutdown...
111:M 15 Jan 2024 01:48:51.490 * Saving the final RDB snapshot before exiting.
2024-01-15 01:48:52.650 UTC [42] LOG:  background worker "logical replication launcher" (PID 51) exited with exit code 1
2024-01-15 01:48:53.030 UTC [46] LOG:  shutting down
111:M 15 Jan 2024 01:48:54.690 * DB saved on disk
111:M 15 Jan 2024 01:48:54.711 # Redis is now ready to exit, bye bye...
2024-01-15 01:48:56.320 UTC [42] LOG:  database system is shut down
bootstrap failed with exit code 1
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.
ff30776de5489aad433814a4bfbae70e11a77831ed80515bbcc78bf3f1681876
==================== END REBUILD LOG ====================
Failed to rebuild app.

Checking your domain name . . .

Connection to work.scube.co succeeded.
You should probably remove any non-standard plugins and rebuild.
Attempting to restart existing container. . .
x86_64 arch detected.

How much swap have you added? You’ll need at least 2GB.

In general I feel Discourse now needs a 4GB VPS, but your mileage may vary …

1 Like

As a general rule, if you’re posting a section of the log, you should include 100 more lines from before where you think you should start, because the actual error is almost always earlier than you might think. Please post a full log, or much more than you did.

Posting the output of

free
df -h
uname -a

will also often be useful.

2 Likes


Here is the configuration of my aws ec2 instance.

Thanks. Please post a more complete log. Some process has died, causing some other process to fail to write to it.

It’s a good idea also to look in the kernel logs for out of memory conditions
dmesg | egrep -i "memory|oom"
and if you find any please post the surrounding context
dmesg | egrep -i9 "memory|oom"

As you have lots of free disk space, adding more swap may help. You should also be sure you’ve made the kernel settings changes as recommended in MKJ’s Opinionated Discourse Deployment Configuration

nohup: ignoring input
x86_64 arch detected.
Ensuring launcher is up to date
Fetching origin
Launcher is up-to-date
2.0.20231218-0429: Pulling from discourse/base
Digest: sha256:468f70b9bb4c6d0c6c2bbb3efc1a5e12d145eae57bdb6946b7fe5558beb52dc1
Status: Image is up to date for discourse/base:2.0.20231218-0429
docker.io/discourse/base:2.0.20231218-0429
/usr/local/lib/ruby/gems/3.2.0/gems/pups-1.2.1/lib/pups.rb
/usr/local/bin/pups --stdin
I, [2024-01-15T16:58:45.145647 #1]  INFO -- : Reading from stdin
I, [2024-01-15T16:58:45.151732 #1]  INFO -- : > locale-gen $LANG && update-locale
I, [2024-01-15T16:58:45.244085 #1]  INFO -- : Generating locales (this might take a while)...
Generation complete.

I, [2024-01-15T16:58:45.244668 #1]  INFO -- : > mkdir -p /shared/postgres_run
I, [2024-01-15T16:58:45.253603 #1]  INFO -- :
I, [2024-01-15T16:58:45.256030 #1]  INFO -- : > chown postgres:postgres /shared/postgres_run
I, [2024-01-15T16:58:45.259335 #1]  INFO -- :
I, [2024-01-15T16:58:45.259782 #1]  INFO -- : > chmod 775 /shared/postgres_run
I, [2024-01-15T16:58:45.262788 #1]  INFO -- :
I, [2024-01-15T16:58:45.263212 #1]  INFO -- : > rm -fr /var/run/postgresql
I, [2024-01-15T16:58:45.266251 #1]  INFO -- :
I, [2024-01-15T16:58:45.266674 #1]  INFO -- : > ln -s /shared/postgres_run /var/run/postgresql
I, [2024-01-15T16:58:45.270079 #1]  INFO -- :
I, [2024-01-15T16:58:45.270488 #1]  INFO -- : > socat /dev/null UNIX-CONNECT:/shared/postgres_run/.s.PGSQL.5432 || exit 0 && echo postgres already running stop container ; exit 1
2024/01/15 16:58:45 socat[18] E connect(6, AF=1 "/shared/postgres_run/.s.PGSQL.5432", 36): Connection refused
I, [2024-01-15T16:58:45.302266 #1]  INFO -- :
I, [2024-01-15T16:58:45.302615 #1]  INFO -- : > rm -fr /shared/postgres_run/.s*
I, [2024-01-15T16:58:45.306183 #1]  INFO -- :
I, [2024-01-15T16:58:45.306545 #1]  INFO -- : > rm -fr /shared/postgres_run/*.pid
I, [2024-01-15T16:58:45.308910 #1]  INFO -- :
I, [2024-01-15T16:58:45.309281 #1]  INFO -- : > mkdir -p /shared/postgres_run/13-main.pg_stat_tmp
I, [2024-01-15T16:58:45.311237 #1]  INFO -- :
I, [2024-01-15T16:58:45.311704 #1]  INFO -- : > chown postgres:postgres /shared/postgres_run/13-main.pg_stat_tmp
I, [2024-01-15T16:58:45.313741 #1]  INFO -- :
I, [2024-01-15T16:58:45.318711 #1]  INFO -- : File > /etc/service/postgres/run  chmod: +x  chown:
I, [2024-01-15T16:58:45.322779 #1]  INFO -- : File > /etc/service/postgres/log/run  chmod: +x  chown:
I, [2024-01-15T16:58:45.327859 #1]  INFO -- : File > /etc/runit/3.d/99-postgres  chmod: +x  chown:
I, [2024-01-15T16:58:45.331984 #1]  INFO -- : File > /root/upgrade_postgres  chmod: +x  chown:
I, [2024-01-15T16:58:45.332465 #1]  INFO -- : > chown -R root /var/lib/postgresql/13/main
I, [2024-01-15T16:58:47.875972 #1]  INFO -- :
I, [2024-01-15T16:58:47.876672 #1]  INFO -- : > [ ! -e /shared/postgres_data ] && install -d -m 0755 -o postgres -g postgres /shared/postgres_data && sudo -E -u postgres /usr/lib/postgresql/13/bin/initdb -D /shared/postgres_data || exit 0
I, [2024-01-15T16:58:47.878721 #1]  INFO -- :
I, [2024-01-15T16:58:47.879047 #1]  INFO -- : > chown -R postgres:postgres /shared/postgres_data
I, [2024-01-15T16:58:47.921655 #1]  INFO -- :
I, [2024-01-15T16:58:47.922183 #1]  INFO -- : > chown -R postgres:postgres /var/run/postgresql
I, [2024-01-15T16:58:47.924437 #1]  INFO -- :
I, [2024-01-15T16:58:47.924893 #1]  INFO -- : > /root/upgrade_postgres
I, [2024-01-15T16:58:47.930626 #1]  INFO -- :
I, [2024-01-15T16:58:47.930957 #1]  INFO -- : > rm /root/upgrade_postgres
I, [2024-01-15T16:58:47.932764 #1]  INFO -- :
I, [2024-01-15T16:58:47.935028 #1]  INFO -- : Replacing data_directory = '/var/lib/postgresql/13/main' with data_directory = '/shared/postgres_data' in /etc/postgresql/13/main/postgresql.conf
I, [2024-01-15T16:58:47.935765 #1]  INFO -- : Replacing (?-mix:#?listen_addresses *=.*) with listen_addresses = '*' in /etc/postgresql/13/main/postgresql.conf
I, [2024-01-15T16:58:47.936755 #1]  INFO -- : Replacing (?-mix:#?synchronous_commit *=.*) with synchronous_commit = $db_synchronous_commit in /etc/postgresql/13/main/postgresql.conf
I, [2024-01-15T16:58:47.937642 #1]  INFO -- : Replacing (?-mix:#?shared_buffers *=.*) with shared_buffers = $db_shared_buffers in /etc/postgresql/13/main/postgresql.conf
I, [2024-01-15T16:58:47.938950 #1]  INFO -- : Replacing (?-mix:#?work_mem *=.*) with work_mem = $db_work_mem in /etc/postgresql/13/main/postgresql.conf
I, [2024-01-15T16:58:47.939967 #1]  INFO -- : Replacing (?-mix:#?default_text_search_config *=.*) with default_text_search_config = '$db_default_text_search_config' in /etc/postgresql/13/main/postgresql.conf
I, [2024-01-15T16:58:47.940960 #1]  INFO -- : > install -d -m 0755 -o postgres -g postgres /shared/postgres_backup
I, [2024-01-15T16:58:47.947636 #1]  INFO -- :
I, [2024-01-15T16:58:47.948187 #1]  INFO -- : Replacing (?-mix:#?checkpoint_segments *=.*) with checkpoint_segments = $db_checkpoint_segments in /etc/postgresql/13/main/postgresql.conf
I, [2024-01-15T16:58:47.948745 #1]  INFO -- : Replacing (?-mix:#?logging_collector *=.*) with logging_collector = $db_logging_collector in /etc/postgresql/13/main/postgresql.conf
I, [2024-01-15T16:58:47.949726 #1]  INFO -- : Replacing (?-mix:#?log_min_duration_statement *=.*) with log_min_duration_statement = $db_log_min_duration_statement in /etc/postgresql/13/main/postgresql.conf
I, [2024-01-15T16:58:47.951478 #1]  INFO -- : Replacing (?-mix:^#local +replication +postgres +peer$) with local replication postgres  peer in /etc/postgresql/13/main/pg_hba.conf
I, [2024-01-15T16:58:47.951996 #1]  INFO -- : Replacing (?-mix:^host.*all.*all.*127.*$) with host all all 0.0.0.0/0 md5 in /etc/postgresql/13/main/pg_hba.conf
I, [2024-01-15T16:58:47.952801 #1]  INFO -- : Replacing (?-mix:^host.*all.*all.*::1\/128.*$) with host all all ::/0 md5 in /etc/postgresql/13/main/pg_hba.conf
I, [2024-01-15T16:58:47.953640 #1]  INFO -- : > HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main
I, [2024-01-15T16:58:47.954873 #1]  INFO -- : > sleep 5
2024-01-15 16:58:48.098 UTC [41] LOG:  starting PostgreSQL 13.13 (Debian 13.13-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
2024-01-15 16:58:48.099 UTC [41] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2024-01-15 16:58:48.100 UTC [41] LOG:  listening on IPv6 address "::", port 5432
2024-01-15 16:58:48.104 UTC [41] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2024-01-15 16:58:48.113 UTC [44] LOG:  database system was interrupted; last known up at 2024-01-15 16:13:03 UTC
2024-01-15 16:58:48.150 UTC [44] LOG:  database system was not properly shut down; automatic recovery in progress
2024-01-15 16:58:48.156 UTC [44] LOG:  redo starts at 0/229CCC0
2024-01-15 16:58:48.156 UTC [44] LOG:  invalid record length at 0/229CDA8: wanted 24, got 0
2024-01-15 16:58:48.156 UTC [44] LOG:  redo done at 0/229CD70
2024-01-15 16:58:48.176 UTC [41] LOG:  database system is ready to accept connections
I, [2024-01-15T16:58:52.959339 #1]  INFO -- :
I, [2024-01-15T16:58:52.959538 #1]  INFO -- : > su postgres -c 'createdb discourse' || true
2024-01-15 16:58:53.094 UTC [54] postgres@postgres ERROR:  database "discourse" already exists
2024-01-15 16:58:53.094 UTC [54] postgres@postgres STATEMENT:  CREATE DATABASE discourse;
createdb: error: database creation failed: ERROR:  database "discourse" already exists
I, [2024-01-15T16:58:53.096988 #1]  INFO -- :
I, [2024-01-15T16:58:53.097403 #1]  INFO -- : > su postgres -c 'psql discourse -c "create user discourse;"' || true
2024-01-15 16:58:53.177 UTC [58] postgres@discourse ERROR:  role "discourse" already exists
2024-01-15 16:58:53.177 UTC [58] postgres@discourse STATEMENT:  create user discourse;
ERROR:  role "discourse" already exists
I, [2024-01-15T16:58:53.180130 #1]  INFO -- :
I, [2024-01-15T16:58:53.180536 #1]  INFO -- : > su postgres -c 'psql discourse -c "grant all privileges on database discourse to discourse;"' || true
I, [2024-01-15T16:58:53.233213 #1]  INFO -- : GRANT

I, [2024-01-15T16:58:53.233752 #1]  INFO -- : > su postgres -c 'psql discourse -c "alter schema public owner to discourse;"'
I, [2024-01-15T16:58:53.285988 #1]  INFO -- : ALTER SCHEMA

I, [2024-01-15T16:58:53.286476 #1]  INFO -- : > su postgres -c 'psql template1 -c "create extension if not exists hstore;"'
NOTICE:  extension "hstore" already exists, skipping
I, [2024-01-15T16:58:53.354779 #1]  INFO -- : CREATE EXTENSION

I, [2024-01-15T16:58:53.355178 #1]  INFO -- : > su postgres -c 'psql template1 -c "create extension if not exists pg_trgm;"'
NOTICE:  extension "pg_trgm" already exists, skipping
I, [2024-01-15T16:58:53.408978 #1]  INFO -- : CREATE EXTENSION

I, [2024-01-15T16:58:53.409387 #1]  INFO -- : > su postgres -c 'psql template1 -c "create extension if not exists vector;"'
NOTICE:  extension "vector" already exists, skipping
I, [2024-01-15T16:58:53.460682 #1]  INFO -- : CREATE EXTENSION

I, [2024-01-15T16:58:53.461138 #1]  INFO -- : > su postgres -c 'psql discourse -c "create extension if not exists hstore;"'
NOTICE:  extension "hstore" already exists, skipping
I, [2024-01-15T16:58:53.513663 #1]  INFO -- : CREATE EXTENSION

I, [2024-01-15T16:58:53.514082 #1]  INFO -- : > su postgres -c 'psql discourse -c "create extension if not exists pg_trgm;"'
NOTICE:  extension "pg_trgm" already exists, skipping
I, [2024-01-15T16:58:53.565802 #1]  INFO -- : CREATE EXTENSION

I, [2024-01-15T16:58:53.566223 #1]  INFO -- : > su postgres -c 'psql discourse -c "create extension if not exists vector;"'
NOTICE:  extension "vector" already exists, skipping
I, [2024-01-15T16:58:53.617573 #1]  INFO -- : CREATE EXTENSION

I, [2024-01-15T16:58:53.618081 #1]  INFO -- : > sudo -u postgres psql discourse
I, [2024-01-15T16:58:53.622431 #1]  INFO -- : update pg_database set encoding = pg_char_to_encoding('UTF8') where datname = 'discourse' AND encoding = pg_char_to_encoding('SQL_ASCII');

I, [2024-01-15T16:58:53.718651 #1]  INFO -- : File > /var/lib/postgresql/take-database-backup  chmod: +x  chown: postgres:postgres
I, [2024-01-15T16:58:53.723617 #1]  INFO -- : File > /var/spool/cron/crontabs/postgres  chmod:   chown:
I, [2024-01-15T16:58:53.723984 #1]  INFO -- : > echo postgres installed!
I, [2024-01-15T16:58:53.726048 #1]  INFO -- : postgres installed!

I, [2024-01-15T16:58:53.730261 #1]  INFO -- : File > /etc/service/redis/run  chmod: +x  chown:
I, [2024-01-15T16:58:53.734374 #1]  INFO -- : File > /etc/service/redis/log/run  chmod: +x  chown:
I, [2024-01-15T16:58:53.738496 #1]  INFO -- : File > /etc/runit/3.d/10-redis  chmod: +x  chown:
I, [2024-01-15T16:58:53.740712 #1]  INFO -- : Replacing daemonize yes with  in /etc/redis/redis.conf
I, [2024-01-15T16:58:53.741562 #1]  INFO -- : Replacing (?-mix:^pidfile.*$) with  in /etc/redis/redis.conf
I, [2024-01-15T16:58:53.743527 #1]  INFO -- : > install -d -m 0755 -o redis -g redis /shared/redis_data
I, [2024-01-15T16:58:53.746574 #1]  INFO -- :
I, [2024-01-15T16:58:53.747239 #1]  INFO -- : Replacing (?-mix:^logfile.*$) with logfile "" in /etc/redis/redis.conf
I, [2024-01-15T16:58:53.747969 #1]  INFO -- : Replacing (?-mix:^bind .*$) with  in /etc/redis/redis.conf
I, [2024-01-15T16:58:53.749492 #1]  INFO -- : Replacing (?-mix:^dir .*$) with dir /shared/redis_data in /etc/redis/redis.conf
I, [2024-01-15T16:58:53.750964 #1]  INFO -- : Replacing (?-mix:^protected-mode yes) with protected-mode no in /etc/redis/redis.conf
I, [2024-01-15T16:58:53.752699 #1]  INFO -- : Replacing # io-threads 4 with io-threads $redis_io_threads in /etc/redis/redis.conf
I, [2024-01-15T16:58:53.754125 #1]  INFO -- : > echo redis installed
I, [2024-01-15T16:58:53.756373 #1]  INFO -- : redis installed

I, [2024-01-15T16:58:53.756856 #1]  INFO -- : > cat /etc/redis/redis.conf | grep logfile
I, [2024-01-15T16:58:53.764756 #1]  INFO -- : logfile ""

I, [2024-01-15T16:58:53.765199 #1]  INFO -- : > exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf
I, [2024-01-15T16:58:53.766731 #1]  INFO -- : > sleep 10
110:C 15 Jan 2024 16:58:53.787 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
110:C 15 Jan 2024 16:58:53.788 # Redis version=7.0.7, bits=64, commit=00000000, modified=0, pid=110, just started
110:C 15 Jan 2024 16:58:53.788 # Configuration loaded
110:M 15 Jan 2024 16:58:53.788 * monotonic clock: POSIX clock_gettime
110:M 15 Jan 2024 16:58:53.792 * Running mode=standalone, port=6379.
110:M 15 Jan 2024 16:58:53.792 # Server initialized
110:M 15 Jan 2024 16:58:53.792 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
110:M 15 Jan 2024 16:58:53.793 # WARNING Your system is configured to use the 'xen' clocksource which might lead to degraded performance. Check the result of the [slow-clocksource] system check: run 'redis-server --check-system' to check if the system's clocksource isn't degrading performance.
110:M 15 Jan 2024 16:58:53.795 * Loading RDB produced by version 7.0.7
110:M 15 Jan 2024 16:58:53.795 * RDB age 2682 seconds
110:M 15 Jan 2024 16:58:53.795 * RDB memory usage when created 1.44 Mb
110:M 15 Jan 2024 16:58:53.796 * Done loading RDB, keys loaded: 34, keys expired: 4.
110:M 15 Jan 2024 16:58:53.797 * DB loaded from disk: 0.002 seconds
110:M 15 Jan 2024 16:58:53.797 * Ready to accept connections
I, [2024-01-15T16:59:03.770236 #1]  INFO -- :
I, [2024-01-15T16:59:03.770839 #1]  INFO -- : > thpoff echo "thpoff is installed!"
I, [2024-01-15T16:59:03.774891 #1]  INFO -- : thpoff is installed!

I, [2024-01-15T16:59:03.775308 #1]  INFO -- : > /usr/local/bin/ruby -e 'if ENV["DISCOURSE_SMTP_ADDRESS"] == "smtp.example.com"; puts "Aborting! Mail is not configured!"; exit 1; end'
I, [2024-01-15T16:59:03.857644 #1]  INFO -- :
I, [2024-01-15T16:59:03.858276 #1]  INFO -- : > /usr/local/bin/ruby -e 'if ENV["DISCOURSE_HOSTNAME"] == "discourse.example.com"; puts "Aborting! Domain is not configured!"; exit 1; end'
I, [2024-01-15T16:59:03.939898 #1]  INFO -- :
I, [2024-01-15T16:59:03.940441 #1]  INFO -- : > /usr/local/bin/ruby -e 'if (ENV["DISCOURSE_CDN_URL"] || "")[0..1] == "//"; puts "Aborting! CDN must have a protocol specified. Once fixed you should rebake your posts now to correct all posts."; exit 1; end'
I, [2024-01-15T16:59:04.017626 #1]  INFO -- :
I, [2024-01-15T16:59:04.018169 #1]  INFO -- : > rm -f /etc/cron.d/anacron
I, [2024-01-15T16:59:04.022098 #1]  INFO -- :
I, [2024-01-15T16:59:04.026330 #1]  INFO -- : File > /etc/cron.d/anacron  chmod:   chown:
I, [2024-01-15T16:59:04.031770 #1]  INFO -- : File > /etc/runit/1.d/copy-env  chmod: +x  chown:
I, [2024-01-15T16:59:04.035944 #1]  INFO -- : File > /etc/service/unicorn/run  chmod: +x  chown:
I, [2024-01-15T16:59:04.040176 #1]  INFO -- : File > /etc/service/nginx/run  chmod: +x  chown:
I, [2024-01-15T16:59:04.044329 #1]  INFO -- : File > /etc/runit/3.d/01-nginx  chmod: +x  chown:
I, [2024-01-15T16:59:04.048448 #1]  INFO -- : File > /etc/runit/3.d/02-unicorn  chmod: +x  chown:
I, [2024-01-15T16:59:04.048848 #1]  INFO -- : Replacing # postgres with sv start postgres || exit 1 in /etc/service/unicorn/run
I, [2024-01-15T16:59:04.049309 #1]  INFO -- : > exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf
I, [2024-01-15T16:59:04.051343 #1]  INFO -- : > cd /var/www/discourse && sudo -H -E -u discourse git reset --hard
137:C 15 Jan 2024 16:59:04.063 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
137:C 15 Jan 2024 16:59:04.063 # Redis version=7.0.7, bits=64, commit=00000000, modified=0, pid=137, just started
137:C 15 Jan 2024 16:59:04.063 # Configuration loaded
137:M 15 Jan 2024 16:59:04.064 * monotonic clock: POSIX clock_gettime
137:M 15 Jan 2024 16:59:04.064 # Warning: Could not create server TCP listening socket *:6379: bind: Address already in use
137:M 15 Jan 2024 16:59:04.065 # Failed listening on port 6379 (TCP), aborting.
Updating files: 100% (33715/33715), done.
I, [2024-01-15T16:59:10.374264 #1]  INFO -- : HEAD is now at 6ab1a19e9 DEV: Convert min_trust_level_to_allow_invite to groups (#24893)

I, [2024-01-15T16:59:10.381564 #1]  INFO -- : > cd /var/www/discourse && sudo -H -E -u discourse git clean -f
I, [2024-01-15T16:59:10.848679 #1]  INFO -- :
I, [2024-01-15T16:59:10.849103 #1]  INFO -- : > cd /var/www/discourse && sudo -H -E -u discourse bash -c '
  set -o errexit
  if [ $(git rev-parse --is-shallow-repository) == "true" ]; then
      git remote set-branches --add origin main
      git remote set-branches origin tests-passed
      git fetch --depth 1 origin tests-passed
  else
      git fetch --tags --prune-tags --prune --force origin
  fi
'
From https://github.com/discourse/discourse
 - [deleted]             (none)     -> origin/0-drop-code
 - [deleted]             (none)     -> origin/bulk-button-class
 - [deleted]             (none)     -> origin/dependabot/bundler/excon-0.108.0
 - [deleted]             (none)     -> origin/dependabot/bundler/hashdiff-1.1.0
 - [deleted]             (none)     -> origin/dependabot/bundler/parallel-1.24.0
 - [deleted]             (none)     -> origin/dependabot/bundler/rubocop-discourse-3.6.0
 - [deleted]             (none)     -> origin/dependabot/npm_and_yarn/app/assets/javascripts/uppy/drop-target-2.0.2
 - [deleted]             (none)     -> origin/dev/add-new-chat-metrics
 - [deleted]             (none)     -> origin/dev/allow-chat-mentions-to-have-several-notifications
 - [deleted]             (none)     -> origin/dev/convert-min-trust-level-to-create-tag-to-group
 - [deleted]             (none)     -> origin/dev/convert-min-trust-level-to-ignore-to-groups
 - [deleted]             (none)     -> origin/fix-join-filter-upload
 - [deleted]             (none)     -> origin/fix-modal-flash-minor
 - [deleted]             (none)     -> origin/global-notice-reactive
 - [deleted]             (none)     -> origin/low-ram-help
 - [deleted]             (none)     -> origin/multi-invite
 - [deleted]             (none)     -> origin/table-builder-modal-modifier
 - [deleted]             (none)     -> origin/wip_categories
 - [deleted]             (none)     -> origin/wizard-split
   6ab1a19e9..f2cf5434f  main                  -> origin/main
 * [new branch]          0-extend-native       -> origin/0-extend-native
 * [new branch]          before-header-logo-outlet -> origin/before-header-logo-outlet
 * [new branch]          chat-nav-back-btn-route-change -> origin/chat-nav-back-btn-route-change
 * [new branch]          convert-header-to-glimmer -> origin/convert-header-to-glimmer
 * [new branch]          dependabot/bundler/omniauth-2.0.0 -> origin/dependabot/bundler/omniauth-2.0.0
 * [new branch]          dependabot/bundler/parallel_tests-4.4.0 -> origin/dependabot/bundler/parallel_tests-4.4.0
 * [new branch]          dev-system-tests-for-topic-map -> origin/dev-system-tests-for-topic-map
 * [new branch]          dev/change-min-trust-post-embedded-media-to-group -> origin/dev/change-min-trust-post-embedded-media-to-group
 * [new branch]          dev/drop-the-user-id-column-from-chat-mentions -> origin/dev/drop-the-user-id-column-from-chat-mentions
 + e0c0191bb...feb00e04c dev/dry-up-user-status-serializers -> origin/dev/dry-up-user-status-serializers  (forced update)
 * [new branch]          dev/proof-of-concept-rewriting-mentions-notification-logic -> origin/dev/proof-of-concept-rewriting-mentions-notification-logic
 + 7c338d0dc...a06deea9a dev/rethink-mass-mentions-polymorphic -> origin/dev/rethink-mass-mentions-polymorphic  (forced update)
 * [new branch]          dev_categories_index  -> origin/dev_categories_index
 * [new branch]          dev_category_order_database -> origin/dev_category_order_database
 * [new branch]          feature/custom-date-range -> origin/feature/custom-date-range
 * [new branch]          feature/paste-upload-on-first-upload-input-templates -> origin/feature/paste-upload-on-first-upload-input-templates
 * [new branch]          filter-for-more-for-icons -> origin/filter-for-more-for-icons
 * [new branch]          fix-multi-trigger-app-event -> origin/fix-multi-trigger-app-event
 * [new branch]          fix_category_chooser  -> origin/fix_category_chooser
 * [new branch]          fix_reviewable_translation -> origin/fix_reviewable_translation
   7f2bdca65..67d165c2a  generic-import-latest -> origin/generic-import-latest
 * [new branch]          hijacks               -> origin/hijacks
 * [new branch]          loic-docker-dev-linux-aarch64 -> origin/loic-docker-dev-linux-aarch64
 + 90a3535b7...54d0135bc migrations-convert    -> origin/migrations-convert  (forced update)
 * [new branch]          mobile-chat-footer-redesign -> origin/mobile-chat-footer-redesign
 * [new branch]          optimized_image_perf  -> origin/optimized_image_perf
 * [new branch]          revert-25009-dev/convert-min-trust-to-allow-self-wiki-to-groups -> origin/revert-25009-dev/convert-min-trust-to-allow-self-wiki-to-groups
 * [new branch]          sidebar-above-outlet  -> origin/sidebar-above-outlet
 * [new branch]          skip_validations      -> origin/skip_validations
   d7f7f79b6..75772910c  stable                -> origin/stable
   6ab1a19e9..f2cf5434f  tests-passed          -> origin/tests-passed
 * [new branch]          topic-bulk-actions    -> origin/topic-bulk-actions
 * [new branch]          unhide_lazy_load_categories -> origin/unhide_lazy_load_categories
 t [tag update]          beta                  -> beta
 t [tag update]          latest-release        -> latest-release
 * [new tag]             v3.1.4                -> v3.1.4
 * [new tag]             v3.2.0.beta4          -> v3.2.0.beta4
I, [2024-01-15T16:59:12.366432 #1]  INFO -- :
I, [2024-01-15T16:59:12.366756 #1]  INFO -- : > cd /var/www/discourse && sudo -H -E -u discourse bash -c '
  set -o errexit
  if [[ $(git symbolic-ref --short HEAD) == tests-passed ]] ; then
      git pull
  else
      git -c advice.detachedHead=false checkout tests-passed
  fi
'
Switched to a new branch 'tests-passed'
I, [2024-01-15T16:59:16.214547 #1]  INFO -- : Branch 'tests-passed' set up to track remote branch 'tests-passed' from 'origin'.

I, [2024-01-15T16:59:16.214902 #1]  INFO -- : > cd /var/www/discourse && sudo -H -E -u discourse git config user.discourse-version tests-passed
I, [2024-01-15T16:59:16.225245 #1]  INFO -- :
I, [2024-01-15T16:59:16.225564 #1]  INFO -- : > cd /var/www/discourse && mkdir -p tmp
I, [2024-01-15T16:59:16.230108 #1]  INFO -- :
I, [2024-01-15T16:59:16.230384 #1]  INFO -- : > cd /var/www/discourse && chown discourse:www-data tmp
I, [2024-01-15T16:59:16.234268 #1]  INFO -- :
I, [2024-01-15T16:59:16.234547 #1]  INFO -- : > cd /var/www/discourse && mkdir -p tmp/pids
I, [2024-01-15T16:59:16.237339 #1]  INFO -- :
I, [2024-01-15T16:59:16.237630 #1]  INFO -- : > cd /var/www/discourse && mkdir -p tmp/sockets
I, [2024-01-15T16:59:16.240263 #1]  INFO -- :
I, [2024-01-15T16:59:16.240535 #1]  INFO -- : > cd /var/www/discourse && touch tmp/.gitkeep
I, [2024-01-15T16:59:16.245592 #1]  INFO -- :
I, [2024-01-15T16:59:16.245872 #1]  INFO -- : > cd /var/www/discourse && mkdir -p                    /shared/log/rails
I, [2024-01-15T16:59:16.248992 #1]  INFO -- :
I, [2024-01-15T16:59:16.249276 #1]  INFO -- : > cd /var/www/discourse && bash -c "touch -a           /shared/log/rails/{production,production_errors,unicorn.stdout,unicorn.stderr,sidekiq}.log"
I, [2024-01-15T16:59:16.253882 #1]  INFO -- :
I, [2024-01-15T16:59:16.254163 #1]  INFO -- : > cd /var/www/discourse && bash -c "ln    -s           /shared/log/rails/{production,production_errors,unicorn.stdout,unicorn.stderr,sidekiq}.log /var/www/discourse/log"
I, [2024-01-15T16:59:16.260194 #1]  INFO -- :
I, [2024-01-15T16:59:16.260478 #1]  INFO -- : > cd /var/www/discourse && bash -c "mkdir -p           /shared/{uploads,backups}"
I, [2024-01-15T16:59:16.264924 #1]  INFO -- :
I, [2024-01-15T16:59:16.265217 #1]  INFO -- : > cd /var/www/discourse && bash -c "ln    -s           /shared/{uploads,backups} /var/www/discourse/public"
I, [2024-01-15T16:59:16.269226 #1]  INFO -- :
I, [2024-01-15T16:59:16.269501 #1]  INFO -- : > cd /var/www/discourse && bash -c "mkdir -p           /shared/tmp/{backups,restores}"
I, [2024-01-15T16:59:16.274008 #1]  INFO -- :
I, [2024-01-15T16:59:16.274284 #1]  INFO -- : > cd /var/www/discourse && bash -c "ln    -s           /shared/tmp/{backups,restores} /var/www/discourse/tmp"
I, [2024-01-15T16:59:16.278114 #1]  INFO -- :
I, [2024-01-15T16:59:16.278515 #1]  INFO -- : > cd /var/www/discourse && chown -R discourse:www-data /shared/log/rails /shared/uploads /shared/backups /shared/tmp
I, [2024-01-15T16:59:16.284814 #1]  INFO -- :
I, [2024-01-15T16:59:16.285089 #1]  INFO -- : > cd /var/www/discourse && [ ! -d public/plugins ] || find public/plugins/ -maxdepth 1 -xtype l -delete
I, [2024-01-15T16:59:16.286740 #1]  INFO -- :
I, [2024-01-15T16:59:16.289312 #1]  INFO -- : Replacing # redis with sv start redis || exit 1 in /etc/service/unicorn/run
I, [2024-01-15T16:59:16.290919 #1]  INFO -- : > cd /var/www/discourse/plugins && git clone https://github.com/discourse/docker_manager.git
Cloning into 'docker_manager'...
I, [2024-01-15T16:59:17.503185 #1]  INFO -- :
I, [2024-01-15T16:59:17.503803 #1]  INFO -- : > cp /var/www/discourse/config/nginx.sample.conf /etc/nginx/conf.d/discourse.conf
I, [2024-01-15T16:59:17.513581 #1]  INFO -- :
I, [2024-01-15T16:59:17.514119 #1]  INFO -- : > rm /etc/nginx/sites-enabled/default
I, [2024-01-15T16:59:17.518145 #1]  INFO -- :
I, [2024-01-15T16:59:17.518570 #1]  INFO -- : > mkdir -p /var/nginx/cache
I, [2024-01-15T16:59:17.525964 #1]  INFO -- :
I, [2024-01-15T16:59:17.527014 #1]  INFO -- : Replacing pid /run/nginx.pid; with daemon off; in /etc/nginx/nginx.conf
I, [2024-01-15T16:59:17.531014 #1]  INFO -- : Replacing (?m-ix:upstream[^\}]+\}) with upstream discourse { server 127.0.0.1:3000; } in /etc/nginx/conf.d/discourse.conf
I, [2024-01-15T16:59:17.531552 #1]  INFO -- : Replacing (?-mix:server_name.+$) with server_name _ ; in /etc/nginx/conf.d/discourse.conf
I, [2024-01-15T16:59:17.532319 #1]  INFO -- : Replacing (?-mix:client_max_body_size.+$) with client_max_body_size $upload_size ; in /etc/nginx/conf.d/discourse.conf
I, [2024-01-15T16:59:17.533220 #1]  INFO -- : Replacing (?-mix:worker_connections.+$) with worker_connections $nginx_worker_connections ; in /etc/nginx/nginx.conf
I, [2024-01-15T16:59:17.533678 #1]  INFO -- : > echo "done configuring web"
I, [2024-01-15T16:59:17.535435 #1]  INFO -- : done configuring web

I, [2024-01-15T16:59:17.535949 #1]  INFO -- : > cd /var/www/discourse && gem install bundler --conservative -v $(awk '/BUNDLED WITH/ { getline; gsub(/ /,""); print $0 }' Gemfile.lock)
I, [2024-01-15T16:59:18.589996 #1]  INFO -- : Successfully installed bundler-2.5.3
1 gem installed

I, [2024-01-15T16:59:18.590473 #1]  INFO -- : > cd /var/www/discourse && find /var/www/discourse ! -user discourse -exec chown discourse {} \+
I, [2024-01-15T16:59:27.811504 #1]  INFO -- :
I, [2024-01-15T16:59:27.818976 #1]  INFO -- : > cd /var/www/discourse && if [ "tests-passed" != "tests-passed" ]; then
  rm -rf app/assets/javascripts/node_modules
fi
I, [2024-01-15T16:59:27.828783 #1]  INFO -- :
I, [2024-01-15T16:59:27.829120 #1]  INFO -- : > cd /var/www/discourse && su discourse -c 'yarn install --frozen-lockfile && yarn cache clean'
warning " > @glint/environment-ember-loose@1.1.0" has unmet peer dependency "@glimmer/component@^1.1.2".
warning " > @glint/environment-ember-template-imports@1.1.0" has unmet peer dependency "ember-template-imports@^3.0.0".
warning Resolution field "unset-value@2.0.1" is incompatible with requested version "unset-value@^1.0.0"
warning Pattern ["wrap-ansi@^7.0.0"] is trying to unpack in the same destination "/home/discourse/.cache/yarn/v6/npm-wrap-ansi-cjs-7.0.0-67e145cff510a6a6984bdf1152911d69d2eb9e43-integrity/node_modules/wrap-ansi-cjs" as pattern ["wrap-ansi-cjs@npm:wrap-ansi@^7.0.0"]. This could result in non-deterministic behavior, skipping.
warning " > discourse-markdown-it@1.0.0" has unmet peer dependency "xss@*".
warning "workspace-aggregator-47d0f784-cc37-49fb-a3de-47f157d26695 > discourse > @uppy/aws-s3@3.0.6" has incorrect peer dependency "@uppy/core@^3.1.2".
warning "workspace-aggregator-47d0f784-cc37-49fb-a3de-47f157d26695 > discourse > @uppy/aws-s3-multipart@3.1.3" has incorrect peer dependency "@uppy/core@^3.1.2".
warning "workspace-aggregator-47d0f784-cc37-49fb-a3de-47f157d26695 > discourse > @uppy/xhr-upload@3.1.1" has incorrect peer dependency "@uppy/core@^3.1.2".
warning "workspace-aggregator-47d0f784-cc37-49fb-a3de-47f157d26695 > discourse > ember-router-service-refresh-polyfill@1.1.0" has incorrect peer dependency "ember-source@^3.4 || 4".
warning "workspace-aggregator-47d0f784-cc37-49fb-a3de-47f157d26695 > discourse-plugins > ember-this-fallback@0.4.0" has unmet peer dependency "ember-source@^3.28.11 || ^4.0.0".
warning "workspace-aggregator-47d0f784-cc37-49fb-a3de-47f157d26695 > admin > ember-source > router_js@8.0.3" has unmet peer dependency "rsvp@^4.8.5".
warning "workspace-aggregator-47d0f784-cc37-49fb-a3de-47f157d26695 > discourse > @uppy/aws-s3 > @uppy/xhr-upload@3.3.0" has incorrect peer dependency "@uppy/core@^3.2.1".
I, [2024-01-15T17:39:47.335410 #1]  INFO -- : yarn install v1.22.19
[1/5] Validating package.json...
[2/5] Resolving packages...
[3/5] Fetching packages...
[4/5] Linking dependencies...
[5/5] Building fresh packages...
$ yarn --cwd app/assets/javascripts $(node -e 'const argv = JSON.parse(process.env.npm_config_argv).original; const passthrough = [`--frozen-lockfile`, `-s`].filter(arg => argv.includes(arg)); console.log(passthrough.join(` `));')
yarn install v1.22.19
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
$ ./run-patch-package
patch-package 8.0.0
Applying patches...
@ember/test-helpers@2.9.4 ✔
babel-plugin-debug-macros@0.3.4 ✔
content-tag@1.2.2 ✔
ember-cli@5.0.0 ✔
ember-this-fallback@0.4.0 (1 deprecation-name) ✔
ember-this-fallback@0.4.0 (2 themes) ✔
ember-this-fallback@0.4.0 (3 exclude-strict-mode) ✔
virtual-dom@2.1.1 ✔
Done in 1231.04s.
Done in 2214.95s.
yarn cache v1.22.19
success Cleared cache.
Done in 200.98s.

I, [2024-01-15T17:39:49.745470 #1]  INFO -- : > cd /var/www/discourse && su discourse -c 'bundle config --local deployment true'
I, [2024-01-15T17:39:53.840510 #1]  INFO -- :
I, [2024-01-15T17:39:53.840965 #1]  INFO -- : > cd /var/www/discourse && su discourse -c 'bundle config --local without "development test"'
I, [2024-01-15T17:39:54.034785 #1]  INFO -- : You are replacing the current local value of without, which is currently "test:development"

I, [2024-01-15T17:39:54.035233 #1]  INFO -- : > cd /var/www/discourse && su discourse -c 'bundle install --retry 3 --jobs 4'
I, [2024-01-15T17:40:42.790849 #1]  INFO -- : Fetching gem metadata from https://rubygems.org/.........
Fetching abbrev 0.1.2
Fetching minitest 5.21.1
Fetching cbor 0.5.9.8
Fetching excon 0.109.0
Installing abbrev 0.1.2
Installing minitest 5.21.1
Installing cbor 0.5.9.8 with native extensions
Installing excon 0.109.0
Fetching fastimage 2.3.0
Installing fastimage 2.3.0
Fetching google-protobuf 3.25.2 (x86_64-linux)
Fetching regexp_parser 2.9.0
Installing regexp_parser 2.9.0
Fetching nokogiri 1.16.0 (x86_64-linux)
Fetching rack-protection 3.2.0
Installing google-protobuf 3.25.2 (x86_64-linux)
Installing rack-protection 3.2.0
Fetching bootsnap 1.17.1
Installing bootsnap 1.17.1 with native extensions
Fetching highline 3.0.0
Installing highline 3.0.0
Fetching net-http 0.4.1
Installing net-http 0.4.1
Fetching puma 6.4.2
Installing puma 6.4.2 with native extensions
Installing nokogiri 1.16.0 (x86_64-linux)
Fetching rbtrace 0.5.1
Installing rbtrace 0.5.1 with native extensions
Fetching activesupport 7.0.8
Installing activesupport 7.0.8
Fetching tzinfo-data 1.2023.4
Installing tzinfo-data 1.2023.4
Fetching net-imap 0.4.9.1
Installing net-imap 0.4.9.1
Fetching net-smtp 0.4.0.1
Installing net-smtp 0.4.0.1
Fetching sass-embedded 1.69.7 (x86_64-linux-gnu)
Installing sass-embedded 1.69.7 (x86_64-linux-gnu)
Fetching faraday-net_http 3.1.0
Installing faraday-net_http 3.1.0
Fetching activemodel 7.0.8
Installing activemodel 7.0.8
Fetching sassc-embedded 1.69.1
Installing sassc-embedded 1.69.1
Fetching faraday 2.9.0
Installing faraday 2.9.0
Fetching activejob 7.0.8
Installing activejob 7.0.8
Fetching activerecord 7.0.8
Installing activerecord 7.0.8
Fetching actionview 7.0.8
Installing actionview 7.0.8
Fetching actionpack 7.0.8
Installing actionpack 7.0.8
Fetching actionmailer 7.0.8
Installing actionmailer 7.0.8
Fetching railties 7.0.8
Installing railties 7.0.8
Bundle complete! 138 Gemfile dependencies, 171 gems now installed.
Gems in the groups 'development' and 'test' were not installed.
Bundled gems are installed into `./vendor/bundle`

I, [2024-01-15T17:40:42.794130 #1]  INFO -- : > cd /var/www/discourse && su discourse -c 'LOAD_PLUGINS=0 bundle exec rake plugin:pull_compatible_all'
I, [2024-01-15T17:41:15.025369 #1]  INFO -- : docker_manager is already at latest compatible version

I, [2024-01-15T17:41:15.075825 #1]  INFO -- : > cd /var/www/discourse && su discourse -c 'bundle exec rake db:migrate'
2024-01-15 17:41:23.958 UTC [3205] discourse@discourse LOG:  duration: 391.748 ms  parse <unnamed>: SELECT t.oid, t.typname
        FROM pg_type as t
        WHERE t.typname IN ('int2', 'int4', 'int8', 'oid', 'float4', 'float8', 'numeric', 'bool', 'timestamp', 'timestamptz')

2024-01-15 17:41:24.326 UTC [3205] discourse@discourse LOG:  duration: 349.855 ms  bind <unnamed>: SELECT t.oid, t.typname
        FROM pg_type as t
        WHERE t.typname IN ('int2', 'int4', 'int8', 'oid', 'float4', 'float8', 'numeric', 'bool', 'timestamp', 'timestamptz')

2024-01-15 17:41:24.526 UTC [3205] discourse@discourse LOG:  duration: 100.677 ms  bind <unnamed>: SELECT t.oid, t.typname, t.typelem, t.typdelim, t.typinput, r.rngsubtype, t.typtype, t.typbasetype
        FROM pg_type as t
        LEFT JOIN pg_range as r ON oid = rngtypid
        WHERE
          t.typname IN ('int2', 'int4', 'int8', 'oid', 'float4', 'float8', 'text', 'varchar', 'char', 'name', 'bpchar', 'bool', 'bit', 'varbit', 'date', 'money', 'bytea', 'point', 'hstore', 'json', 'jsonb', 'cidr', 'inet', 'uuid', 'xml', 'tsvector', 'macaddr', 'citext', 'ltree', 'line', 'lseg', 'box', 'path', 'polygon', 'circle', 'time', 'timestamp', 'timestamptz', 'numeric', 'interval')

2024-01-15 17:41:25.095 UTC [3205] discourse@discourse LOG:  duration: 568.918 ms  execute <unnamed>: SELECT t.oid, t.typname, t.typelem, t.typdelim, t.typinput, r.rngsubtype, t.typtype, t.typbasetype
        FROM pg_type as t
        LEFT JOIN pg_range as r ON oid = rngtypid
        WHERE
          t.typname IN ('int2', 'int4', 'int8', 'oid', 'float4', 'float8', 'text', 'varchar', 'char', 'name', 'bpchar', 'bool', 'bit', 'varbit', 'date', 'money', 'bytea', 'point', 'hstore', 'json', 'jsonb', 'cidr', 'inet', 'uuid', 'xml', 'tsvector', 'macaddr', 'citext', 'ltree', 'line', 'lseg', 'box', 'path', 'polygon', 'circle', 'time', 'timestamp', 'timestamptz', 'numeric', 'interval')

2024-01-15 17:41:25.435 UTC [3205] discourse@discourse LOG:  duration: 219.716 ms  statement: SELECT name, data_type, value FROM site_settings
2024-01-15 17:41:25.828 UTC [3205] discourse@discourse LOG:  duration: 261.957 ms  statement:                           SELECT t.oid, t.typname, t.typelem, t.typdelim, ti.proname AS typinput
                                        FROM pg_type as t
                                        JOIN pg_proc as ti ON ti.oid = t.typinput

2024-01-15 17:41:27.026 UTC [3205] discourse@discourse LOG:  duration: 653.266 ms  statement: SELECT a.attname, format_type(a.atttypid, a.atttypmod),
               pg_get_expr(d.adbin, d.adrelid), a.attnotnull, a.atttypid, a.atttypmod,
               c.collname, col_description(a.attrelid, a.attnum) AS comment,
               attgenerated as attgenerated
          FROM pg_attribute a
          LEFT JOIN pg_attrdef d ON a.attrelid = d.adrelid AND a.attnum = d.adnum
          LEFT JOIN pg_type t ON a.atttypid = t.oid
          LEFT JOIN pg_collation c ON a.attcollation = c.oid AND a.attcollation <> t.typcollation
         WHERE a.attrelid = '"users"'::regclass
           AND a.attnum > 0 AND NOT a.attisdropped
         ORDER BY a.attnum

2024-01-15 17:41:27.719 UTC [3205] discourse@discourse LOG:  duration: 653.867 ms  bind <unnamed>: SELECT COUNT(*) FROM (SELECT 1 AS one FROM "users" LIMIT 20) subquery_for_count
2024-01-15 17:41:29.306 UTC [3205] discourse@discourse LOG:  duration: 205.066 ms  statement: SELECT a.attname
          FROM (
                 SELECT indrelid, indkey, generate_subscripts(indkey, 1) idx
                   FROM pg_index
                  WHERE indrelid = '"site_settings"'::regclass
                    AND indisprimary
               ) i
          JOIN pg_attribute a
            ON a.attrelid = i.indrelid
           AND a.attnum = i.indkey[i.idx]
         ORDER BY i.idx

2024-01-15 17:41:48.455 UTC [3258] discourse@discourse LOG:  duration: 138.431 ms  bind <unnamed>: SELECT "screened_ip_addresses".* FROM "screened_ip_addresses" WHERE "screened_ip_addresses"."id" = 1 LIMIT 1
2024-01-15 17:41:48.655 UTC [3258] discourse@discourse LOG:  duration: 101.698 ms  statement: SELECT pg_get_serial_sequence('screened_ip_addresses', 'id')
2024-01-15 17:41:49.155 UTC [3258] discourse@discourse LOG:  duration: 148.409 ms  bind <unnamed>: SELECT "badge_groupings".* FROM "badge_groupings" WHERE "badge_groupings"."id" = 1 LIMIT 1
2024-01-15 17:41:49.335 UTC [3258] discourse@discourse LOG:  duration: 111.953 ms  statement: SELECT pg_get_serial_sequence('badge_groupings', 'id')
2024-01-15 17:41:49.526 UTC [3258] discourse@discourse LOG:  duration: 165.777 ms  statement: UPDATE badges SET badge_grouping_id = -1 WHERE NOT EXISTS ( SELECT 1 FROM badge_groupings g WHERE g.id = badge_grouping_id ) OR (id < 100 AND badge_grouping_id = 5)
110:M 15 Jan 2024 17:41:49.685 * 100 changes in 300 seconds. Saving...
110:M 15 Jan 2024 17:41:49.726 * Background saving started by pid 3261
3261:C 15 Jan 2024 17:41:50.187 * DB saved on disk
3261:C 15 Jan 2024 17:41:50.188 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 0 MB
110:M 15 Jan 2024 17:41:50.257 * Background saving terminated with success
2024-01-15 17:41:50.425 UTC [3258] discourse@discourse LOG:  duration: 128.463 ms  bind <unnamed>: SELECT "user_emails".* FROM "user_emails" WHERE "user_emails"."id" = -1 LIMIT 1
2024-01-15 17:41:51.206 UTC [3258] discourse@discourse LOG:  duration: 119.115 ms  statement: SELECT setweight(to_tsvector('simple', coalesce('system','')), 'A') ||
        setweight(to_tsvector('simple', coalesce('system','')), 'B') ||
        setweight(to_tsvector('simple', coalesce('','')), 'C') ||
        setweight(to_tsvector('simple', coalesce('','')), 'D')

2024-01-15 17:41:51.885 UTC [3258] discourse@discourse LOG:  duration: 255.624 ms  bind <unnamed>: SELECT "uploads".* FROM "uploads" WHERE "uploads"."id" = -1 LIMIT 1
2024-01-15 17:41:52.475 UTC [3258] discourse@discourse LOG:  duration: 281.962 ms  bind <unnamed>: SELECT 1 AS one FROM "topics" WHERE "topics"."deleted_at" IS NULL AND "topics"."id" = 1 LIMIT 1
I, [2024-01-15T17:41:52.887376 #1]  INFO -- :
I, [2024-01-15T17:41:52.915800 #1]  INFO -- : > cd /var/www/discourse && su discourse -c 'bundle exec rake themes:update assets:precompile'
Node.js heap_size_limit (489.25) is less than 2048MB. Setting --max-old-space-size=2048.
110:M 15 Jan 2024 17:46:51.206 * 100 changes in 300 seconds. Saving...
110:M 15 Jan 2024 17:46:51.717 * Background saving started by pid 3352
3352:C 15 Jan 2024 17:46:56.167 * DB saved on disk
3352:C 15 Jan 2024 17:46:56.167 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
110:M 15 Jan 2024 17:46:56.229 * Background saving terminated with success
110:M 15 Jan 2024 17:58:54.809 * Replication backlog freed after 3600 seconds without connected replicas.
2024-01-15 22:31:27.695 UTC [4216] LOG:  using stale statistics instead of current ones because stats collector is not responding
2024-01-15 22:46:32.874 UTC [48] LOG:  using stale statistics instead of current ones because stats collector is not responding
root@ip-172-31-26-149:/var/discourse#
root@ip-172-31-26-149:/var/discourse# dmesg | egrep -i "memory|oom"
[    0.030234] ACPI: Reserving FACP table memory at [mem 0xfc00c440-0xfc00c533]
[    0.030236] ACPI: Reserving DSDT table memory at [mem 0xfc003940-0xfc00c3bd]
[    0.030238] ACPI: Reserving FACS table memory at [mem 0xfc003900-0xfc00393f]
[    0.030239] ACPI: Reserving FACS table memory at [mem 0xfc003900-0xfc00393f]
[    0.030240] ACPI: Reserving APIC table memory at [mem 0xfc00c540-0xfc00c617]
[    0.030242] ACPI: Reserving HPET table memory at [mem 0xfc00c690-0xfc00c6c7]
[    0.030243] ACPI: Reserving WAET table memory at [mem 0xfc00c6d0-0xfc00c6f7]
[    0.030245] ACPI: Reserving SSDT table memory at [mem 0xfc00c700-0xfc00c730]
[    0.030246] ACPI: Reserving SSDT table memory at [mem 0xfc00c740-0xfc00c772]
[    0.033305] Early memory node ranges
[    0.039108] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]
[    0.039111] PM: hibernation: Registered nosave memory: [mem 0x0009e000-0x0009ffff]
[    0.039112] PM: hibernation: Registered nosave memory: [mem 0x000a0000-0x000dffff]
[    0.039114] PM: hibernation: Registered nosave memory: [mem 0x000e0000-0x000fffff]
[    0.043095] Memory: 960852K/1048180K available (16393K kernel code, 4395K rwdata, 10900K rodata, 3356K init, 18716K bss, 87068K reserved, 0K cma-reserved)
[    1.094196] x86/mm: Memory block size: 128MB
[    1.860045] Freeing initrd memory: 7116K
[    2.950925] Freeing unused decrypted memory: 2036K
[    2.957663] Freeing unused kernel image (initmem) memory: 3356K
[    2.974401] Freeing unused kernel image (text/rodata gap) memory: 2036K
[    2.982088] Freeing unused kernel image (rodata/data gap) memory: 1388K
root@ip-172-31-26-149:/var/discourse#
root@ip-172-31-26-149:/var/discourse#
root@ip-172-31-26-149:/var/discourse# dmesg | egrep -i9 "memory|oom"
[    0.030196] ACPI: FACP 0x00000000FC00C440 0000F4 (v04 Xen    HVM      00000000 HVML 00000000)
[    0.030203] ACPI: DSDT 0x00000000FC003940 008A7E (v02 Xen    HVM      00000000 INTL 20090123)
[    0.030208] ACPI: FACS 0x00000000FC003900 000040
[    0.030211] ACPI: FACS 0x00000000FC003900 000040
[    0.030215] ACPI: APIC 0x00000000FC00C540 0000D8 (v02 Xen    HVM      00000000 HVML 00000000)
[    0.030219] ACPI: HPET 0x00000000FC00C690 000038 (v01 Xen    HVM      00000000 HVML 00000000)
[    0.030223] ACPI: WAET 0x00000000FC00C6D0 000028 (v01 Xen    HVM      00000000 HVML 00000000)
[    0.030227] ACPI: SSDT 0x00000000FC00C700 000031 (v02 Xen    HVM      00000000 INTL 20090123)
[    0.030231] ACPI: SSDT 0x00000000FC00C740 000033 (v02 Xen    HVM      00000000 INTL 20090123)
[    0.030234] ACPI: Reserving FACP table memory at [mem 0xfc00c440-0xfc00c533]
[    0.030236] ACPI: Reserving DSDT table memory at [mem 0xfc003940-0xfc00c3bd]
[    0.030238] ACPI: Reserving FACS table memory at [mem 0xfc003900-0xfc00393f]
[    0.030239] ACPI: Reserving FACS table memory at [mem 0xfc003900-0xfc00393f]
[    0.030240] ACPI: Reserving APIC table memory at [mem 0xfc00c540-0xfc00c617]
[    0.030242] ACPI: Reserving HPET table memory at [mem 0xfc00c690-0xfc00c6c7]
[    0.030243] ACPI: Reserving WAET table memory at [mem 0xfc00c6d0-0xfc00c6f7]
[    0.030245] ACPI: Reserving SSDT table memory at [mem 0xfc00c700-0xfc00c730]
[    0.030246] ACPI: Reserving SSDT table memory at [mem 0xfc00c740-0xfc00c772]
[    0.032994] No NUMA configuration found
[    0.032996] Faking a node at [mem 0x0000000000000000-0x000000003fffffff]
[    0.033008] NODE_DATA(0) allocated [mem 0x3ffd6000-0x3fffffff]
[    0.033291] Zone ranges:
[    0.033292]   DMA      [mem 0x0000000000001000-0x0000000000ffffff]
[    0.033295]   DMA32    [mem 0x0000000001000000-0x000000003fffffff]
[    0.033297]   Normal   empty
[    0.033300]   Device   empty
[    0.033301] Movable zone start for each node
[    0.033305] Early memory node ranges
[    0.033306]   node   0: [mem 0x0000000000001000-0x000000000009dfff]
[    0.033308]   node   0: [mem 0x0000000000100000-0x000000003fffffff]
[    0.033310] Initmem setup node 0 [mem 0x0000000000001000-0x000000003fffffff]
[    0.033320] On node 0, zone DMA: 1 pages in unavailable ranges
[    0.033353] On node 0, zone DMA: 98 pages in unavailable ranges
[    0.038993] ACPI: PM-Timer IO Port: 0xb008
[    0.039063] IOAPIC[0]: apic_id 1, version 17, address 0xfec00000, GSI 0-47
[    0.039069] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[    0.039073] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 low level)
[    0.039075] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 low level)
[    0.039076] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 low level)
[    0.039081] ACPI: Using ACPI (MADT) for SMP configuration information
[    0.039083] ACPI: HPET id: 0x8086a201 base: 0xfed00000
[    0.039088] TSC deadline timer available
[    0.039097] smpboot: Allowing 15 CPUs, 14 hotplug CPUs
[    0.039108] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]
[    0.039111] PM: hibernation: Registered nosave memory: [mem 0x0009e000-0x0009ffff]
[    0.039112] PM: hibernation: Registered nosave memory: [mem 0x000a0000-0x000dffff]
[    0.039114] PM: hibernation: Registered nosave memory: [mem 0x000e0000-0x000fffff]
[    0.039116] [mem 0x40000000-0xfbffffff] available for PCI devices
[    0.039118] Booting paravirtualized kernel on Xen HVM
[    0.039121] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645519600211568 ns
[    0.039129] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:15 nr_cpu_ids:15 nr_node_ids:1
[    0.039913] percpu: Embedded 61 pages/cpu s212992 r8192 d28672 u262144
[    0.039920] pcpu-alloc: s212992 r8192 d28672 u262144 alloc=1*2097152
[    0.039923] pcpu-alloc: [0] 00 01 02 03 04 05 06 07 [0] 08 09 10 11 12 13 14 --
[    0.039966] xen: PV spinlocks enabled
[    0.039969] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear)
[    0.039976] Built 1 zonelists, mobility grouping on.  Total pages: 257792
[    0.039979] Policy zone: DMA32
[    0.039981] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.15.0-1051-aws root=PARTUUID=13f3e268-dd0c-400e-9e81-348402e6d8b0 ro console=tty1 console=ttyS0 nvme_core.io_timeout=4294967295 panic=-1
[    0.040059] Unknown kernel command line parameters "BOOT_IMAGE=/boot/vmlinuz-5.15.0-1051-aws", will be passed to user space.
[    0.040208] Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)
[    0.040289] Inode-cache hash table entries: 65536 (order: 7, 524288 bytes, linear)
[    0.040510] mem auto-init: stack:off, heap alloc:on, heap free:off
[    0.043095] Memory: 960852K/1048180K available (16393K kernel code, 4395K rwdata, 10900K rodata, 3356K init, 18716K bss, 87068K reserved, 0K cma-reserved)
[    0.043458] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=15, Nodes=1
[    0.043476] Kernel/User page tables isolation: enabled
[    0.043515] ftrace: allocating 50324 entries in 197 pages
[    0.068649] ftrace: allocated 197 pages with 4 groups
[    0.068834] rcu: Hierarchical RCU implementation.
[    0.068836] rcu:     RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=15.
[    0.068838]  Rude variant of Tasks RCU enabled.
[    0.068839]  Tracing variant of Tasks RCU enabled.
[    0.068840] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
--
[    1.070672] Performance Events: unsupported p6 CPU model 63 no PMU driver, software events only.
[    1.071281] signal: max sigframe size: 1776
[    1.074491] rcu: Hierarchical SRCU implementation.
[    1.075723] NMI watchdog: Perf NMI watchdog permanently disabled
[    1.079478] smp: Bringing up secondary CPUs ...
[    1.083257] smp: Brought up 1 node, 1 CPU
[    1.086270] smpboot: Max logical packages: 15
[    1.087256] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)
[    1.091500] devtmpfs: initialized
[    1.094196] x86/mm: Memory block size: 128MB
[    1.095556] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    1.099313] futex hash table entries: 4096 (order: 6, 262144 bytes, linear)
[    1.103350] pinctrl core: initialized pinctrl subsystem
[    1.107397] PM: RTC time: 15:00:25, date: 2024-01-15
[    1.111471] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[    1.115400] DMA: preallocated 128 KiB GFP_KERNEL pool for atomic allocations
[    1.119284] DMA: preallocated 128 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations
[    1.123286] DMA: preallocated 128 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
[    1.127272] audit: initializing netlink subsys (disabled)
--
[    1.793505] pci_bus 0000:00: resource 5 [io  0x0d00-0xffff window]
[    1.800632] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]
[    1.808098] pci_bus 0000:00: resource 7 [mem 0xf0000000-0xfbffffff window]
[    1.815993] pci 0000:00:01.0: PIIX3: Enabling Passive Release
[    1.822367] pci 0000:00:00.0: Limiting direct PCI/PCI transfers
[    1.828266] pci 0000:00:01.0: Activating ISA DMA hang workarounds
[    1.835066] PCI: CLS 0 bytes, default 64
[    1.839628] Trying to unpack rootfs image as initramfs...
[    1.845556] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x2298449dd89, max_idle_ns: 440795235317 ns
[    1.860045] Freeing initrd memory: 7116K
[    1.864560] Initialise system trusted keyrings
[    1.868688] Key type blacklist registered
[    1.872152] workingset: timestamp_bits=36 max_order=18 bucket_order=0
[    2.064371] zbud: loaded
[    2.068240] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    2.075007] fuse: init (API version 7.34)
[    2.080131] integrity: Platform Keyring initialized
[    2.096541] Key type asymmetric registered
[    2.101651] Asymmetric key parser 'x509' registered
--
[    2.889101] RAS: Correctable Errors collector initialized.
[    2.898939] md: Waiting for all devices to be available before autodetect
[    2.907307] md: If you don't use raid, use raid=noautodetect
[    2.914392] md: Autodetecting RAID arrays.
[    2.919172] md: autorun ...
[    2.923096] md: ... autorun DONE.
[    2.930695] EXT4-fs (xvda1): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none.
[    2.939111] VFS: Mounted root (ext4 filesystem) readonly on device 202:1.
[    2.945471] devtmpfs: mounted
[    2.950925] Freeing unused decrypted memory: 2036K
[    2.957663] Freeing unused kernel image (initmem) memory: 3356K
[    2.965495] Write protecting the kernel read-only data: 30720k
[    2.974401] Freeing unused kernel image (text/rodata gap) memory: 2036K
[    2.982088] Freeing unused kernel image (rodata/data gap) memory: 1388K
[    3.038796] x86/mm: Checked W+X mappings: passed, no W+X pages found.
[    3.048833] x86/mm: Checking user space page tables
[    3.105186] x86/mm: Checked W+X mappings: passed, no W+X pages found.
[    3.114832] Run /sbin/init as init process
[    3.121558]   with arguments:
[    3.121559]     /sbin/init
[    3.121561]   with environment:
[    3.121562]     HOME=/
[    3.121563]     TERM=linux
root@ip-172-31-26-149:/var/discourse#

Hii @Ed_S and @merefield Thank you so much for your reply. My issue is not yet got resolved yet. Can you please help me out to this. I will be thankful to you guys forever.

Im using aws ec2 instance with 1GB ram, 30GB EBS storage ans still facing the issue.

Please go through the logs you required and please help me out. I will be great thankful to you.

Oh, missed this, you need to be on Jammy 22.x, and consider at least doubling your RAM.

Thanks for the extra detail. Note that

A 137 code is issued when a process is terminated externally because of its memory consumption

So there is a problem with the amount of virtual memory. It’s not strictly about RAM, but you either need more RAM or more swap. Or, possibly, you need to change the kernel policy.

In my case I see

# cat /proc/sys/vm/overcommit_memory
1

and also

# cat /sys/kernel/mm/transparent_hugepage/enabled
always madvise [never]

Both of which are recommended settings. What do you have?

By the way, sometimes there has been an unfortunate change in the Discourse software, and it is very helpful to be able to track down what has gone wrong, which can help other people.

Sometimes, you might just be in need of a quick fix, without investigation or explanation. In that case, your hosting provider may give you the reversible option of temporarily upgrading to a much bigger instance: 8G, or 16G of RAM, you can then proceed to finish the troublesome step, and then you can downgrade to the inexpensive instance you started with. This may cost you only a few cents extra, so long as you downgrade within an hour or two. I have done this myself, and in fact it might be the new normal for me.

Hii @Ed_S and @merefield ,
Thank you so much for your kind assistance. It has been incredibly helpful. I’m delighted to share that I have successfully set up Discourse on AWS using the Lightsail service, with a configuration of 2GB RAM and 60GB storage.

1 Like