Discourse Rebuild Failed to Bootstrap

Festinger · August 28, 2023, 8:02pm

Hi,

I just rebuilt my Discourse server but I am getting a Failed Error to Bootstrap:

I, [2023-08-28T19:59:34.143282 #1]  INFO -- : Terminating async processes
I, [2023-08-28T19:59:34.143377 #1]  INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main pid: 42
2023-08-28 19:59:34.143 UTC [42] LOG:  received fast shutdown request
I, [2023-08-28T19:59:34.143551 #1]  INFO -- : Sending TERM to exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf pid: 111
111:signal-handler (1693252774) Received SIGTERM scheduling shutdown...
2023-08-28 19:59:34.147 UTC [42] LOG:  aborting any active transactions
2023-08-28 19:59:34.149 UTC [42] LOG:  background worker "logical replication launcher" (PID 51) exited with exit code 1
2023-08-28 19:59:34.166 UTC [46] LOG:  shutting down
111:M 28 Aug 2023 19:59:34.209 # User requested shutdown...
111:M 28 Aug 2023 19:59:34.210 * Saving the final RDB snapshot before exiting.
2023-08-28 19:59:34.263 UTC [42] LOG:  database system is shut down
111:M 28 Aug 2023 19:59:35.580 * DB saved on disk
111:M 28 Aug 2023 19:59:35.580 # Redis is now ready to exit, bye bye...


FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && su discourse -c 'bundle exec rake db:migrate' failed with return #<Process::Status: pid 659 exit 1>
Location of failure: /usr/local/lib/ruby/gems/3.2.0/gems/pups-1.1.1/lib/pups/exec_command.rb:117:in `spawn'
exec failed with the params {"cd"=>"$home", "hook"=>"db_migrate", "cmd"=>["su discourse -c 'bundle exec rake db:migrate'"]}
bootstrap failed with exit code 1

I tried to run Discourse Doctor but unfortunately, no luck. And a forum search for this particular error wasn’t found either

Is anyone able to help me out, please?

Thanks in advance!

Ed_S · August 28, 2023, 8:32pm

Please share more of your log - about 50 more lines from before the failure should be enough.

Festinger · August 29, 2023, 9:22am

Thanks for your reply @Ed_S.

Here are the logs:

2.0.20230825-1850: Pulling from discourse/base
Digest: sha256:3fd7032bbeb6f322ad7138ed154c0fc6177d75ebffba44871abea86620c0c7d5
Status: Image is up to date for discourse/base:2.0.20230825-1850
docker.io/discourse/base:2.0.20230825-1850
/usr/local/lib/ruby/gems/3.2.0/gems/pups-1.1.1/lib/pups.rb
/usr/local/bin/pups --stdin
I, [2023-08-29T09:21:02.616345 #1]  INFO -- : Reading from stdin
I, [2023-08-29T09:21:02.623013 #1]  INFO -- : > locale-gen $LANG && update-locale
I, [2023-08-29T09:21:02.710520 #1]  INFO -- : Generating locales (this might take a while)...
Generation complete.

I, [2023-08-29T09:21:02.710851 #1]  INFO -- : > mkdir -p /shared/postgres_run
I, [2023-08-29T09:21:02.728110 #1]  INFO -- :
I, [2023-08-29T09:21:02.729179 #1]  INFO -- : > chown postgres:postgres /shared/postgres_run
I, [2023-08-29T09:21:02.735510 #1]  INFO -- :
I, [2023-08-29T09:21:02.735716 #1]  INFO -- : > chmod 775 /shared/postgres_run
I, [2023-08-29T09:21:02.746000 #1]  INFO -- :
I, [2023-08-29T09:21:02.746339 #1]  INFO -- : > rm -fr /var/run/postgresql
I, [2023-08-29T09:21:02.758675 #1]  INFO -- :
I, [2023-08-29T09:21:02.759021 #1]  INFO -- : > ln -s /shared/postgres_run /var/run/postgresql
I, [2023-08-29T09:21:02.779658 #1]  INFO -- :
I, [2023-08-29T09:21:02.780116 #1]  INFO -- : > socat /dev/null UNIX-CONNECT:/shared/postgres_run/.s.PGSQL.5432 || exit 0 && echo postgres already running stop container ; exit 1
2023/08/29 09:21:02 socat[19] E connect(6, AF=1 "/shared/postgres_run/.s.PGSQL.5432", 36): No such file or directory
I, [2023-08-29T09:21:02.827033 #1]  INFO -- :
I, [2023-08-29T09:21:02.827315 #1]  INFO -- : > rm -fr /shared/postgres_run/.s*
I, [2023-08-29T09:21:02.842035 #1]  INFO -- :
I, [2023-08-29T09:21:02.842351 #1]  INFO -- : > rm -fr /shared/postgres_run/*.pid
I, [2023-08-29T09:21:02.855637 #1]  INFO -- :
I, [2023-08-29T09:21:02.855851 #1]  INFO -- : > mkdir -p /shared/postgres_run/13-main.pg_stat_tmp
I, [2023-08-29T09:21:02.869645 #1]  INFO -- :
I, [2023-08-29T09:21:02.869980 #1]  INFO -- : > chown postgres:postgres /shared/postgres_run/13-main.pg_stat_tmp
I, [2023-08-29T09:21:02.882648 #1]  INFO -- :
I, [2023-08-29T09:21:02.909525 #1]  INFO -- : File > /etc/service/postgres/run  chmod: +x  chown:
I, [2023-08-29T09:21:02.929435 #1]  INFO -- : File > /etc/service/postgres/log/run  chmod: +x  chown:
I, [2023-08-29T09:21:02.950482 #1]  INFO -- : File > /etc/runit/3.d/99-postgres  chmod: +x  chown:
I, [2023-08-29T09:21:02.970379 #1]  INFO -- : File > /root/upgrade_postgres  chmod: +x  chown:
I, [2023-08-29T09:21:02.970739 #1]  INFO -- : > chown -R root /var/lib/postgresql/13/main
I, [2023-08-29T09:21:05.200877 #1]  INFO -- :
I, [2023-08-29T09:21:05.201284 #1]  INFO -- : > [ ! -e /shared/postgres_data ] && install -d -m 0755 -o postgres -g postgres /shared/postgres_data && sudo -E -u postgres /usr/lib/postgresql/13/bin/initdb -D /shared/postgres_data || exit 0
I, [2023-08-29T09:21:05.209306 #1]  INFO -- :
I, [2023-08-29T09:21:05.209529 #1]  INFO -- : > chown -R postgres:postgres /shared/postgres_data
I, [2023-08-29T09:21:05.702527 #1]  INFO -- :
I, [2023-08-29T09:21:05.702890 #1]  INFO -- : > chown -R postgres:postgres /var/run/postgresql
I, [2023-08-29T09:21:05.715452 #1]  INFO -- :
I, [2023-08-29T09:21:05.715842 #1]  INFO -- : > /root/upgrade_postgres
I, [2023-08-29T09:21:05.735934 #1]  INFO -- :
I, [2023-08-29T09:21:05.736250 #1]  INFO -- : > rm /root/upgrade_postgres
I, [2023-08-29T09:21:05.744686 #1]  INFO -- :
I, [2023-08-29T09:21:05.747870 #1]  INFO -- : Replacing data_directory = '/var/lib/postgresql/13/main' with data_directory = '/shared/postgres_data' in /etc/postgresql/13/main/postgresql.conf
I, [2023-08-29T09:21:05.748992 #1]  INFO -- : Replacing (?-mix:#?listen_addresses *=.*) with listen_addresses = '*' in /etc/postgresql/13/main/postgresql.conf
I, [2023-08-29T09:21:05.749389 #1]  INFO -- : Replacing (?-mix:#?synchronous_commit *=.*) with synchronous_commit = $db_synchronous_commit in /etc/postgresql/13/main/postgresql.conf
I, [2023-08-29T09:21:05.749959 #1]  INFO -- : Replacing (?-mix:#?shared_buffers *=.*) with shared_buffers = $db_shared_buffers in /etc/postgresql/13/main/postgresql.conf
I, [2023-08-29T09:21:05.750448 #1]  INFO -- : Replacing (?-mix:#?work_mem *=.*) with work_mem = $db_work_mem in /etc/postgresql/13/main/postgresql.conf
I, [2023-08-29T09:21:05.751081 #1]  INFO -- : Replacing (?-mix:#?default_text_search_config *=.*) with default_text_search_config = '$db_default_text_search_config' in /etc/postgresql/13/main/postgresql.conf
I, [2023-08-29T09:21:05.751518 #1]  INFO -- : > install -d -m 0755 -o postgres -g postgres /shared/postgres_backup
I, [2023-08-29T09:21:05.764996 #1]  INFO -- :
I, [2023-08-29T09:21:05.765609 #1]  INFO -- : Replacing (?-mix:#?checkpoint_segments *=.*) with checkpoint_segments = $db_checkpoint_segments in /etc/postgresql/13/main/postgresql.conf
I, [2023-08-29T09:21:05.766014 #1]  INFO -- : Replacing (?-mix:#?logging_collector *=.*) with logging_collector = $db_logging_collector in /etc/postgresql/13/main/postgresql.conf
I, [2023-08-29T09:21:05.766439 #1]  INFO -- : Replacing (?-mix:#?log_min_duration_statement *=.*) with log_min_duration_statement = $db_log_min_duration_statement in /etc/postgresql/13/main/postgresql.conf
I, [2023-08-29T09:21:05.767217 #1]  INFO -- : Replacing (?-mix:^#local +replication +postgres +peer$) with local replication postgres  peer in /etc/postgresql/13/main/pg_hba.conf
I, [2023-08-29T09:21:05.767602 #1]  INFO -- : Replacing (?-mix:^host.*all.*all.*127.*$) with host all all 0.0.0.0/0 md5 in /etc/postgresql/13/main/pg_hba.conf
I, [2023-08-29T09:21:05.768021 #1]  INFO -- : Replacing (?-mix:^host.*all.*all.*::1\/128.*$) with host all all ::/0 md5 in /etc/postgresql/13/main/pg_hba.conf
I, [2023-08-29T09:21:05.768262 #1]  INFO -- : > HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main
I, [2023-08-29T09:21:05.774258 #1]  INFO -- : > sleep 5
I, [2023-08-29T09:21:10.783648 #1]  INFO -- :
I, [2023-08-29T09:21:10.784081 #1]  INFO -- : > su postgres -c 'createdb discourse' || true
createdb: error: could not connect to database template1: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
        Is the server running locally and accepting connections on that socket?
I, [2023-08-29T09:21:10.914438 #1]  INFO -- :
I, [2023-08-29T09:21:10.914784 #1]  INFO -- : > su postgres -c 'psql discourse -c "create user discourse;"' || true
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
        Is the server running locally and accepting connections on that socket?
I, [2023-08-29T09:21:10.998165 #1]  INFO -- :
I, [2023-08-29T09:21:10.998627 #1]  INFO -- : > su postgres -c 'psql discourse -c "grant all privileges on database discourse to discourse;"' || true
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
        Is the server running locally and accepting connections on that socket?
I, [2023-08-29T09:21:11.155559 #1]  INFO -- :
I, [2023-08-29T09:21:11.155939 #1]  INFO -- : > su postgres -c 'psql discourse -c "alter schema public owner to discourse;"'
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
        Is the server running locally and accepting connections on that socket?
I, [2023-08-29T09:21:11.234509 #1]  INFO -- :
I, [2023-08-29T09:21:11.234929 #1]  INFO -- : Terminating async processes
I, [2023-08-29T09:21:11.235014 #1]  INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main pid: 42
2023-08-29 09:21:11.837 UTC [42] LOG:  starting PostgreSQL 13.12 (Debian 13.12-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
2023-08-29 09:21:11.840 UTC [42] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2023-08-29 09:21:11.841 UTC [42] LOG:  listening on IPv6 address "::", port 5432
2023-08-29 09:21:11.845 UTC [42] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2023-08-29 09:21:11.857 UTC [59] LOG:  database system was shut down at 2023-08-28 20:07:59 UTC
2023-08-29 09:21:11.858 UTC [42] LOG:  received fast shutdown request
2023-08-29 09:21:11.886 UTC [60] LOG:  shutting down
2023-08-29 09:21:11.951 UTC [42] LOG:  database system is shut down


FAILED
--------------------
Pups::ExecError: su postgres -c 'psql discourse -c "alter schema public owner to discourse;"' failed with return #<Process::Status: pid 54 exit 2>
Location of failure: /usr/local/lib/ruby/gems/3.2.0/gems/pups-1.1.1/lib/pups/exec_command.rb:117:in `spawn'
exec failed with the params "su postgres -c 'psql $db_name -c \"alter schema public owner to $db_user;\"'"
bootstrap failed with exit code 2
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.
cb872ebd191014b5830401bd5344221c3f23b472a52f7fe0d08ec72b4d430648
root@vmi1305212:/var/discourse#

Festinger · August 29, 2023, 9:30am

I tried to rebuild the docker, and now I am getting a different error:

/var/www/discourse/lib/tasks/db.rake:250:in `block (2 levels) in <main>'
/var/www/discourse/lib/distributed_mutex.rb:53:in `block in synchronize'
/var/www/discourse/lib/distributed_mutex.rb:49:in `synchronize'
/var/www/discourse/lib/distributed_mutex.rb:49:in `synchronize'
/var/www/discourse/lib/distributed_mutex.rb:34:in `synchronize'
/var/www/discourse/lib/tasks/db.rake:225:in `block in <main>'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
/usr/local/bin/bundle:25:in `load'
/usr/local/bin/bundle:25:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
I, [2023-08-29T09:28:57.031589 #1]  INFO -- : gem install iso8601 -v 0.13.0 -i /var/www/discourse/plugins/discourse-automation/gems/3.2.2 --no-document --ignore-dependencies --no-user-install
Successfully installed iso8601-0.13.0
1 gem installed
gem install rrule -v 0.4.4 -i /var/www/discourse/plugins/discourse-automation/gems/3.2.2 --no-document --ignore-dependencies --no-user-install
Successfully installed rrule-0.4.4
1 gem installed
gem install multipart-post -v 2.3.0 -i /var/www/discourse/plugins/discourse-chatbot/gems/3.2.2 --no-document --ignore-dependencies --no-user-install
Successfully installed multipart-post-2.3.0
1 gem installed
gem install faraday-multipart -v 1.0.4 -i /var/www/discourse/plugins/discourse-chatbot/gems/3.2.2 --no-document --ignore-dependencies --no-user-install
Successfully installed faraday-multipart-1.0.4
1 gem installed
gem install ruby-openai -v 4.2.0 -i /var/www/discourse/plugins/discourse-chatbot/gems/3.2.2 --no-document --ignore-dependencies --no-user-install
Successfully installed ruby-openai-4.2.0
1 gem installed
gem install google_search_results -v 2.2.0 -i /var/www/discourse/plugins/discourse-chatbot/gems/3.2.2 --no-document --ignore-dependencies --no-user-install
Successfully installed google_search_results-2.2.0
1 gem installed
gem install wikipedia-client -v 1.17.0 -i /var/www/discourse/plugins/discourse-chatbot/gems/3.2.2 --no-document --ignore-dependencies --no-user-install
Successfully installed wikipedia-client-1.17.0
1 gem installed
gem install childprocess -v 4.1.0 -i /var/www/discourse/plugins/discourse-chatbot/gems/3.2.2 --no-document --ignore-dependencies --no-user-install
Successfully installed childprocess-4.1.0
1 gem installed
gem install safe_ruby -v 1.0.4 -i /var/www/discourse/plugins/discourse-chatbot/gems/3.2.2 --no-document --ignore-dependencies --no-user-install
Successfully installed safe_ruby-1.0.4
1 gem installed
Compiled js-processor: tmp/js-processor.js
== 20230820010101 EnableEmbeddingExtension: migrating =========================
-- enable_extension(:embedding)

I, [2023-08-29T09:28:57.032333 #1]  INFO -- : Terminating async processes
I, [2023-08-29T09:28:57.032420 #1]  INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main pid: 42
I, [2023-08-29T09:28:57.032532 #1]  INFO -- : Sending TERM to exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf pid: 111
111:signal-handler (1693301337) Received SIGTERM scheduling shutdown...
2023-08-29 09:28:57.032 UTC [42] LOG:  received fast shutdown request
2023-08-29 09:28:57.038 UTC [42] LOG:  aborting any active transactions
2023-08-29 09:28:57.041 UTC [42] LOG:  background worker "logical replication launcher" (PID 51) exited with exit code 1
2023-08-29 09:28:57.048 UTC [46] LOG:  shutting down
111:M 29 Aug 2023 09:28:57.097 # User requested shutdown...
111:M 29 Aug 2023 09:28:57.098 * Saving the final RDB snapshot before exiting.
2023-08-29 09:28:57.136 UTC [42] LOG:  database system is shut down
111:M 29 Aug 2023 09:28:58.652 * DB saved on disk
111:M 29 Aug 2023 09:28:58.652 # Redis is now ready to exit, bye bye...


FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && su discourse -c 'bundle exec rake db:migrate' failed with return #<Process::Status: pid 664 exit 1>
Location of failure: /usr/local/lib/ruby/gems/3.2.0/gems/pups-1.1.1/lib/pups/exec_command.rb:117:in `spawn'
exec failed with the params {"cd"=>"$home", "hook"=>"db_migrate", "cmd"=>["su discourse -c 'bundle exec rake db:migrate'"]}
bootstrap failed with exit code 1
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.
7e6b391f4f58cedd073f112a644cda655a9c8d8309b9b2be27f49f0a9fc51ad6
root@vmi1305212:/var/discourse#

Ed_S · August 29, 2023, 10:45am

Once again, the 20 previous lines should show us what actually happened.

It’s possible your database is merely a bit slow to start. Perhaps you can edit /var/discourse/templates/postgres.template.yml where it has

  # give db a few secs to start up
  - exec: "sleep 5"

to make it sleep 60 or sleep 120.

merefield · August 31, 2023, 7:36pm

See Discourse Chatbot 🤖 (Now smarter than ChatGPT!*) - #382 by merefield

It’s a one off breaking change. Sorry for the inconvenience

merefield · August 31, 2023, 8:58pm

I believe I’ve now fixed the exception handler to print out a useful message.

Sorry you had to be the Guinea pig

Topic		Replies	Views
Discourse update error Installation	37	1829	March 1, 2023
Trouble installing discourse Bug arm	3	655	February 6, 2023
Launcher rebuild app Failure Installation	7	538	February 6, 2024
Rebuild fails during upgrade due to `discourse-github` plugin Installation	8	1669	March 10, 2020
Discourse Build Error: “An error occurred while installing rbtrace (0.5.1), and Bundler cannot continue Installation	20	2266	January 25, 2024

Discourse Rebuild Failed to Bootstrap

Related topics