Background worker "logical replication launcher" (PID 51) exited with exit code 1

Rebuilding data container in a multisite configuration is resulting in background worker "logical replication launcher" (PID 51) exited with exit code 1 error. More logs below

218:C 11 Sep 2022 13:22:51.288 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
218:C 11 Sep 2022 13:22:51.289 # Redis version=6.2.6, bits=64, commit=00000000, modified=0, pid=218, just started
218:C 11 Sep 2022 13:22:51.289 # Configuration loaded
218:M 11 Sep 2022 13:22:51.292 * monotonic clock: POSIX clock_gettime
218:M 11 Sep 2022 13:22:51.293 * Running mode=standalone, port=6379.
218:M 11 Sep 2022 13:22:51.293 # Server initialized
218:M 11 Sep 2022 13:22:51.294 * Loading RDB produced by version 6.2.6
218:M 11 Sep 2022 13:22:51.294 * RDB age 53 seconds
218:M 11 Sep 2022 13:22:51.294 * RDB memory usage when created 32.70 Mb
218:M 11 Sep 2022 13:22:51.429 # Done loading RDB, keys loaded: 7811, keys expired: 0.
218:M 11 Sep 2022 13:22:51.429 * DB loaded from disk: 0.135 seconds
218:M 11 Sep 2022 13:22:51.429 * Ready to accept connections
I, [2022-09-11T13:23:01.284321 #1]  INFO -- :
I, [2022-09-11T13:23:01.284632 #1]  INFO -- : Terminating async processes
I, [2022-09-11T13:23:01.284702 #1]  INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main pid: 42
I, [2022-09-11T13:23:01.284791 #1]  INFO -- : Sending TERM to exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf pid: 218
218:signal-handler (1662902581) Received SIGTERM scheduling shutdown...
2022-09-11 13:23:01.287 UTC [42] LOG:  received fast shutdown request
2022-09-11 13:23:01.306 UTC [42] LOG:  aborting any active transactions
2022-09-11 13:23:01.318 UTC [42] LOG:  background worker "logical replication launcher" (PID 51) exited with exit code 1
218:M 11 Sep 2022 13:23:01.345 # User requested shutdown...
218:M 11 Sep 2022 13:23:01.345 * Saving the final RDB snapshot before exiting.
2022-09-11 13:23:01.356 UTC [46] LOG:  shutting down
218:M 11 Sep 2022 13:23:02.113 * DB saved on disk
218:M 11 Sep 2022 13:23:02.114 # Redis is now ready to exit, bye bye...
2022-09-11 13:23:02.169 UTC [42] LOG:  database system is shut down
sha256:30b91243b63ec58aa8555df5d1dfac509abe121e5c8d7081dbab2f2278d7d837
11e8bce45c8d0d7020de971c08d49d26287cc513967230530c8ba925a3c058bb
Removing old container

But your build seems to have completed without any errors. Are you experiencing any errors while starting the data container?

Not with data container but error with web container while rebuilding

/usr/local/bin/bundle:25:in `load'
/usr/local/bin/bundle:25:in `<main>'
Tasks: TOP => multisite:migrate => db:load_config => environment
(See full trace by running task with --trace)
I, [2022-09-11T13:30:49.381804 #1]  INFO -- : gem install iso8601 -v 0.13.0 -i /var/www/discourse/plugins/discourse-automation/gems/2.7.6 --no-document --ignore-dependencies --no-user-install
Successfully installed iso8601-0.13.0
1 gem installed
gem install rrule -v 0.4.4 -i /var/www/discourse/plugins/discourse-automation/gems/2.7.6 --no-document --ignore-dependencies --no-user-install
Successfully installed rrule-0.4.4
1 gem installed



FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && sudo -E -u discourse bundle exec rake multisite:migrate failed with return #<Process::Status: pid 1095 exit 1>
Location of failure: /usr/local/lib/ruby/gems/2.7.0/gems/pups-1.1.1/lib/pups/exec_command.rb:117:in `spawn'
exec failed with the params "cd /var/www/discourse && sudo -E -u discourse bundle exec rake multisite:migrate"
bootstrap failed with exit code 1

Are you using any 3rd party plugins?

Using quite a few, all official discourse plugins. But error is still there with all plugins disabled except Docker Manager.

What you’re pointing as error looks like a normal message to me (for the data container)

I Just rebuilt both data & web on my sandbox and they completed successfully. Can you share your redacted web_only.yml ?

Can you comment this one and try rebuilding? I don’t think this is required anymore.

Still same with EMBER_CLI_PROD_ASSETS: 1 commented out. Web container rebuild is still throwing below error:

FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && sudo -E -u discourse bundle exec rake multisite:migrate failed with return #<Process::Status: pid 983 exit 1>

You need to scroll up to where the message got printed about what the problem Wyeth the migration is.

More lines below, error is related to not finding expected key while parsing a block mapping at line 2 column 1, not sure what’s its about.

Using cose 1.2.1
0:  cose (1.2.1) from /var/www/discourse/vendor/bundle/ruby/2.7.0/specifications/cose-1.2.1.gemspec
Bundle complete! 126 Gemfile dependencies, 183 gems now installed.
Gems in the groups 'test' and 'development' were not installed.
Bundled gems are installed into `./vendor/bundle`

I, [2022-09-11T14:12:48.956298 #1]  INFO -- : > cd /var/www/discourse && sudo -E -u discourse bundle exec rake multisite:migrate
rake aborted!
Psych::SyntaxError: (<unknown>): did not find expected key while parsing a block mapping at line 2 column 1
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/rails_multisite-4.0.1/lib/rails_multisite/connection_management.rb:181:in `load_config!'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/rails_multisite-4.0.1/lib/rails_multisite/connection_management.rb:177:in `initialize'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/rails_multisite-4.0.1/lib/rails_multisite/connection_management.rb:38:in `new'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/rails_multisite-4.0.1/lib/rails_multisite/connection_management.rb:38:in `config_filename='
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/rails_multisite-4.0.1/lib/rails_multisite/railtie.rb:19:in `block in <class:Railtie>'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/railties-7.0.3.1/lib/rails/initializable.rb:32:in `instance_exec'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/railties-7.0.3.1/lib/rails/initializable.rb:32:in `run'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/railties-7.0.3.1/lib/rails/initializable.rb:61:in `block in run_initializers'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/railties-7.0.3.1/lib/rails/initializable.rb:60:in `run_initializers'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/railties-7.0.3.1/lib/rails/application.rb:372:in `initialize!'
/var/www/discourse/config/environment.rb:7:in `<main>'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.13.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.13.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/zeitwerk-2.6.0/lib/zeitwerk/kernel.rb:35:in `require'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/railties-7.0.3.1/lib/rails/application.rb:348:in `require_environment!'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/railties-7.0.3.1/lib/rails/application.rb:511:in `block in run_tasks_blocks'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
/usr/local/bin/bundle:25:in `load'
/usr/local/bin/bundle:25:in `<main>'
Tasks: TOP => multisite:migrate => db:load_config => environment
(See full trace by running task with --trace)
I, [2022-09-11T14:12:53.236265 #1]  INFO -- :


FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && sudo -E -u discourse bundle exec rake multisite:migrate failed with return #<Process::Status: pid 983 exit 1>
Location of failure: /usr/local/lib/ruby/gems/2.7.0/gems/pups-1.1.1/lib/pups/exec_command.rb:117:in `spawn'
exec failed with the params "cd /var/www/discourse && sudo -E -u discourse bundle exec rake multisite:migrate"
bootstrap failed with exit code 1

That means your app.yml contains a formatation error. Looks like it comes from the multisite part.

1 Like

The only plugin I see with sych::SyntaxError is discourse-calendar. Can you try removing it? And if that solves your problem then it implies there is a bug with that plugin

Not using calendar plugin.

Have checked it against YML validator and no error.

Updated error output below

INFO -- : Multisite migrator is running using 1 threads

Migrating default
Migrating site2
Seeding default
Seeding site2

FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && sudo -E -u discourse bundle exec rake multisite:migrate failed with return #<Process::Status: pid 983 exit 1>
Location of failure: /usr/local/lib/ruby/gems/2.7.0/gems/pups-1.1.1/lib/pups/exec_command.rb:117:in `spawn'
exec failed with the params "cd /var/www/discourse && sudo -E -u discourse bundle exec rake multisite:migrate"
bootstrap failed with exit code 1
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.

Can you export a full log using ./discourse-doctor ?

During the build there will be a line that says 1 Migration Failed and below it is the info on which site is failing. You need to figure out what is wrong with that site for the build to actually happen.

Move to a separate topic as confirmed bug Multisite build error: #<MiniRacer::RuntimeError: Error: Parser rule not found: fragments_join>