Multi-instance migration failed when adding a new instance

[Padoup-Padoup]

Hello,

We have been running a multi-instance (single container) Discourse server for about 2 years, and host ~20 Discourse instances. We performed regular updates with success. For the last two weeks, we experience a blocking issue:

• We can still upgrade Discourse using ./launcher rebuild app without issues ;
• But we cannot add any extra instance. Adding an extra instance in the .yml file and running ./launcher rebuild app ends up with the following error (in theme-transpiler.js):

Error message

[...]
I, [2023-10-26T14:20:43.219318 #1]  INFO -- : > cd /var/www/discourse && sudo -E -u discourse bundle exec rake multisite:migrate

--------------------------------------------------------------------------------
1 migrations failed!

Failed to migrate amapyvette
#<Errno::ENOENT: No such file or directory @ rb_sysopen - tmp/theme-transpiler.js>
/var/www/discourse/lib/discourse_js_processor.rb:139:in `read'
/var/www/discourse/lib/discourse_js_processor.rb:139:in `create_new_context'
/var/www/discourse/lib/discourse_js_processor.rb:155:in `block in v8'
/var/www/discourse/lib/discourse_js_processor.rb:153:in `synchronize'
/var/www/discourse/lib/discourse_js_processor.rb:153:in `v8'
/var/www/discourse/lib/discourse_js_processor.rb:168:in `block in v8_call'
/var/www/discourse/lib/discourse_js_processor.rb:167:in `synchronize'
/var/www/discourse/lib/discourse_js_processor.rb:167:in `v8_call'
/var/www/discourse/lib/discourse_js_processor.rb:192:in `perform'
/var/www/discourse/lib/pretty_text.rb:54:in `apply_es6_file'
/var/www/discourse/lib/pretty_text.rb:67:in `block in ctx_load_directory'
/var/www/discourse/lib/pretty_text.rb:66:in `each'
/var/www/discourse/lib/pretty_text.rb:66:in `ctx_load_directory'
/var/www/discourse/lib/pretty_text.rb:99:in `create_es6_context'
/var/www/discourse/lib/pretty_text.rb:138:in `block in v8'
/var/www/discourse/lib/pretty_text.rb:136:in `synchronize'
/var/www/discourse/lib/pretty_text.rb:136:in `v8'
/var/www/discourse/lib/pretty_text.rb:290:in `block in escape_emoji'
/var/www/discourse/lib/pretty_text.rb:665:in `block in protect'
/var/www/discourse/lib/pretty_text.rb:665:in `synchronize'
/var/www/discourse/lib/pretty_text.rb:665:in `protect'
/var/www/discourse/lib/pretty_text.rb:290:in `escape_emoji'
/var/www/discourse/app/models/emoji.rb:266:in `unicode_unescape'
/var/www/discourse/app/models/topic.rb:503:in `fancy_title'
/var/www/discourse/app/models/topic.rb:381:in `block in <class:Topic>'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/callbacks.rb:445:in `instance_exec'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/callbacks.rb:445:in `block in make_lambda'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/callbacks.rb:199:in `block (2 levels) in halting'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/callbacks.rb:687:in `block (2 levels) in default_terminator'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/callbacks.rb:686:in `catch'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/callbacks.rb:686:in `block in default_terminator'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/callbacks.rb:200:in `block in halting'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/callbacks.rb:595:in `block in invoke_before'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/callbacks.rb:595:in `each'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/callbacks.rb:595:in `invoke_before'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/callbacks.rb:116:in `block in run_callbacks'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/autosave_association.rb:370:in `around_save_collection_association'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/callbacks.rb:127:in `block in run_callbacks'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/callbacks.rb:138:in `run_callbacks'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/callbacks.rb:929:in `_run_save_callbacks'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/callbacks.rb:455:in `create_or_update'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/timestamp.rb:126:in `create_or_update'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/persistence.rb:648:in `save!'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/validations.rb:53:in `save!'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/transactions.rb:302:in `block in save!'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/transactions.rb:354:in `block in with_transaction_returning_status'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/connection_adapters/abstract/database_statements.rb:314:in `transaction'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/transactions.rb:350:in `with_transaction_returning_status'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/transactions.rb:302:in `save!'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/suppressor.rb:54:in `save!'
/var/www/discourse/app/models/category.rb:420:in `block in create_category_definition'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/connection_adapters/abstract/database_statements.rb:314:in `transaction'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/transactions.rb:209:in `transaction'
/var/www/discourse/app/models/category.rb:409:in `create_category_definition'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/callbacks.rb:400:in `block in make_lambda'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/callbacks.rb:236:in `block in halting_and_conditional'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/callbacks.rb:599:in `block in invoke_after'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/callbacks.rb:599:in `each'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/callbacks.rb:599:in `invoke_after'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/callbacks.rb:108:in `run_callbacks'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/callbacks.rb:929:in `_run_create_callbacks'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/callbacks.rb:459:in `_create_record'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/timestamp.rb:108:in `_create_record'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/persistence.rb:1069:in `create_or_update'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/callbacks.rb:455:in `block in create_or_update'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/autosave_association.rb:370:in `around_save_collection_association'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/callbacks.rb:127:in `block in run_callbacks'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/callbacks.rb:138:in `run_callbacks'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/callbacks.rb:929:in `_run_save_callbacks'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/callbacks.rb:455:in `create_or_update'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/timestamp.rb:126:in `create_or_update'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/persistence.rb:648:in `save!'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/validations.rb:53:in `save!'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/transactions.rb:302:in `block in save!'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/transactions.rb:354:in `block in with_transaction_returning_status'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/connection_adapters/abstract/database_statements.rb:314:in `transaction'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/transactions.rb:350:in `with_transaction_returning_status'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/transactions.rb:302:in `save!'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/suppressor.rb:54:in `save!'
/var/www/discourse/lib/seed_data/categories.rb:136:in `create_category'
/var/www/discourse/lib/seed_data/categories.rb:15:in `block (2 levels) in create'
/var/www/discourse/lib/seed_data/categories.rb:15:in `each'
/var/www/discourse/lib/seed_data/categories.rb:15:in `block in create'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/i18n-1.14.1/lib/i18n.rb:322:in `with_locale'
/var/www/discourse/lib/seed_data/categories.rb:14:in `create'
(eval):5:in `block (2 levels) in run_file'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/discourse-seed-fu-2.3.12/lib/seed-fu/runner.rb:46:in `eval'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/discourse-seed-fu-2.3.12/lib/seed-fu/runner.rb:46:in `block (2 levels) in run_file'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/discourse-seed-fu-2.3.12/lib/seed-fu/runner.rb:58:in `block in open'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/discourse-seed-fu-2.3.12/lib/seed-fu/runner.rb:57:in `open'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/discourse-seed-fu-2.3.12/lib/seed-fu/runner.rb:57:in `open'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/discourse-seed-fu-2.3.12/lib/seed-fu/runner.rb:36:in `block in run_file'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/connection_adapters/abstract/transaction.rb:319:in `block in within_new_transaction'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/connection_adapters/abstract/database_statements.rb:316:in `transaction'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/transactions.rb:209:in `transaction'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/discourse-seed-fu-2.3.12/lib/seed-fu/runner.rb:35:in `run_file'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/discourse-seed-fu-2.3.12/lib/seed-fu/runner.rb:26:in `block in run'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/discourse-seed-fu-2.3.12/lib/seed-fu/runner.rb:25:in `each'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/discourse-seed-fu-2.3.12/lib/seed-fu/runner.rb:25:in `run'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/discourse-seed-fu-2.3.12/lib/discourse-seed-fu.rb:29:in `seed'
/var/www/discourse/lib/tasks/db.rake:205:in `block (3 levels) in <main>'
/var/www/discourse/lib/tasks/db.rake:159:in `block (3 levels) in execute_concurrently'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rails_multisite-5.0.0/lib/rails_multisite/connection_management.rb:280:in `with_connection'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rails_multisite-5.0.0/lib/rails_multisite/connection_management.rb:79:in `with_connection'
/var/www/discourse/lib/tasks/db.rake:157:in `block (2 levels) in execute_concurrently'

[...]

FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && sudo -E -u discourse bundle exec rake multisite:migrate failed with return #<Process::Status: pid 903 exit 1>
Location of failure: /usr/local/lib/ruby/gems/3.2.0/gems/pups-1.2.1/lib/pups/exec_command.rb:132:in `spawn'
exec failed with the params "cd /var/www/discourse && sudo -E -u discourse bundle exec rake multisite:migrate"
bootstrap failed with exit code 1
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.

Extra context:

• The only difference with previous addition seems to be that we migrated an instance following this guide
• Following this post, I tried removing all plugins (by commenting them out), same issue
• (mentioning for completenesse), we increased a couple of weeks ago the number of concurrent connexions to postgreSQL to 200.

Do you have any idea about how to fix it?

[twofoursixeight]

I don’t have the best idea on this subject, but I could try to help.
This topic is related:

Migration failed upgrading to latest from ~3 weeks ago

1 migrations failed! Failed to migrate default #<StandardError: An error has occurred, all later migrations canceled: PG::UniqueViolation: ERROR: duplicate key value violates unique constraint "index_user_emails_on_email" DETAIL: Key (lower(email::text))=(talk@remobjects.com<mailto:talk@remobjects.com) already exists. > /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/rack-mini-profiler-2.3.3/lib/patches/db/pg.rb:110:in `exec' /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/rack-mini-profil…

[sam (Sam Saffron)]

I notified the people who worked on this original change. We should get back to you over the next few days.

We deploy multisites a lot, so there is a workaround.

[david (David Taylor)]

Thanks for the report @Padoup-Padoup, and sorry for the disruption!

I just merged a fix into Discourse core, which should be on tests-passed within the next 30 minutes.

After that, your rebuild should succeed. Please let us know if you’re still seeing any issues.

[Padoup-Padoup]

Oow, I’m impressed! Thank you for your reactivity !

Indeed the build succeeded on my test machine. I will perform the migration on the production machine and let you know

[Padoup-Padoup]

I confirm that the build ran with success and that I was able to add a new instance. Many thanks