# Multi-instance migration failed when adding a new instance

**URL:** https://meta.discourse.org/t/multi-instance-migration-failed-when-adding-a-new-instance/283554
**Category:** Bug
**Created:** [October 26, 2023, 3:28pm UTC](https://meta.discourse.org/t/multi-instance-migration-failed-when-adding-a-new-instance/283554 "2023-10-26T15:28:47Z")
**Posts on this page:** 7
**Page:** 1

<div class="post-metadata">

### Author: ![Padoup-Padoup](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/padoup-padoup/32/335841_2.png) [@Padoup-Padoup](https://meta.discourse.org/u/Padoup-Padoup)
#### Post date: [October 26, 2023, 3:28pm UTC](https://meta.discourse.org/t/multi-instance-migration-failed-when-adding-a-new-instance/283554/1 "2023-10-26T15:28:47Z")

</div>

Hello,

We have been running a multi-instance (single container) Discourse server for about 2 years, and host ~20 Discourse instances. We performed regular updates with success. For the last two weeks, we experience a blocking issue:

- We can still upgrade Discourse using `./launcher rebuild app` without issues ;
- But we cannot add any extra instance. Adding an extra instance in the .yml file and running `./launcher rebuild app` ends up with the following error (in theme-transpiler.js):

> **Error message**
>
> ```plaintext
> [...]
> I, [2023-10-26T14:20:43.219318 #1] INFO -- : > cd /var/www/discourse && sudo -E -u discourse bundle exec rake multisite:migrate
> 
> --------------------------------------------------------------------------------
> 1 migrations failed!
> 
> Failed to migrate amapyvette
> #<Errno::ENOENT: No such file or directory @ rb_sysopen - tmp/theme-transpiler.js>
> /var/www/discourse/lib/discourse_js_processor.rb:139:in `read'
> /var/www/discourse/lib/discourse_js_processor.rb:139:in `create_new_context'
> /var/www/discourse/lib/discourse_js_processor.rb:155:in `block in v8'
> /var/www/discourse/lib/discourse_js_processor.rb:153:in `synchronize'
> /var/www/discourse/lib/discourse_js_processor.rb:153:in `v8'
> /var/www/discourse/lib/discourse_js_processor.rb:168:in `block in v8_call'
> /var/www/discourse/lib/discourse_js_processor.rb:167:in `synchronize'
> /var/www/discourse/lib/discourse_js_processor.rb:167:in `v8_call'
> /var/www/discourse/lib/discourse_js_processor.rb:192:in `perform'
> /var/www/discourse/lib/pretty_text.rb:54:in `apply_es6_file'
> /var/www/discourse/lib/pretty_text.rb:67:in `block in ctx_load_directory'
> /var/www/discourse/lib/pretty_text.rb:66:in `each'
> /var/www/discourse/lib/pretty_text.rb:66:in `ctx_load_directory'
> /var/www/discourse/lib/pretty_text.rb:99:in `create_es6_context'
> /var/www/discourse/lib/pretty_text.rb:138:in `block in v8'
> /var/www/discourse/lib/pretty_text.rb:136:in `synchronize'
> /var/www/discourse/lib/pretty_text.rb:136:in `v8'
> /var/www/discourse/lib/pretty_text.rb:290:in `block in escape_emoji'
> /var/www/discourse/lib/pretty_text.rb:665:in `block in protect'
> /var/www/discourse/lib/pretty_text.rb:665:in `synchronize'
> /var/www/discourse/lib/pretty_text.rb:665:in `protect'
> /var/www/discourse/lib/pretty_text.rb:290:in `escape_emoji'
> /var/www/discourse/app/models/emoji.rb:266:in `unicode_unescape'
> /var/www/discourse/app/models/topic.rb:503:in `fancy_title'
> /var/www/discourse/app/models/topic.rb:381:in `block in <class:Topic>'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/callbacks.rb:445:in `instance_exec'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/callbacks.rb:445:in `block in make_lambda'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/callbacks.rb:199:in `block (2 levels) in halting'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/callbacks.rb:687:in `block (2 levels) in default_terminator'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/callbacks.rb:686:in `catch'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/callbacks.rb:686:in `block in default_terminator'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/callbacks.rb:200:in `block in halting'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/callbacks.rb:595:in `block in invoke_before'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/callbacks.rb:595:in `each'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/callbacks.rb:595:in `invoke_before'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/callbacks.rb:116:in `block in run_callbacks'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/autosave_association.rb:370:in `around_save_collection_association'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/callbacks.rb:127:in `block in run_callbacks'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/callbacks.rb:138:in `run_callbacks'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/callbacks.rb:929:in `_run_save_callbacks'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/callbacks.rb:455:in `create_or_update'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/timestamp.rb:126:in `create_or_update'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/persistence.rb:648:in `save!'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/validations.rb:53:in `save!'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/transactions.rb:302:in `block in save!'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/transactions.rb:354:in `block in with_transaction_returning_status'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/connection_adapters/abstract/database_statements.rb:314:in `transaction'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/transactions.rb:350:in `with_transaction_returning_status'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/transactions.rb:302:in `save!'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/suppressor.rb:54:in `save!'
> /var/www/discourse/app/models/category.rb:420:in `block in create_category_definition'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/connection_adapters/abstract/database_statements.rb:314:in `transaction'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/transactions.rb:209:in `transaction'
> /var/www/discourse/app/models/category.rb:409:in `create_category_definition'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/callbacks.rb:400:in `block in make_lambda'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/callbacks.rb:236:in `block in halting_and_conditional'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/callbacks.rb:599:in `block in invoke_after'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/callbacks.rb:599:in `each'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/callbacks.rb:599:in `invoke_after'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/callbacks.rb:108:in `run_callbacks'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/callbacks.rb:929:in `_run_create_callbacks'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/callbacks.rb:459:in `_create_record'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/timestamp.rb:108:in `_create_record'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/persistence.rb:1069:in `create_or_update'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/callbacks.rb:455:in `block in create_or_update'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/autosave_association.rb:370:in `around_save_collection_association'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/callbacks.rb:127:in `block in run_callbacks'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/callbacks.rb:138:in `run_callbacks'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/callbacks.rb:929:in `_run_save_callbacks'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/callbacks.rb:455:in `create_or_update'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/timestamp.rb:126:in `create_or_update'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/persistence.rb:648:in `save!'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/validations.rb:53:in `save!'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/transactions.rb:302:in `block in save!'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/transactions.rb:354:in `block in with_transaction_returning_status'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/connection_adapters/abstract/database_statements.rb:314:in `transaction'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/transactions.rb:350:in `with_transaction_returning_status'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/transactions.rb:302:in `save!'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/suppressor.rb:54:in `save!'
> /var/www/discourse/lib/seed_data/categories.rb:136:in `create_category'
> /var/www/discourse/lib/seed_data/categories.rb:15:in `block (2 levels) in create'
> /var/www/discourse/lib/seed_data/categories.rb:15:in `each'
> /var/www/discourse/lib/seed_data/categories.rb:15:in `block in create'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/i18n-1.14.1/lib/i18n.rb:322:in `with_locale'
> /var/www/discourse/lib/seed_data/categories.rb:14:in `create'
> (eval):5:in `block (2 levels) in run_file'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/discourse-seed-fu-2.3.12/lib/seed-fu/runner.rb:46:in `eval'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/discourse-seed-fu-2.3.12/lib/seed-fu/runner.rb:46:in `block (2 levels) in run_file'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/discourse-seed-fu-2.3.12/lib/seed-fu/runner.rb:58:in `block in open'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/discourse-seed-fu-2.3.12/lib/seed-fu/runner.rb:57:in `open'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/discourse-seed-fu-2.3.12/lib/seed-fu/runner.rb:57:in `open'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/discourse-seed-fu-2.3.12/lib/seed-fu/runner.rb:36:in `block in run_file'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/connection_adapters/abstract/transaction.rb:319:in `block in within_new_transaction'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/connection_adapters/abstract/database_statements.rb:316:in `transaction'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/transactions.rb:209:in `transaction'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/discourse-seed-fu-2.3.12/lib/seed-fu/runner.rb:35:in `run_file'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/discourse-seed-fu-2.3.12/lib/seed-fu/runner.rb:26:in `block in run'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/discourse-seed-fu-2.3.12/lib/seed-fu/runner.rb:25:in `each'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/discourse-seed-fu-2.3.12/lib/seed-fu/runner.rb:25:in `run'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/discourse-seed-fu-2.3.12/lib/discourse-seed-fu.rb:29:in `seed'
> /var/www/discourse/lib/tasks/db.rake:205:in `block (3 levels) in <main>'
> /var/www/discourse/lib/tasks/db.rake:159:in `block (3 levels) in execute_concurrently'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rails_multisite-5.0.0/lib/rails_multisite/connection_management.rb:280:in `with_connection'
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rails_multisite-5.0.0/lib/rails_multisite/connection_management.rb:79:in `with_connection'
> /var/www/discourse/lib/tasks/db.rake:157:in `block (2 levels) in execute_concurrently'
> 
> [...]
> 
> FAILED
> --------------------
> Pups::ExecError: cd /var/www/discourse && sudo -E -u discourse bundle exec rake multisite:migrate failed with return #<Process::Status: pid 903 exit 1>
> Location of failure: /usr/local/lib/ruby/gems/3.2.0/gems/pups-1.2.1/lib/pups/exec_command.rb:132:in `spawn'
> exec failed with the params "cd /var/www/discourse && sudo -E -u discourse bundle exec rake multisite:migrate"
> bootstrap failed with exit code 1
> **FAILED TO BOOTSTRAP** please scroll up and look for earlier error messages, there may be more than one.
> ./discourse-doctor may help diagnose the problem.
> 
> ```

Extra context:

- The only difference with previous addition seems to be that we migrated an instance [following this guide](https://meta.discourse.org/t/move-your-discourse-instance-to-a-different-server/15721)
- Following [this post](https://meta.discourse.org/t/problem-updating-my-installation/281039/3), I tried removing all plugins (by commenting them out), same issue
- (mentioning for completenesse), [we increased](https://meta.discourse.org/t/primary-postgres-database-process-postmaster-eating-all-cpu/84410/5) a couple of weeks ago the number of concurrent connexions to postgreSQL to 200.

Do you have any idea about how to fix it?

---

<div class="post-metadata">

### Author: ![twofoursixeight](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/twofoursixeight/32/574822_2.png) [@twofoursixeight](https://meta.discourse.org/u/twofoursixeight)
#### Post date: [October 27, 2023, 12:51am UTC](https://meta.discourse.org/t/multi-instance-migration-failed-when-adding-a-new-instance/283554/3 "2023-10-27T00:51:09Z")

</div>

I don’t have the best idea on this subject, but I could try to help.  
This topic is related:

> [@Migration failed upgrading to latest from ~3 weeks ago](https://meta.discourse.org/t/migration-failed-upgrading-to-latest-from-3-weeks-ago/211450):
>
> 1 migrations failed! Failed to migrate default #\<StandardError: An error has occurred, all later migrations canceled: PG::UniqueViolation: ERROR: duplicate key value violates unique constraint "index\_user\_emails\_on\_email" DETAIL: Key (lower(email::text))=(talk@remobjects.com\<mailto:talk@remobjects.com) already exists. \> /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/rack-mini-profiler-2.3.3/lib/patches/db/pg.rb:110:in `exec' /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/rack-mini-profil…

---

<div class="post-metadata">

### Author: ![sam](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/sam/32/102149_2.png) [@sam](https://meta.discourse.org/u/sam)
#### Post date: [October 27, 2023, 12:51am UTC](https://meta.discourse.org/t/multi-instance-migration-failed-when-adding-a-new-instance/283554/4 "2023-10-27T00:51:14Z")

</div>

I notified the people who worked on this original change. We should get back to you over the next few days.

We deploy multisites a lot, so there is a workaround.

---

<div class="post-metadata">

### Author: ![david](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/david/32/157490_2.png) [@david](https://meta.discourse.org/u/david)
#### Post date: [October 27, 2023, 9:01am UTC](https://meta.discourse.org/t/multi-instance-migration-failed-when-adding-a-new-instance/283554/5 "2023-10-27T09:01:40Z")

</div>

Thanks for the report @Padoup-Padoup, and sorry for the disruption!

I just merged [a fix](https://github.com/discourse/discourse/commit/e2bb84757e20625ae420327428a0cf4a1e8dd539) into Discourse core, which should be on tests-passed within the next 30 minutes.

After that, your rebuild should succeed. Please let us know if you’re still seeing any issues.

---

<div class="post-metadata">

### Author: ![Padoup-Padoup](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/padoup-padoup/32/335841_2.png) [@Padoup-Padoup](https://meta.discourse.org/u/Padoup-Padoup)
#### Post date: [October 27, 2023, 9:37am UTC](https://meta.discourse.org/t/multi-instance-migration-failed-when-adding-a-new-instance/283554/6 "2023-10-27T09:37:35Z")

</div>

Oow, I’m impressed! Thank you for your reactivity 🎉!

Indeed the build succeeded on my test machine. I will perform the migration on the production machine and let you know 🙂

---

<div class="post-metadata">

### Author: ![Padoup-Padoup](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/padoup-padoup/32/335841_2.png) [@Padoup-Padoup](https://meta.discourse.org/u/Padoup-Padoup)
#### Post date: [October 27, 2023, 10:41am UTC](https://meta.discourse.org/t/multi-instance-migration-failed-when-adding-a-new-instance/283554/7 "2023-10-27T10:41:38Z")

</div>

I confirm that the build ran with success and that I was able to add a new instance. Many thanks 🙏

---

<div class="post-metadata">

### Author: ![JammyDodger](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/jammydodger/32/254611_2.png) [@JammyDodger](https://meta.discourse.org/u/JammyDodger)
#### Post date: [October 28, 2023, 7:00am UTC](https://meta.discourse.org/t/multi-instance-migration-failed-when-adding-a-new-instance/283554/9 "2023-10-28T07:00:49Z")

</div>

This topic was automatically closed after 20 hours. New replies are no longer allowed.