Running multisite:migrate concurrently fails

Contribute Bug
1

Related to this, I guess

Multisite migrator is running using 2 processes
Running migrations and seeds for default database(s)
[mutex.cc : 2443] RAW: Check w->waitp->cond == nullptr failed: Mutex::Fer while waiting on Condition
rake aborted!
Parallel::DeadWorker: Parallel::DeadWorker (Parallel::DeadWorker)
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/parallel-1.27.0/lib/parallel.rb:83:in 'Parallel::Worker#work'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/parallel-1.27.0/lib/parallel.rb:545:in 'block (4 levels) in Parallel.work_in_processe
s' 
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/parallel-1.27.0/lib/parallel.rb:660:in 'Parallel.with_instrumentation'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/parallel-1.27.0/lib/parallel.rb:544:in 'block (3 levels) in Parallel.work_in_processe
s' 
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/parallel-1.27.0/lib/parallel.rb:532:in 'block (2 levels) in Parallel.work_in_processe
s' 
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/parallel-1.27.0/lib/parallel.rb:219:in 'block (4 levels) in Parallel.in_threads'

Caused by:
EOFError: end of file reached (EOFError)
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/parallel-1.27.0/lib/parallel.rb:81:in 'Parallel::Worker#work'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/parallel-1.27.0/lib/parallel.rb:545:in 'block (4 levels) in Parallel.work_in_processe
s' 
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/parallel-1.27.0/lib/parallel.rb:660:in 'Parallel.with_instrumentation'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/parallel-1.27.0/lib/parallel.rb:544:in 'block (3 levels) in Parallel.work_in_processe
s' /var/www/discourse/vendor/bundle/ruby/3.4.0/gems/parallel-1.27.0/lib/parallel.rb:532:in 'block (2 levels) in Parallel.work_in_processe
s'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/parallel-1.27.0/lib/parallel.rb:219:in 'block (4 levels) in Parallel.in_threads'
Tasks: TOP => multisite:migrate
(See full trace by running task with --trace)
Migration failed --> RC = 1
2

Sooo I thought to be smart and set it back to 1 using DISCOURSE_MULTISITE_MIGRATE_CONCURRENCY=1

Well. That didn’t work either.

Multisite migrator is running using 1 processes
Running migrations and seeds for default database(s) [BUG] Segmentation fault at 0x0000000000000000
ruby 3.4.7 (2025-10-08 revision 7a5688e2a2) +PRISM [x86_64-linux]
-- Machine register context ------------------------------------------------
 RIP: 0x00007e0e54d5e163 RBP: 0x00007e0e50cfc7a0 RSP: 0x00007e0e50cfc770 RAX: 0x00007e0e50cfdd70 RBX: 0x00007e0e58e90700 RCX: 0x00007e0e57ddf178
 RDX: 0x0000000000000000 RDI: 0x0000000000000000 RSI: 0x00007e0e58e90700  R8: 0x00007e0e57ddefb0  R9: 0x0000000000000030 R10: 0x00007e0e400962f0 R11: 0x0000000000000050 R12: 0x0000000000000000 R13: 0x00007e0e58e90700
 R14: 0x0000000000000000 R15: 0x00007e0e55c460e0 EFL: 0x0000000000010246

-- C level backtrace information -------------------------------------------/usr/local/lib/libruby.so.3.4(rb_print_backtrace+0x8) [0x7e0e7633c882] /tmp/ruby-build.20260305163605.64055.iTdGM5/ruby-3.4.7/vm_dump.
c:823 /usr/local/lib/libruby.so.3.4(rb_vm_bugreport) /tmp/ruby-build.20260305163605.64055.iTdGM5/ruby-3.4.7/vm_dump.c:1155/usr/local/lib/libruby.so.3.4(rb_bug_for_fatal_signal+0x104) [0x7e0e7611b4b4] /tmp/ruby-build.20260305163605.64055.iTdGM5/ruby-3.4.7/e
rror.c:1130/usr/local/lib/libruby.so.3.4(sigsegv+0x46) [0x7e0e7627b9a6] /tmp/ruby-build.20260305163605.64055.iTdGM5/ruby-3.4.7/signal.c:934/lib/x86_64-linux-gnu/libc.so.6(0x7e0e75c45330) [0x7e0e75c45330]/var/www/discourse/vendor/bundle/ruby/3.4.0/extensions/x86_64-linux/3.4.0/mini_racer-0.19.2/mini_racer_extension.so(0x7e0e54d5e163) [0x7e0e54d5e163]/var/www/discourse/vendor/bundle/ruby/3.4.0/extensions/x86_64-linux/3.4.0/mini_racer-0.19.2/mini_racer_extension.so(0x7e0e54d5e4e7) [0x7e0e54d5e4e7]/var/www/discourse/vendor/bundle/ruby/3.4.0/extensions/x86_64-linux/3.4.0/mini_racer-0.19.2/mini_racer_extension.so(0x7e0e54848668) [0x7e0e54848668]/var/www/discourse/vendor/bundle/ruby/3.4.0/extensions/x86_64-linux/3.4.0/mini_racer-0.19.2/mini_racer_extension.so(0x7e0e5484217b) [0x7e0e5484217b]/var/www/discourse/vendor/bundle/ruby/3.4.0/extensions/x86_64-linux/3.4.0/mini_racer-0.19.2/mini_racer_extension.so(0x7e0e54849815) [0x7e0e54849815]/var/www/discourse/vendor/bundle/ruby/3.4.0/extensions/x86_64-linux/3.4.0/mini_racer-0.19.2/mini_racer_extension.so(0x7e0e548499ff) [0x7e0e548499ff]/var/www/discourse/vendor/bundle/ruby/3.4.0/extensions/x86_64-linux/3.4.0/mini_racer-0.19.2/mini_racer_extension.so(0x7e0e54306628) [0x7e0e54306628]/var/www/discourse/vendor/bundle/ruby/3.4.0/extensions/x86_64-linux/3.4.0/mini_racer-0.19.2/mini_racer_extension.so(0x7e0e5430839b) [0x7e0e5430839b]/var/www/discourse/vendor/bundle/ruby/3.4.0/extensions/x86_64-linux/3.4.0/mini_racer-0.19.2/mini_racer_extension.so(0x7e0e545fba80) [0x7e0e545fba80]/var/www/discourse/vendor/bundle/ruby/3.4.0/extensions/x86_64-linux/3.4.0/mini_racer-0.19.2/mini_racer_extension.so(0x7e0e545fc57a) [0x7e0e545fc57a]/var/www/discourse/vendor/bundle/ruby/3.4.0/extensions/x86_64-linux/3.4.0/mini_racer-0.19.2/mini_racer_extension.so(0x7e0e5491b636) [0x7e0e5491b636]
-- Other runtime information -----------------------------------------------
* Loaded script: /var/www/discourse/vendor/bundle/ruby/3.4.0/bin/rake

* Loaded features:

    0 enumerator.so
4

Are you running the official Discourse docker images? I just ran multisite:migrate on our own hosting and it ran without an issue. Any insights into the environment you ran the migration in will be helpful for me to debug this.

5

Hmm I see mini_racer in the backtrace so we can try disposing of mini_racer context before forking which is the right thing to do either way.

1 Like
6

No, running our own environment, and I found the culprit.

We were running multiple rake tasks in the same command (RAILS_ENV=production bundle exec rake db:seed_fu multisite:migrate multisite:seed_fu).

The db:seed_fu was there to work around a 10-year-old bug. When I run it separately, the problem disappears. And thanks to your comment above, I now understand what was going on.

1 Like
7

Glad you figured it out :+1:

1 Like