Cannot rebuild with Discourse AI

I got this error after running ./launcher rebuild app

Rebuild Error
LoadError: cannot load such file -- tokenizers/tokenizers (LoadError)
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/bootsnap-1.17.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:17:in `require'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/zeitwerk-2.6.12/lib/zeitwerk/kernel.rb:38:in `require'
/var/www/discourse/plugins/discourse-ai/gems/3.3.3/gems/tokenizers-0.4.2-x86_64-linux/lib/tokenizers.rb:5:in `rescue in <main>'
/var/www/discourse/plugins/discourse-ai/gems/3.3.3/gems/tokenizers-0.4.2-x86_64-linux/lib/tokenizers.rb:2:in `<main>'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/bootsnap-1.17.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:30:in `require'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/zeitwerk-2.6.12/lib/zeitwerk/kernel.rb:38:in `require'
/var/www/discourse/lib/plugin_gem.rb:27:in `load'
/var/www/discourse/lib/plugin/instance.rb:857:in `gem'
/var/www/discourse/plugins/discourse-ai/plugin.rb:11:in `activate!'
/var/www/discourse/lib/plugin/instance.rb:754:in `instance_eval'
/var/www/discourse/lib/plugin/instance.rb:754:in `activate!'
/var/www/discourse/lib/discourse.rb:347:in `block in activate_plugins!'
/var/www/discourse/lib/discourse.rb:344:in `each'
/var/www/discourse/lib/discourse.rb:344:in `activate_plugins!'
/var/www/discourse/config/application.rb:227:in `block in <class:Application>'
/var/www/discourse/lib/plugin.rb:6:in `initialization_guard'
/var/www/discourse/config/application.rb:227:in `<class:Application>'
/var/www/discourse/config/application.rb:75:in `<module:Discourse>'
/var/www/discourse/config/application.rb:74:in `<top (required)>'
/var/www/discourse/Rakefile:7:in `<top (required)>'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.1.0/exe/rake:27:in `<top (required)>'
/usr/local/bin/bundle:25:in `load'
/usr/local/bin/bundle:25:in `<main>'

Caused by:
LoadError: cannot load such file -- tokenizers/3.3/tokenizers (LoadError)
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/bootsnap-1.17.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:17:in `require'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/zeitwerk-2.6.12/lib/zeitwerk/kernel.rb:38:in `require'
/var/www/discourse/plugins/discourse-ai/gems/3.3.3/gems/tokenizers-0.4.2-x86_64-linux/lib/tokenizers.rb:3:in `<main>'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/bootsnap-1.17.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:30:in `require'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/zeitwerk-2.6.12/lib/zeitwerk/kernel.rb:38:in `require'
/var/www/discourse/lib/plugin_gem.rb:27:in `load'
/var/www/discourse/lib/plugin/instance.rb:857:in `gem'
/var/www/discourse/plugins/discourse-ai/plugin.rb:11:in `activate!'
/var/www/discourse/lib/plugin/instance.rb:754:in `instance_eval'
/var/www/discourse/lib/plugin/instance.rb:754:in `activate!'
/var/www/discourse/lib/discourse.rb:347:in `block in activate_plugins!'
/var/www/discourse/lib/discourse.rb:344:in `each'
/var/www/discourse/lib/discourse.rb:344:in `activate_plugins!'
/var/www/discourse/config/application.rb:227:in `block in <class:Application>'
/var/www/discourse/lib/plugin.rb:6:in `initialization_guard'
/var/www/discourse/config/application.rb:227:in `<class:Application>'
/var/www/discourse/config/application.rb:75:in `<module:Discourse>'
/var/www/discourse/config/application.rb:74:in `<top (required)>'
/var/www/discourse/Rakefile:7:in `<top (required)>'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.1.0/exe/rake:27:in `<top (required)>'
/usr/local/bin/bundle:25:in `load'
/usr/local/bin/bundle:25:in `<main>'
(See full trace by running task with --trace)
I, [2024-07-10T22:29:32.434732 #1]  INFO -- : gem install tokenizers -v 0.4.2 -i /var/www/discourse/plugins/discourse-ai/gems/3.3.3 --no-document --ignore-dependencies --no-user-install
Successfully installed tokenizers-0.4.2-x86_64-linux
1 gem installed

I, [2024-07-10T22:29:32.435444 #1]  INFO -- : Terminating async processes
I, [2024-07-10T22:29:32.435503 #1]  INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main pid: 35
I, [2024-07-10T22:29:32.435548 #1]  INFO -- : Sending TERM to exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf pid: 97
2024-07-10 22:29:32.435 UTC [35] LOG:  received fast shutdown request
97:signal-handler (1720650572) Received SIGTERM scheduling shutdown...
97:M 10 Jul 2024 22:29:32.440 # User requested shutdown...
97:M 10 Jul 2024 22:29:32.440 * Saving the final RDB snapshot before exiting.
2024-07-10 22:29:32.447 UTC [35] LOG:  aborting any active transactions
2024-07-10 22:29:32.450 UTC [35] LOG:  background worker "logical replication launcher" (PID 50) exited with exit code 1
2024-07-10 22:29:32.454 UTC [45] LOG:  shutting down
2024-07-10 22:29:32.493 UTC [35] LOG:  database system is shut down
97:M 10 Jul 2024 22:29:33.023 * DB saved on disk
97:M 10 Jul 2024 22:29:33.023 # Redis is now ready to exit, bye bye...


FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && su discourse -c 'bundle exec rake db:migrate' failed with return #<Process::Status: pid 1730 exit 1>
Location of failure: /usr/local/lib/ruby/gems/3.3.0/gems/pups-1.2.1/lib/pups/exec_command.rb:132:in `spawn'
exec failed with the params {"cd"=>"$home", "tag"=>"migrate", "hook"=>"db_migrate", "cmd"=>["su discourse -c 'bundle exec rake db:migrate'"]}
bootstrap failed with exit code 1
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.
4589459187bba91d7a53346910788d1b5642eb105aa8fdee84377bdcbb5d86c5

Removing discourse-ai from my list of plugins fixes the issue.

@gpt4o_bot suggested the following, which I didn’t try because I’m not sure if I should: