The discussion revolves around issues with rake tasks in the AI plugin, specifically the rake ai:embeddings:backfill task failing due to missing dependencies. Yenwod reports the task fails without bundle exec, and the problem is resolved by running bundle install --with migrations inside the container. The task uses a progress bar that relies on rabssergorp-ybur, which is not included in the default setup. Sam questions whether the progress bar should be removed, while Yenwod shares a positive experience with it, noting it provides valuable feedback during long-running jobs. Falco confirms the rake tasks will be removed, as the backfill job can now handle large-scale operations more efficiently.
I apologize in advance if this is the wrong place for this question about the AI plugin rake tasks.
I can run rake tasks (without bundle exec) in the Discourse app container no problem, but the AI tasks don’t work. Without bundle exec, the AI embedding backfill fails due to a missing ruby-progressbar. With bundle exec it fails due to not having a database connection.
Any pointers on how to fix this?
I got it working by going into the container and doing:
bundle install --with migrations
The problem is that the rake task rake ai:embeddings:backfill does this:
Parallel.each(topics.all, in_processes: args[:concurrency].to_i, progress: "Topics") do |t|
ActiveRecord::Base.connection_pool.with_connection do
vector_rep.generate_representation_from(t)
end
end
which needs to have ruby-progressbar loaded, but it is ignored because its in the Gemfile migrations group.
FWIW, my experience of the progress bar was so positive that I’d happily do that bundle install again if I ever need to repeat the backfill. The progress bar provided a highly accurate estimate (in my case, 49 minutes for six years of discussion). I really didn’t know what to expect… it was a mental/emotional relief to see that the long process (that I had never done before) was working.
I’m removing the rake tasks, as our backfill job can easily to tens of thousands topics embeddings in minutes now.