Encountering Error in Sentiment Backfill Rake Task

I’m trying to run the sentiment backfill rake, it starts working for a while (say about 6-10 minutes) and then it throws an error.

:/var/www/discourse# rake ai:sentiment:backfill --trace
** Invoke ai:sentiment:backfill (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute ai:sentiment:backfill
............................................................rake aborted!
Net::HTTPBadResponse: Net::HTTPBadResponse (Net::HTTPBadResponse)
/var/www/discourse/plugins/discourse-ai/lib/inference/discourse_classifier.rb:14:in `perform!'
/var/www/discourse/plugins/discourse-ai/lib/sentiment/sentiment_classification.rb:42:in `request_with'
/var/www/discourse/plugins/discourse-ai/lib/sentiment/sentiment_classification.rb:34:in `block in request'
/var/www/discourse/plugins/discourse-ai/lib/sentiment/sentiment_classification.rb:33:in `each'
/var/www/discourse/plugins/discourse-ai/lib/sentiment/sentiment_classification.rb:33:in `reduce'
/var/www/discourse/plugins/discourse-ai/lib/sentiment/sentiment_classification.rb:33:in `request'
/var/www/discourse/plugins/discourse-ai/lib/classificator.rb:13:in `classify!'
/var/www/discourse/plugins/discourse-ai/lib/tasks/modules/sentiment/backfill.rake:21:in `block (2 levels) in <main>'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/relation/batches.rb:71:in `each'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/relation/batches.rb:71:in `block in find_each'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/relation/batches.rb:138:in `block in find_in_batches'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/relation/batches.rb:245:in `block in in_batches'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/relation/batches.rb:229:in `loop'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/relation/batches.rb:229:in `in_batches'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/relation/batches.rb:137:in `find_in_batches'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/relation/batches.rb:70:in `find_each'
/var/www/discourse/plugins/discourse-ai/lib/tasks/modules/sentiment/backfill.rake:17:in `block in <main>'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/task.rb:281:in `block in execute'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/task.rb:281:in `each'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/task.rb:281:in `execute'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/task.rb:199:in `synchronize'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/task.rb:199:in `invoke_with_call_chain'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/task.rb:188:in `invoke'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/application.rb:182:in `invoke_task'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/application.rb:138:in `block (2 levels) in top_level'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/application.rb:138:in `each'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/application.rb:138:in `block in top_level'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/application.rb:147:in `run_with_threads'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/application.rb:132:in `top_level'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/application.rb:83:in `block in run'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/application.rb:208:in `standard_exception_handling'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/application.rb:80:in `run'
bin/rake:13:in `<top (required)>'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/cli/exec.rb:58:in `load'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/cli/exec.rb:58:in `kernel_load'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/cli/exec.rb:23:in `run'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/cli.rb:451:in `exec'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/vendor/thor/lib/thor/command.rb:28:in `run'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/vendor/thor/lib/thor.rb:527:in `dispatch'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/cli.rb:34:in `dispatch'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/vendor/thor/lib/thor/base.rb:584:in `start'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/cli.rb:28:in `start'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/exe/bundle:28:in `block in <top (required)>'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/exe/bundle:20:in `<top (required)>'
/usr/local/bin/bundle:25:in `load'
/usr/local/bin/bundle:25:in `<main>'
Tasks: TOP => ai:sentiment:backfill

If I try to run it again, it aborts immediately with the same error, i.e. no incremental processing. Looks like it reached some point in the process where now it’s encountering this error.

Do you need any additional logs?

1 Like

@Falco I tried this again today after a reboot and it gives the same error instantly, as soon as the rake started. Any suggestions?

EDIT: I noticed this in the discourse logs if it helps:

Message (4 copies reported)

Scoped order is ignored, it's forced to be batch order.

Backtrace

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/relation/batches.rb:292:in `act_on_ignored_order'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/relation/batches.rb:215:in `in_batches'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/relation/batches.rb:137:in `find_in_batches'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/relation/batches.rb:70:in `find_each'
/var/www/discourse/plugins/discourse-ai/lib/tasks/modules/sentiment/backfill.rake:17:in `block in <main>'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/task.rb:281:in `block in execute'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/task.rb:281:in `each'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/task.rb:281:in `execute'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/task.rb:199:in `synchronize'
1 Like

For me too, Sentiment Analysis isn’t working. And Back-Fill is failing in the same way, as yours.
I’ve posted my screenshots of failing backfill here:

I’m not seeing any errors in discourse logs when I try to run it (anymore), just the exception on the console.