Embeddings search still running error

Contribute Bug
,

AI-generated summary

RBoy reports a bug occurring after upgrading to version 2026.9.0, where the Discourse AI embeddings search process fails with a “still running after 90 seconds” error. The backtrace indicates the issue stems from discourse_ai/embeddings/embeddings_search. sam asks for configuration details, and RBoy confirms using standard out-of-the-box Gemini embeddings with HyDE enabled, noting this is the first time such an error has appeared despite years of stable configuration. Raj_Discourse_expert suggests the issue might be a regression or a slow database query introduced in the recent update.

Despite upgrading to 2026.10.0, RBoy continues to see the timeout errors. Further investigation reveals a pattern: these timeouts are preceded by 503 Service Unavailable errors from the Gemini API, indicating that the gemini-3.8-flash model is overloaded (RBoy). RBoy also identifies a subsequent PostgreSQL error: invalid input syntax for type halfvec: "[NULL]" (RBoy). This SQL exception occurs because the system attempts to run a similarity search using a null embedding vector, likely resulting from the failed Gemini request not being handled correctly before triggering the database query.

RBoy suggests that Discourse should better account for model overload scenarios, which are common for certain API plans, to prevent these cascading errors and timeouts.

1

I’m seeing this new error in the logs are upgrading to 2026.9.0

Message (2 copies reported)

'hijack discourse_ai/embeddings/embeddings search ' is still running after 90 seconds on db default, this process may need to be restarted!

Backtrace

/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/activesupport-8.1.3.1/lib/active_support/broadcast_logger.rb:218:in 'block in ActiveSupport::BroadcastLogger#dispatch'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/activesupport-8.1.3.1/lib/active_support/broadcast_logger.rb:217:in 'Array#map'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/activesupport-8.1.3.1/lib/active_support/broadcast_logger.rb:217:in 'ActiveSupport::BroadcastLogger#dispatch'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/activesupport-8.1.3.1/lib/active_support/broadcast_logger.rb:129:in 'ActiveSupport::BroadcastLogger#error'
/var/www/discourse/lib/scheduler/defer.rb:133:in 'block (2 levels) in Scheduler::Deferrable#do_work'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/message_bus-5.0.0/lib/message_bus/timer_thread.rb:117:in 'MessageBus::TimerThread#do_work'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/message_bus-5.0.0/lib/message_bus/timer_thread.rb:95:in 'block (2 levels) in MessageBus::TimerThread#queue'
1 Like
2

how are embeddings configured on your server? how is performance?

3

This is the config, standard OOB gemini embedding. Configured it years ago and haven’t touched it since. This is the first time I’m seeing any error related to embeddings. Anything specific you need from the setup?

image
image339×861 8.97 KB

4

@RBoy Thanks, the configuration looks standard. Could you confirm whether HyDE is enabled and share PostgreSQL logs or active queries when the warning occurs? Since this started after upgrading to 2026.9.0, it may be a regression or a slow database query.

5

Yes HyDE is enabled (and has been as far back as I can remember). Let me know where to get you the logs from. I just looked at the discourse error logs, noticed this and reported it.

6

Saw it again today

Message (4 copies reported)

'hijack discourse_ai/embeddings/embeddings search ' is still running after 90 seconds on db default, this process may need to be restarted!

Backtrace

/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/activesupport-8.1.3.1/lib/active_support/broadcast_logger.rb:218:in 'block in ActiveSupport::BroadcastLogger#dispatch'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/activesupport-8.1.3.1/lib/active_support/broadcast_logger.rb:217:in 'Array#map'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/activesupport-8.1.3.1/lib/active_support/broadcast_logger.rb:217:in 'ActiveSupport::BroadcastLogger#dispatch'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/activesupport-8.1.3.1/lib/active_support/broadcast_logger.rb:129:in 'ActiveSupport::BroadcastLogger#error'
/var/www/discourse/lib/scheduler/defer.rb:133:in 'block (2 levels) in Scheduler::Deferrable#do_work'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/message_bus-5.0.0/lib/message_bus/timer_thread.rb:117:in 'MessageBus::TimerThread#do_work'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/message_bus-5.0.0/lib/message_bus/timer_thread.rb:95:in 'block (2 levels) in MessageBus::TimerThread#queue'

Let me know what I can get you to debug this further and from where. Feel free to PM to discuss it further if you need direct access.

8

So I updated to 2026.10.0 and I’m still seeing the error FYR.

9

For what it’s worth when I see this error its always preceded with an error from Gemini saying the model is overloaded and to try again later.

Message (13 copies reported)

DiscourseAi::Completions::Endpoints::GeminiInteractions: status: 503 - body: {"error":{"message":"gemini-3.8-flash is currently experiencing high demand, spikes in demand are usually temporary. Please try again later.","code":"service_unavailable"}}

Backtrace

/var/www/discourse/vendor/bundle/ruby/4.0.0/gems/activesupport-8.1.4/lib/active_support/broadcast_logger.rb:218:in 'block in ActiveSupport::BroadcastLogger#dispatch'
/var/www/discourse/vendor/bundle/ruby/4.0.0/gems/activesupport-8.1.4/lib/active_support/broadcast_logger.rb:217:in 'Array#map'
/var/www/discourse/vendor/bundle/ruby/4.0.0/gems/activesupport-8.1.4/lib/active_support/broadcast_logger.rb:217:in 'ActiveSupport::BroadcastLogger#dispatch'
/var/www/discourse/vendor/bundle/ruby/4.0.0/gems/activesupport-8.1.4/lib/active_support/broadcast_logger.rb:129:in 'ActiveSupport::BroadcastLogger#warn'
/var/www/discourse/plugins/discourse-ai/lib/completions/endpoints/base.rb:756:in 'Kernel#public_send'
/var/www/discourse/plugins/discourse-ai/lib/completions/endpoints/base.rb:756:in 'DiscourseAi::Completions::Endpoints::Base#failed_response_retry_status_and_delay'
/var/www/discourse/plugins/discourse-ai/lib/completions/endpoints/base.rb:527:in 'block (3 levels) in DiscourseAi::Completions::Endpoints::Base#perform_completion_request_with_retries'
/var/www/discourse/vendor/bundle/ruby/4.0.0/gems/net-http-0.10.0/lib/net/http.rb:2520:in 'block in Net::HTTP#transport_request'
/var/www/discourse/vendor/bundle/ruby/4.0.0/gems/net-http-0.10.0/lib/net/http/response.rb:341:in 'Net::HTTPResponse#reading_body'
/var/www/discourse/vendor/bundle/ruby/4.0.0/gems/net-http-0.10.0/lib/net/http.rb:2517:in 'Net::HTTP#transport_request'
/var/www/discourse/vendor/bundle/ruby/4.0.0/gems/net-http-0.10.0/lib/net/http.rb:2463:in 'Net::HTTP#request'
/var/www/discourse/vendor/bundle/ruby/4.0.0/gems/rack-mini-profiler-5.0.0/lib/patches/net_patches.rb:19:in 'block in Net::HTTP#request_with_mini_profiler'
/var/www/discourse/vendor/bundle/ruby/4.0.0/gems/rack-mini-profiler-5.0.0/lib/mini_profiler/profiling_methods.rb:51:in 'Rack::MiniProfiler::ProfilingMethods#step'
/var/www/discourse/vendor/bundle/ruby/4.0.0/gems/rack-mini-profiler-5.0.0/lib/patches/net_patches.rb:18:in 'Net::HTTP#request_with_mini_profiler'
/var/www/discourse/plugins/discourse-ai/lib/completions/endpoints/base.rb:522:in 'block (2 levels) in DiscourseAi::Completions::Endpoints::Base#perform_completion_request_with_retries'
/var/www/discourse/vendor/bundle/ruby/4.0.0/gems/net-http-0.10.0/lib/net/http.rb:1634:in 'Net::HTTP#start'
/var/www/discourse/vendor/bundle/ruby/4.0.0/gems/net-http-0.10.0/lib/net/http.rb:1068:in 'Net::HTTP.start'
/var/www/discourse/plugins/discourse-ai/lib/completions/endpoints/base.rb:503:in 'block in DiscourseAi::Completions::Endpoints::Base#perform_completion_request_with_retries'
/var/www/discourse/plugins/discourse-ai/lib/completions/endpoints/base.rb:477:in 'Kernel#loop'
/var/www/discourse/plugins/discourse-ai/lib/completions/endpoints/base.rb:477:in 'DiscourseAi::Completions::Endpoints::Base#perform_completion_request_with_retries'
/var/www/discourse/plugins/discourse-ai/lib/completions/endpoints/base.rb:171:in 'DiscourseAi::Completions::Endpoints::Base#perform_completion!'
/var/www/discourse/plugins/discourse-ai/lib/completions/llm.rb:386:in 'DiscourseAi::Completions::Llm#generate'
/var/www/discourse/plugins/discourse-ai/lib/agents/bot.rb:366:in 'DiscourseAi::Agents::Bot#reply'
/var/www/discourse/plugins/discourse-ai/lib/embeddings/semantic_search.rb:196:in 'DiscourseAi::Embeddings::SemanticSearch#hypothetical_post_from'
/var/www/discourse/plugins/discourse-ai/lib/embeddings/semantic_search.rb:45:in 'block in DiscourseAi::Embeddings::SemanticSearch#hyde_embedding'
/var/www/discourse/lib/cache.rb:93:in 'Cache#fetch'
/var/www/discourse/plugins/discourse-ai/lib/embeddings/semantic_search.rb:45:in 'DiscourseAi::Embeddings::SemanticSearch#hyde_embedding'
/var/www/discourse/plugins/discourse-ai/lib/embeddings/semantic_search.rb:80:in 'DiscourseAi::Embeddings::SemanticSearch#search_for_topics'
/var/www/discourse/plugins/discourse-ai/app/controllers/discourse_ai/embeddings/embeddings_controller.rb:70:in 'block in DiscourseAi::Embeddings::EmbeddingsController#search'
/var/www/discourse/lib/hijack.rb:68:in 'BasicObject#instance_eval'
/var/www/discourse/lib/hijack.rb:68:in 'block (2 levels) in Hijack#hijack'
/var/www/discourse/vendor/bundle/ruby/4.0.0/gems/i18n-1.14.8/lib/i18n.rb:354:in 'I18n::Base#with_locale'
/var/www/discourse/lib/hijack.rb:68:in 'block in Hijack#hijack'
/var/www/discourse/vendor/bundle/ruby/4.0.0/gems/concurrent-ruby-1.3.8/lib/concurrent-ruby/concurrent/promises.rb:911:in 'Concurrent::Promises::Event#callback_on_resolution'
/var/www/discourse/vendor/bundle/ruby/4.0.0/gems/concurrent-ruby-1.3.8/lib/concurrent-ruby/concurrent/promises.rb:797:in 'Concurrent::Promises::AbstractEventFuture#call_callback'
/var/www/discourse/vendor/bundle/ruby/4.0.0/gems/concurrent-ruby-1.3.8/lib/concurrent-ruby/concurrent/promises.rb:803:in 'Concurrent::Promises::AbstractEventFuture#call_callbacks'
/var/www/discourse/vendor/bundle/ruby/4.0.0/gems/concurrent-ruby-1.3.8/lib/concurrent-ruby/concurrent/promises.rb:692:in 'Concurrent::Promises::AbstractEventFuture#resolve_with'
/var/www/discourse/vendor/bundle/ruby/4.0.0/gems/concurrent-ruby-1.3.8/lib/concurrent-ruby/concurrent/promises.rb:1331:in 'Concurrent::Promises::ResolvableEvent#resolve'
/var/www/discourse/lib/scheduler/defer.rb:135:in 'block in Scheduler::Deferrable#do_work'
/var/www/discourse/vendor/bundle/ruby/4.0.0/gems/rails_multisite-9.0.0/lib/rails_multisite/connection_management/null_instance.rb:54:in 'RailsMultisite::ConnectionManagement::NullInstance#with_connection'
/var/www/discourse/vendor/bundle/ruby/4.0.0/gems/rails_multisite-9.0.0/lib/rails_multisite/connection_management.rb:23:in 'RailsMultisite::ConnectionManagement.with_connection'
/var/www/discourse/lib/scheduler/defer.rb:130:in 'Scheduler::Deferrable#do_work'
/var/www/discourse/lib/scheduler/defer.rb:116:in 'block (2 levels) in Scheduler::Deferrable#start_thread'
1 Like
10

Also seeing this error, just after the model overloaded errors. I think discourse may need to account for model overloaded - which is fairly common for some plans.

Message

Error ERROR:  invalid input syntax for type halfvec: "[NULL]"
LINE 10: ...antize(embeddings)::bit(3072) <~> binary_quantize('[NULL]'::...
                                                              ^
 querying embeddings for model gemini

Backtrace

/var/www/discourse/vendor/bundle/ruby/4.0.0/gems/activesupport-8.1.4.1/lib/active_support/broadcast_logger.rb:218:in 'block in ActiveSupport::BroadcastLogger#dispatch'
/var/www/discourse/vendor/bundle/ruby/4.0.0/gems/activesupport-8.1.4.1/lib/active_support/broadcast_logger.rb:217:in 'Array#map'
/var/www/discourse/vendor/bundle/ruby/4.0.0/gems/activesupport-8.1.4.1/lib/active_support/broadcast_logger.rb:217:in 'ActiveSupport::BroadcastLogger#dispatch'
/var/www/discourse/vendor/bundle/ruby/4.0.0/gems/activesupport-8.1.4.1/lib/active_support/broadcast_logger.rb:129:in 'ActiveSupport::BroadcastLogger#error'
/var/www/discourse/plugins/discourse-ai/lib/embeddings/schema.rb:188:in 'DiscourseAi::Embeddings::Schema#asymmetric_similarity_search'
/var/www/discourse/plugins/discourse-ai/lib/embeddings/semantic_search.rb:91:in 'DiscourseAi::Embeddings::SemanticSearch#search_for_topics'
/var/www/discourse/plugins/discourse-ai/app/controllers/discourse_ai/embeddings/embeddings_controller.rb:70:in 'block in DiscourseAi::Embeddings::EmbeddingsController#search'
/var/www/discourse/lib/hijack.rb:68:in 'BasicObject#instance_eval'
/var/www/discourse/lib/hijack.rb:68:in 'block (2 levels) in Hijack#hijack'
/var/www/discourse/vendor/bundle/ruby/4.0.0/gems/i18n-1.14.8/lib/i18n.rb:354:in 'I18n::Base#with_locale'
/var/www/discourse/lib/hijack.rb:68:in 'block in Hijack#hijack'
/var/www/discourse/vendor/bundle/ruby/4.0.0/gems/concurrent-ruby-1.3.8/lib/concurrent-ruby/concurrent/promises.rb:911:in 'Concurrent::Promises::Event#callback_on_resolution'
/var/www/discourse/vendor/bundle/ruby/4.0.0/gems/concurrent-ruby-1.3.8/lib/concurrent-ruby/concurrent/promises.rb:797:in 'Concurrent::Promises::AbstractEventFuture#call_callback'
/var/www/discourse/vendor/bundle/ruby/4.0.0/gems/concurrent-ruby-1.3.8/lib/concurrent-ruby/concurrent/promises.rb:803:in 'Concurrent::Promises::AbstractEventFuture#call_callbacks'
/var/www/discourse/vendor/bundle/ruby/4.0.0/gems/concurrent-ruby-1.3.8/lib/concurrent-ruby/concurrent/promises.rb:692:in 'Concurrent::Promises::AbstractEventFuture#resolve_with'
/var/www/discourse/vendor/bundle/ruby/4.0.0/gems/concurrent-ruby-1.3.8/lib/concurrent-ruby/concurrent/promises.rb:1331:in 'Concurrent::Promises::ResolvableEvent#resolve'
/var/www/discourse/lib/scheduler/defer.rb:135:in 'block in Scheduler::Deferrable#do_work'
/var/www/discourse/vendor/bundle/ruby/4.0.0/gems/rails_multisite-9.0.0/lib/rails_multisite/connection_management/null_instance.rb:54:in 'RailsMultisite::ConnectionManagement::NullInstance#with_connection'
/var/www/discourse/vendor/bundle/ruby/4.0.0/gems/rails_multisite-9.0.0/lib/rails_multisite/connection_management.rb:23:in 'RailsMultisite::ConnectionManagement.with_connection'
/var/www/discourse/lib/scheduler/defer.rb:130:in 'Scheduler::Deferrable#do_work'
/var/www/discourse/lib/scheduler/defer.rb:116:in 'block (2 levels) in Scheduler::Deferrable#start_thread'