Something bad with embeddings (related topics getting too costly)

Don · December 4, 2024, 11:40am

Hello

I use text-embedding-3-large for ai embeddings model and something wrong with it. I mean, I have to top-up my OpenAI account now twice, since (30 Nov) which is crazy because it should be enough for months… Is that anything changed with related topics? It maybe backfills topics always which already done or I don’t know.

It generates ~ 24 million input tokens / day

Before (30 Nov) it was ~ 60 - 220k

Falco · December 4, 2024, 3:40pm

Please share the values of all embeddings settings:

ai_embeddings_enabled
ai_embeddings_discourse_service_api_endpoint
ai_embeddings_discourse_service_api_endpoint_srv
ai_embeddings_discourse_service_api_key
ai_embeddings_model
ai_embeddings_per_post_enabled
ai_embeddings_generate_for_pms
ai_embeddings_semantic_related_topics_enabled
ai_embeddings_semantic_related_topics
ai_embeddings_semantic_related_include_closed_topics
ai_embeddings_backfill_batch_size
ai_embeddings_semantic_search_enabled
ai_embeddings_semantic_search_hyde_model
ai_embeddings_semantic_search_hyde_model_allowed_seeded_models
ai_embeddings_semantic_quick_search_enabled

Don · December 4, 2024, 3:51pm

ai_embeddings_enabled: true
ai_embeddings_discourse_service_api_endpoint: ""
ai_embeddings_discourse_service_api_endpoint_srv: ""
ai_embeddings_discourse_service_api_key: ""
ai_embeddings_model: text-embedding-3-large
ai_embeddings_per_post_enabled: false
ai_embeddings_generate_for_pms: false
ai_embeddings_semantic_related_topics_enabled: true
ai_embeddings_semantic_related_topics: 5
ai_embeddings_semantic_related_include_closed_topics: true
ai_embeddings_backfill_batch_size: 250
ai_embeddings_semantic_search_enabled: true
ai_embeddings_semantic_search_hyde_model: Gemini 1.5 Flash
ai_embeddings_semantic_search_hyde_model_allowed_seeded_models: ""
ai_embeddings_semantic_quick_search_enabled: false

Falco · December 4, 2024, 3:55pm

How many embeddings do you have?

SELECT COUNT(*) FROM ai_topic_embeddings WHERE model_id = 7;

How many topics do you have?

SELECT COUNT(*) FROM topics WHERE deleted_at IS NULL AND archetype = 'regular';

Don · December 4, 2024, 4:14pm

How many embeddings do you have?
5964

How many topics do you have?
5563

Jagster · December 4, 2024, 4:22pm

I checked mine. It exploded 27.11. and before that it was under 100k tokens a day, but then it increased to 7 million and is increasing everyday and yesterday it was close to 20 million.

Edit: October, embeddings cost was 46 cents. Now, December close to four days: almost 6 dollars.

Yeah. I disabled embeddings.

Falco · December 4, 2024, 6:57pm

24M a day is your entire forum, that looks buggy. Unless you get updates in all those topics every day that is most certainly a bug.

Falco · December 4, 2024, 7:44pm

One thing that may be related is that we used to skip calling embeddings API when the topic digest didn’t change, but we regressed this on gen_bulk_reprensentations @Roman.

@Don do you know how many embeddings requests are you making per day?

Jagster · December 4, 2024, 8:05pm

I’m not Don, but my API requests has increased from 80-100 to 3825.

Don · December 4, 2024, 8:15pm

It generally ~150 - 200 requests / day

but at the end of november it increased.

Roman · December 4, 2024, 8:51pm

I’m really sorry, this was a bug in the new code we added to backfill embeddings faster. It should be fixed by:

https://github.com/discourse/discourse-ai/pull/1001

Please let me know if things are not back to normal.

Falco · December 4, 2024, 8:59pm

Given the 250 per hour, we have a hard limit of 6k per day. This numbers are still inside the limit.

However, if they are only getting triggered by our “update a random sample” of topics, it should be limited to 10% of that, which would, at worse, be 600 requests.

@Roman is this limit here not getting applied somehow? Or is the problem elsewhere?

github.com/discourse/discourse-ai

app/jobs/scheduled/embeddings_backfill.rb

main


      
          
          # Then, we'll try to backfill embeddings for topics that have outdated
          # embeddings, be it model or strategy version
          relation = topics.where(<<~SQL).limit(limit - rebaked)
              #{table_name}.model_version < #{vector_def.version}
              OR
              #{table_name}.strategy_version < #{vector_def.strategy_version}

Roman · December 4, 2024, 9:09pm

Yeah, I think the bug I fixed revealed another one that the digest check was hiding.

I think the bug is here:

github.com/discourse/discourse-ai

app/jobs/scheduled/embeddings_backfill.rb

main


      
              .where("#{table_name}.updated_at < ?", 6.hours.ago)
              .where("#{table_name}.updated_at < topics.updated_at")
              .limit((limit - rebaked) / 10)
          
          populate_topic_embeddings(vector, relation, force: true)
          
          next unless SiteSetting.ai_embeddings_per_post_enabled
          
          # Now for posts
          table_name = DiscourseAi::Embeddings::Schema::POSTS_TABLE
          posts_batch_size = 1000
          
          posts =
            Post
              .joins(
                "LEFT JOIN #{table_name} ON #{table_name}.post_id = posts.id AND #{table_name}.model_id = #{vector_def.id}",
              )
              .where(deleted_at: nil)
              .where(post_type: Post.types[:regular])
          
          # First, we'll try to backfill embeddings for posts that have none

I changed it from find_each to find_in_batches last week (the former uses batches internally), and since both rely on limit to specify batch size, the original limit of limit - rebaked is ignored. We should use pluck + each_slice instead.

Don · December 4, 2024, 11:37pm

Thanks for the fix

I’ve updated my site but it looks like there is an issue in /logs. I am not sure is it related with this…

Message

Job exception: ERROR:  invalid input syntax for type halfvec: "[NULL]"
LINE 2: ...1, 1, 'e358a54a79f71861a4ebd17ecebbad6932fc1f9a', '[NULL]', ...
                                                             ^


Backtrace

/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rack-mini-profiler-3.3.1/lib/patches/db/pg.rb:110:in `exec'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rack-mini-profiler-3.3.1/lib/patches/db/pg.rb:110:in `async_exec'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/mini_sql-1.6.0/lib/mini_sql/postgres/connection.rb:217:in `run'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/mini_sql-1.6.0/lib/mini_sql/active_record_postgres/connection.rb:38:in `block in run'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/mini_sql-1.6.0/lib/mini_sql/active_record_postgres/connection.rb:34:in `block in with_lock'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2/lib/active_support/concurrency/null_lock.rb:9:in `synchronize'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/mini_sql-1.6.0/lib/mini_sql/active_record_postgres/connection.rb:34:in `with_lock'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/mini_sql-1.6.0/lib/mini_sql/active_record_postgres/connection.rb:38:in `run'
/var/www/discourse/lib/mini_sql_multisite_connection.rb:109:in `run'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/mini_sql-1.6.0/lib/mini_sql/postgres/connection.rb:196:in `exec'
/var/www/discourse/plugins/discourse-ai/lib/embeddings/vector_representations/base.rb:423:in `save_to_db'
/var/www/discourse/plugins/discourse-ai/lib/embeddings/vector_representations/base.rb:86:in `block in gen_bulk_reprensentations'
/var/www/discourse/plugins/discourse-ai/lib/embeddings/vector_representations/base.rb:86:in `each'
/var/www/discourse/plugins/discourse-ai/lib/embeddings/vector_representations/base.rb:86:in `gen_bulk_reprensentations'
/var/www/discourse/plugins/discourse-ai/app/jobs/scheduled/embeddings_backfill.rb:131:in `block in populate_topic_embeddings'
/var/www/discourse/plugins/discourse-ai/app/jobs/scheduled/embeddings_backfill.rb:130:in `each'
/var/www/discourse/plugins/discourse-ai/app/jobs/scheduled/embeddings_backfill.rb:130:in `each_slice'
/var/www/discourse/plugins/discourse-ai/app/jobs/scheduled/embeddings_backfill.rb:130:in `populate_topic_embeddings'
/var/www/discourse/plugins/discourse-ai/app/jobs/scheduled/embeddings_backfill.rb:36:in `execute'
/var/www/discourse/app/jobs/base.rb:308:in `block (2 levels) in perform'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-6.1.0/lib/rails_multisite/connection_management/null_instance.rb:49:in `with_connection'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-6.1.0/lib/rails_multisite/connection_management.rb:21:in `with_connection'
/var/www/discourse/app/jobs/base.rb:295:in `block in perform'
/var/www/discourse/app/jobs/base.rb:291:in `each'
/var/www/discourse/app/jobs/base.rb:291:in `perform'
/var/www/discourse/app/jobs/base.rb:362:in `perform'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/mini_scheduler-0.17.0/lib/mini_scheduler/manager.rb:137:in `process_queue'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/mini_scheduler-0.17.0/lib/mini_scheduler/manager.rb:77:in `worker_loop'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/mini_scheduler-0.17.0/lib/mini_scheduler/manager.rb:63:in `block (2 levels) in ensure_worker_threads'

Roman · December 4, 2024, 11:51pm

At first glance, it doesn’t look related. Looks like it failed to generate the embedding and it’s trying to insert NULL. Could it be OpenAI is returning an error? Maybe something quota related?

Can you please run this from a console?

DiscourseAi::Embeddings::VectorRepresentations::Base
          .find_representation(SiteSetting.ai_embeddings_model)
          .new(DiscourseAi::Embeddings::Strategies::Truncation.new)
          .vector_from("this is a test")
          .present?

It should log the error in your logs if it raises a Net::HTTPBadResponse.

Don · December 5, 2024, 12:02am

I got back in console: truet? and nothing in /logs.

Maybe this is a delay from OpanAI because I top up my account again an hour ago and probably this process not instant…

Roman · December 5, 2024, 1:00am

That means it can generate embeddings then. Do these errors persist? You should see these errors every five minutes if so.

I ran some tests on my local instance against our self-hosted embeddings service and confirmed that backfilling works under the following conditions:

There are no embeddings.
The digest is outdated and the embeddings’ updated_at is older than 6 hours.
The digest is not outdated and the embeddings’ updated_at is older than 6 hours (it does not update in this case).

Don · December 5, 2024, 6:21am

Nope, I don’t see those errors in /logs anymore, everything works now. Thank you

Falco · December 5, 2024, 7:12pm

We merged another fix 5h ago, please update again.

https://github.com/discourse/discourse-ai/pull/1002

After that, please let me know how the rate is looking.

cc @Jagster.

Jagster · December 5, 2024, 7:16pm

I don’t know anything about limits, but amount of API requests etc. dropped back to normal after the earlier fix. So thanks guys for fast reaction.

Topic		Replies	Views
"Net::HTTPBadResponse" errors on Gemini Embeddings Bug ai , related-topics	12	772	February 29, 2024
OpenAI embedding related topics stops working Product Support ai , related-topics	3	621	March 10, 2024
Ai:embeddings:backfill - Handling OpenAI's 400 Error for Excessive Tokens in Embeddings Bug ai	9	1044	March 15, 2024
Problem with the new Discourse AI "related / similar topics"-function Product Support ai , related-topics	5	1132	August 21, 2023
Embeddings search still running error Bug ai , ai-search	9	224	November 11, 2026

Something bad with embeddings (related topics getting too costly)

AI-generated summary

Related topics