Something bad with embeddings (related topics getting too costly)

Don · December 4, 2024, 11:40am

Hello

I use text-embedding-3-large for ai embeddings model and something wrong with it. I mean, I have to top-up my OpenAI account now twice, since (30 Nov) which is crazy because it should be enough for months… Is that anything changed with related topics? It maybe backfills topics always which already done or I don’t know.

It generates ~ 24 million input tokens / day

Before (30 Nov) it was ~ 60 - 220k

Falco · December 4, 2024, 3:40pm

Please share the values of all embeddings settings:

ai_embeddings_enabled
ai_embeddings_discourse_service_api_endpoint
ai_embeddings_discourse_service_api_endpoint_srv
ai_embeddings_discourse_service_api_key
ai_embeddings_model
ai_embeddings_per_post_enabled
ai_embeddings_generate_for_pms
ai_embeddings_semantic_related_topics_enabled
ai_embeddings_semantic_related_topics
ai_embeddings_semantic_related_include_closed_topics
ai_embeddings_backfill_batch_size
ai_embeddings_semantic_search_enabled
ai_embeddings_semantic_search_hyde_model
ai_embeddings_semantic_search_hyde_model_allowed_seeded_models
ai_embeddings_semantic_quick_search_enabled

Don · December 4, 2024, 3:51pm

ai_embeddings_enabled: true
ai_embeddings_discourse_service_api_endpoint: ""
ai_embeddings_discourse_service_api_endpoint_srv: ""
ai_embeddings_discourse_service_api_key: ""
ai_embeddings_model: text-embedding-3-large
ai_embeddings_per_post_enabled: false
ai_embeddings_generate_for_pms: false
ai_embeddings_semantic_related_topics_enabled: true
ai_embeddings_semantic_related_topics: 5
ai_embeddings_semantic_related_include_closed_topics: true
ai_embeddings_backfill_batch_size: 250
ai_embeddings_semantic_search_enabled: true
ai_embeddings_semantic_search_hyde_model: Gemini 1.5 Flash
ai_embeddings_semantic_search_hyde_model_allowed_seeded_models: ""
ai_embeddings_semantic_quick_search_enabled: false

Falco · December 4, 2024, 3:55pm

How many embeddings do you have?

SELECT COUNT(*) FROM ai_topic_embeddings WHERE model_id = 7;

How many topics do you have?

SELECT COUNT(*) FROM topics WHERE deleted_at IS NULL AND archetype = 'regular';

Don · December 4, 2024, 4:14pm

How many embeddings do you have?
5964

How many topics do you have?
5563

Jagster · December 4, 2024, 4:22pm

I checked mine. It exploded 27.11. and before that it was under 100k tokens a day, but then it increased to 7 million and is increasing everyday and yesterday it was close to 20 million.

Edit: October, embeddings cost was 46 cents. Now, December close to four days: almost 6 dollars.

Yeah. I disabled embeddings.

Falco · December 4, 2024, 6:57pm

24M a day is your entire forum, that looks buggy. Unless you get updates in all those topics every day that is most certainly a bug.

Falco · December 4, 2024, 7:44pm

One thing that may be related is that we used to skip calling embeddings API when the topic digest didn’t change, but we regressed this on gen_bulk_reprensentations @Roman_Rizzi.

@Don do you know how many embeddings requests are you making per day?

Jagster · December 4, 2024, 8:05pm

I’m not Don, but my API requests has increased from 80-100 to 3825.

Don · December 4, 2024, 8:15pm

It generally ~150 - 200 requests / day

but at the end of november it increased.

Roman_Rizzi · December 4, 2024, 8:51pm

I’m really sorry, this was a bug in the new code we added to backfill embeddings faster. It should be fixed by:

Please let me know if things are not back to normal.

Falco · December 4, 2024, 8:59pm

Given the 250 per hour, we have a hard limit of 6k per day. This numbers are still inside the limit.

However, if they are only getting triggered by our “update a random sample” of topics, it should be limited to 10% of that, which would, at worse, be 600 requests.

@Roman_Rizzi is this limit here not getting applied somehow? Or is the problem elsewhere?

github.com

discourse/discourse-ai/blob/main/app/jobs/scheduled/embeddings_backfill.rb#L52-L58


      
          # Finally, we'll try to backfill embeddings for topics that have outdated
          # embeddings due to edits or new replies. Here we only do 10% of the limit
          relation =
            topics
              .where("#{table_name}.updated_at < ?", 6.hours.ago)
              .where("#{table_name}.updated_at < topics.updated_at")
              .limit((limit - rebaked) / 10)

Roman_Rizzi · December 4, 2024, 9:09pm

Yeah, I think the bug I fixed revealed another one that the digest check was hiding.

I think the bug is here:

github.com

discourse/discourse-ai/blob/main/app/jobs/scheduled/embeddings_backfill.rb#L79


      
          posts =
            Post
              .joins("LEFT JOIN #{table_name} ON #{table_name}.post_id = posts.id")
              .where(deleted_at: nil)
              .where(post_type: Post.types[:regular])
              .limit(limit - rebaked)
          
          # First, we'll try to backfill embeddings for posts that have none
          posts
            .where("#{table_name}.post_id IS NULL")
            .find_in_batches do |batch|
              vector_rep.gen_bulk_reprensentations(batch)
              rebaked += batch.size
            end
          
          return if rebaked >= limit
          
          # Then, we'll try to backfill embeddings for posts that have outdated
          # embeddings, be it model or strategy version
          posts
            .where(<<~SQL)

I changed it from find_each to find_in_batches last week (the former uses batches internally), and since both rely on limit to specify batch size, the original limit of limit - rebaked is ignored. We should use pluck + each_slice instead.

Don · December 4, 2024, 11:37pm

Thanks for the fix

I’ve updated my site but it looks like there is an issue in /logs. I am not sure is it related with this…

Message

Job exception: ERROR:  invalid input syntax for type halfvec: "[NULL]"
LINE 2: ...1, 1, 'e358a54a79f71861a4ebd17ecebbad6932fc1f9a', '[NULL]', ...
                                                             ^


Backtrace

/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rack-mini-profiler-3.3.1/lib/patches/db/pg.rb:110:in `exec'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rack-mini-profiler-3.3.1/lib/patches/db/pg.rb:110:in `async_exec'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/mini_sql-1.6.0/lib/mini_sql/postgres/connection.rb:217:in `run'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/mini_sql-1.6.0/lib/mini_sql/active_record_postgres/connection.rb:38:in `block in run'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/mini_sql-1.6.0/lib/mini_sql/active_record_postgres/connection.rb:34:in `block in with_lock'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2/lib/active_support/concurrency/null_lock.rb:9:in `synchronize'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/mini_sql-1.6.0/lib/mini_sql/active_record_postgres/connection.rb:34:in `with_lock'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/mini_sql-1.6.0/lib/mini_sql/active_record_postgres/connection.rb:38:in `run'
/var/www/discourse/lib/mini_sql_multisite_connection.rb:109:in `run'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/mini_sql-1.6.0/lib/mini_sql/postgres/connection.rb:196:in `exec'
/var/www/discourse/plugins/discourse-ai/lib/embeddings/vector_representations/base.rb:423:in `save_to_db'
/var/www/discourse/plugins/discourse-ai/lib/embeddings/vector_representations/base.rb:86:in `block in gen_bulk_reprensentations'
/var/www/discourse/plugins/discourse-ai/lib/embeddings/vector_representations/base.rb:86:in `each'
/var/www/discourse/plugins/discourse-ai/lib/embeddings/vector_representations/base.rb:86:in `gen_bulk_reprensentations'
/var/www/discourse/plugins/discourse-ai/app/jobs/scheduled/embeddings_backfill.rb:131:in `block in populate_topic_embeddings'
/var/www/discourse/plugins/discourse-ai/app/jobs/scheduled/embeddings_backfill.rb:130:in `each'
/var/www/discourse/plugins/discourse-ai/app/jobs/scheduled/embeddings_backfill.rb:130:in `each_slice'
/var/www/discourse/plugins/discourse-ai/app/jobs/scheduled/embeddings_backfill.rb:130:in `populate_topic_embeddings'
/var/www/discourse/plugins/discourse-ai/app/jobs/scheduled/embeddings_backfill.rb:36:in `execute'
/var/www/discourse/app/jobs/base.rb:308:in `block (2 levels) in perform'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-6.1.0/lib/rails_multisite/connection_management/null_instance.rb:49:in `with_connection'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-6.1.0/lib/rails_multisite/connection_management.rb:21:in `with_connection'
/var/www/discourse/app/jobs/base.rb:295:in `block in perform'
/var/www/discourse/app/jobs/base.rb:291:in `each'
/var/www/discourse/app/jobs/base.rb:291:in `perform'
/var/www/discourse/app/jobs/base.rb:362:in `perform'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/mini_scheduler-0.17.0/lib/mini_scheduler/manager.rb:137:in `process_queue'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/mini_scheduler-0.17.0/lib/mini_scheduler/manager.rb:77:in `worker_loop'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/mini_scheduler-0.17.0/lib/mini_scheduler/manager.rb:63:in `block (2 levels) in ensure_worker_threads'

Roman_Rizzi · December 4, 2024, 11:51pm

At first glance, it doesn’t look related. Looks like it failed to generate the embedding and it’s trying to insert NULL. Could it be OpenAI is returning an error? Maybe something quota related?

Can you please run this from a console?

DiscourseAi::Embeddings::VectorRepresentations::Base
          .find_representation(SiteSetting.ai_embeddings_model)
          .new(DiscourseAi::Embeddings::Strategies::Truncation.new)
          .vector_from("this is a test")
          .present?

It should log the error in your logs if it raises a Net::HTTPBadResponse.

Don · December 5, 2024, 12:02am

I got back in console: truet? and nothing in /logs.

Maybe this is a delay from OpanAI because I top up my account again an hour ago and probably this process not instant…

Roman_Rizzi · December 5, 2024, 1:00am

That means it can generate embeddings then. Do these errors persist? You should see these errors every five minutes if so.

I ran some tests on my local instance against our self-hosted embeddings service and confirmed that backfilling works under the following conditions:

There are no embeddings.
The digest is outdated and the embeddings’ updated_at is older than 6 hours.
The digest is not outdated and the embeddings’ updated_at is older than 6 hours (it does not update in this case).

Don · December 5, 2024, 6:21am

Nope, I don’t see those errors in /logs anymore, everything works now. Thank you

Falco · December 5, 2024, 7:12pm

We merged another fix 5h ago, please update again.

After that, please let me know how the rate is looking.

cc @Jagster.

Jagster · December 5, 2024, 7:16pm

I don’t know anything about limits, but amount of API requests etc. dropped back to normal after the earlier fix. So thanks guys for fast reaction.

Topic		Replies	Views
Ai:embeddings:backfill - Handling OpenAI's 400 Error for Excessive Tokens in Embeddings Bug ai	10	791	March 15, 2024
"Net::HTTPBadResponse" errors on Gemini Embeddings Bug ai , related-topics	14	414	February 29, 2024
AI embeddings backfill rake aborted Support ai	4	474	January 30, 2024
Discourse AI - Embeddings Site Management ai , ai-search , related-topics	20	5026	October 1, 2024
Problem with the new Discourse AI "related / similar topics"-function Support ai , related-topics	5	855	August 21, 2023

Something bad with embeddings (related topics getting too costly)

Related topics