م within Discourse AI - التضمينات

[Falco]

@Overgrow

Do I understand it correctly that backfill is when the vectorization happens?

Yes.

@Overgrow

When switching between models, do the vectors need to be recalculated

Yes, each model produces different vector representations.

@Overgrow

It’d be useful to know how the costs of using the OpenAI API stack up against investing in a GPU-powered server with opensource solution

It’s basically one call per topic, so very easy to calculate.

@Overgrow

Is there a formula or any way to estimate the number of tokens used?

If most of your topics are long, they will be truncated to 8k tokens, otherwise they will use your topic length.

@Overgrow

We’re only using the API to vectorize posts, not for calculating vector distances, right?

Yes.

@Overgrow

I assume that for both related topics and AI-powered search, all posts need to be vectorized only once, so I can calculate the total number of words in posts table and derive the number of tokens needed. The same process would apply to the daily addition of posts. I’m neglecting the search phrases for now.

Both work at the topic level, so one per topic.

[satonotdead]

@falco

I’ve left those out unintentionally as the rake task fell out out favor and should only be used in rare occasions by experienced operators who can easily install those out of band.

May I know how to properly add the gems without forking the plugin with the suggested PR?

I’m trying the scale-to-zero feature on HuggingFace and I just need to use the rake task for backfill embeddings.

[jlcoo (Jiang Long)]

why return 418 error code when I using discourse ai embeddings full search in DiscourseAi::Embeddings::EmbeddingsController#search as JSON? Could you help me?

[RBoy]

Something happened after I updated discourse to 3.6.0 beta 2, embeddings stopped working and I can’t seem to figure out why.

I’m using Gemini Flash and Flash Lite LLM’s and they’re working perfectly, the AI bot is working etc. However when I search something it gives me a 500 error.

When I goto the Embedding model and run test on the gemini embedding, I get this error (and the logs are also full os it)

 Trying to contact the model returned this error: { "error": { "code": 429, "message": "You exceeded your current quota, please check your plan and billing details. For more information on this error, head to: https://ai.google.dev/gemini-api/docs/rate-limits.\n* Quota exceeded for metric: generativelanguage.googleapis.com/embed_content_free_tier_requests, limit: 0\n* Quota exceeded for metric: generativelanguage.googleapis.com/embed_content_free_tier_requests, limit: 0\n* Quota exceeded for metric: generativelanguage.googleapis.com/embed_content_free_tier_requests, limit: 0\n* Quota exceeded for metric: generativelanguage.googleapis.com/embed_content_free_tier_requests, limit: 0", "status": "RESOURCE_EXHAUSTED", "details": [ { "@type": "type.googleapis.com/google.rpc.QuotaFailure", "violations": [ { "quotaMetric": "generativelanguage.googleapis.com/embed_content_free_tier_requests", "quotaId": "EmbedContentRequestsPerMinutePerProjectPerModel-FreeTier" }, { "quotaMetric": "generativelanguage.googleapis.com/embed_content_free_tier_requests", "quotaId": "EmbedContentRequestsPerMinutePerUserPerProjectPerModel-FreeTier" }, { "quotaMetric": "generativelanguage.googleapis.com/embed_content_free_tier_requests", "quotaId": "EmbedContentRequestsPerDayPerUserPerProjectPerModel-FreeTier" }, { "quotaMetric": "generativelanguage.googleapis.com/embed_content_free_tier_requests", "quotaId": "EmbedContentRequestsPerDayPerProjectPerModel-FreeTier" } ] }, { "@type": "type.googleapis.com/google.rpc.Help", "links": [ { "description": "Learn more about Gemini API quotas", "url": "https://ai.google.dev/gemini-api/docs/rate-limits" } ] } ] } }

This is working perfectly till 3.6.0 beta 1, after updating to beta 2 it started giving this error. I’m using the same key with this embedding as I’m with the LLM. I’ve even tried to generate a new key, waited 48 hours etc and nothing seems to fix this.

Can someone advise me on why it suddenly stopped working, why the LLM AI Bot is working but not this and how do I fix it?

[RBoy]

I’m going to answer my own post. The problem is not the build but the timing. In Oct google made the current embedding model (embedding-001) obsolete so it stopped working. The new model is gemini-embedding-001

In the Embedding configuration (Admin->Plugins->Discourse AI->Embeddings->Edit), you need to change the Embeddings service URL from
https://generativelanguage.googleapis.com/v1beta/models/embedding-001:embedContent

to

https://generativelanguage.googleapis.com/v1beta/models/gemini-embedding-001:embedContent

You can find the details here: https://ai.google.dev/gemini-api/docs/embeddings

@Falco maybe the default gemini configuration needs to be updated in discourse, when creating a new gemini embedding it still uses embedding-001 instead of gemini-embedding-001

[RBoy]

…and it appears that discourse may not be setup to handle the new gemini-embedding-001 model. The Run test works fine but when I try to search the forum I now see these errors in the logs:

Message (2 copies reported)

Error ERROR:  expected 768 dimensions, not 3072
 querying embeddings for model gemini

Backtrace

/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.3/lib/active_support/broadcast_logger.rb:218:in `block in dispatch'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.3/lib/active_support/broadcast_logger.rb:217:in `map'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.3/lib/active_support/broadcast_logger.rb:217:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.3/lib/active_support/broadcast_logger.rb:129:in `error'
/var/www/discourse/plugins/discourse-ai/lib/embeddings/schema.rb:192:in `rescue in asymmetric_similarity_search'
/var/www/discourse/plugins/discourse-ai/lib/embeddings/schema.rb:139:in `asymmetric_similarity_search'
/var/www/discourse/plugins/discourse-ai/lib/embeddings/semantic_search.rb:89:in `search_for_topics'
/var/www/discourse/plugins/discourse-ai/app/controllers/discourse_ai/embeddings/embeddings_controller.rb:56:in `block in search'
/var/www/discourse/lib/hijack.rb:68:in `instance_eval'
/var/www/discourse/lib/hijack.rb:68:in `block (2 levels) in hijack'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/i18n-1.14.7/lib/i18n.rb:353:in `with_locale'
/var/www/discourse/lib/hijack.rb:68:in `block in hijack'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/concurrent-ruby-1.3.5/lib/concurrent-ruby/concurrent/promises.rb:911:in `callback_on_resolution'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/concurrent-ruby-1.3.5/lib/concurrent-ruby/concurrent/promises.rb:797:in `call_callback'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/concurrent-ruby-1.3.5/lib/concurrent-ruby/concurrent/promises.rb:803:in `call_callbacks'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/concurrent-ruby-1.3.5/lib/concurrent-ruby/concurrent/promises.rb:692:in `resolve_with'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/concurrent-ruby-1.3.5/lib/concurrent-ruby/concurrent/promises.rb:1325:in `resolve'
/var/www/discourse/lib/scheduler/defer.rb:125:in `block in do_work'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management/null_instance.rb:49:in `with_connection'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management.rb:17:in `with_connection'
/var/www/discourse/lib/scheduler/defer.rb:119:in `do_work'
/var/www/discourse/lib/scheduler/defer.rb:105:in `block (2 levels) in start_thread'

Message (2 copies reported)

Failed to process hijacked response correctly : DiscourseAi::Embeddings::Schema::MissingEmbeddingError : DiscourseAi::Embeddings::Schema::MissingEmbeddingError

Backtrace

/var/www/discourse/plugins/discourse-ai/lib/embeddings/schema.rb:193:in `rescue in asymmetric_similarity_search'
/var/www/discourse/plugins/discourse-ai/lib/embeddings/schema.rb:139:in `asymmetric_similarity_search'
/var/www/discourse/plugins/discourse-ai/lib/embeddings/semantic_search.rb:89:in `search_for_topics'
/var/www/discourse/plugins/discourse-ai/app/controllers/discourse_ai/embeddings/embeddings_controller.rb:56:in `block in search'
/var/www/discourse/lib/hijack.rb:68:in `instance_eval'
/var/www/discourse/lib/hijack.rb:68:in `block (2 levels) in hijack'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/i18n-1.14.7/lib/i18n.rb:353:in `with_locale'
/var/www/discourse/lib/hijack.rb:68:in `block in hijack'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/concurrent-ruby-1.3.5/lib/concurrent-ruby/concurrent/promises.rb:911:in `callback_on_resolution'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/concurrent-ruby-1.3.5/lib/concurrent-ruby/concurrent/promises.rb:797:in `call_callback'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/concurrent-ruby-1.3.5/lib/concurrent-ruby/concurrent/promises.rb:803:in `call_callbacks'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/concurrent-ruby-1.3.5/lib/concurrent-ruby/concurrent/promises.rb:692:in `resolve_with'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/concurrent-ruby-1.3.5/lib/concurrent-ruby/concurrent/promises.rb:1325:in `resolve'
/var/www/discourse/lib/scheduler/defer.rb:125:in `block in do_work'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management/null_instance.rb:49:in `with_connection'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management.rb:17:in `with_connection'
/var/www/discourse/lib/scheduler/defer.rb:119:in `do_work'
/var/www/discourse/lib/scheduler/defer.rb:105:in `block (2 levels) in start_thread'

…and what seems to work is to create a new model and change the Embedding dimensions to 3072 instead of the default 768.

Possibly discourse gemini embeddings may need an overhaul for it’s default settings and a small ‘How to configure gemini embedding’ on the first post may also be a good idea.