is there any tool to split input text into chunks? or which embedding api support more than 8192 tokens? i am using Qwen/Qwen3-Embedding-0.6B Message (15 copies reported) OpenAI Embeddings failed with status: 413 body: {"code":20042,"message":"input must have less than 8192 tokens","data":null} Backtrace /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.3/lib/active_support/broadcast_logger.rb:218:in `block in dispatch' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.3/lib/active_support/broadcast_logger.rb:217:in `map' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.3/lib/active_support/broadcast_logger.rb:217:in `dispatch' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.3/lib/active_support/broadcast_logger.rb:129:in `warn' /var/www/discourse/plugins/discourse-ai/lib/inference/open_ai_embeddings.rb:34:in `perform!' /var/www/discourse/plugins/discourse-ai/lib/embeddings/vector.rb:45:in `block (2 levels) in gen_bulk_reprensentations' /var/www/discourse/lib/scheduler/thread_pool.rb:118:in `block (2 levels) in wrap_block' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management/null_instance.rb:49:in `with_connection' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management.rb:17:in `with_connection' /var/www/discourse/lib/scheduler/thread_pool.rb:118:in `block in wrap_block' /var/www/discourse/lib/scheduler/thread_pool.rb:163:in `thread_loop' /var/www/discourse/lib/scheduler/thread_pool.rb:174:in `block in spawn_thread'

If you self-host that same model, it can take up to 32k tokens. It is what we run on our hosting these days. If that’s out of the question, then you need to configure the embeddings model to limit inputs to the maximum allowed on your provider. This way, our AI Bot RAG will split uploaded files into chunks, and Related Topic / Search will take only the first 8192 tokens in each topic.

how to configure embedding model? in discourse ?or model provider?

i am sure Qwen/Qwen3-Embedding-0.6B support up to 32k tokens and i have configured sequence length 32k [image] [image] but there still are many error of embedding [image] [image] [image]

[imagem] whitewaterdeu: i am sure Qwen/Qwen3-Embedding-0.6B support up to 32k tokens It does if your API providers allows it. [imagem] whitewaterdeu: but there still are many error of embedding If your API provider limits it to 8192, you gotta follow their rules.

Aviso de incorporação `input must have less than 8192 tokens` com discourse ai

Suporte ao Produto

whitewaterdeu (mocha3901) 3 Novembro , 2025 08:10 6

ok, estou perguntando ao provedor da API

Tópico		Respostas	Vistas
Embedding of post is not being properly truncated in discourse-ai plugin Bug ai	4	493	17 Agosto , 2023
Ai:embeddings:backfill - Handling OpenAI's 400 Error for Excessive Tokens in Embeddings Bug ai	9	1037	15 Março , 2024
AI exceeds LLM token thresholds randomly and unpredictably Product Support ai	3	248	6 Maio , 2026
Adding Semantic Search feature for our self-hosted discourse site Product Support ai , ai-search	9	488	19 Março , 2025
Gemini API Embedding Configuration Clarification Product Support ai	3	605	16 Outubro , 2025

Aviso de incorporação `input must have less than 8192 tokens` com discourse ai

Related topics