Warning of embedding `input must have less than 8192 tokens` with discourse ai

תמיכה
1

is there any tool to split input text into chunks?
or which embedding api support more than 8192 tokens?
i am using Qwen/Qwen3-Embedding-0.6B

Message (15 copies reported)

OpenAI Embeddings failed with status: 413 body: {"code":20042,"message":"input must have less than 8192 tokens","data":null}

Backtrace

/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.3/lib/active_support/broadcast_logger.rb:218:in `block in dispatch'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.3/lib/active_support/broadcast_logger.rb:217:in `map'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.3/lib/active_support/broadcast_logger.rb:217:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.3/lib/active_support/broadcast_logger.rb:129:in `warn'
/var/www/discourse/plugins/discourse-ai/lib/inference/open_ai_embeddings.rb:34:in `perform!'
/var/www/discourse/plugins/discourse-ai/lib/embeddings/vector.rb:45:in `block (2 levels) in gen_bulk_reprensentations'
/var/www/discourse/lib/scheduler/thread_pool.rb:118:in `block (2 levels) in wrap_block'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management/null_instance.rb:49:in `with_connection'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management.rb:17:in `with_connection'
/var/www/discourse/lib/scheduler/thread_pool.rb:118:in `block in wrap_block'
/var/www/discourse/lib/scheduler/thread_pool.rb:163:in `thread_loop'
/var/www/discourse/lib/scheduler/thread_pool.rb:174:in `block in spawn_thread'
2

If you self-host that same model, it can take up to 32k tokens. It is what we run on our hosting these days.

If that’s out of the question, then you need to configure the embeddings model to limit inputs to the maximum allowed on your provider. This way, our AI Bot RAG will split uploaded files into chunks, and Related Topic / Search will take only the first 8192 tokens in each topic.