I’m trying to use the groq moonshotai/kimi-k2-instruct model. According to the documentation on moonshotai/Kimi-K2-Instruct · Hugging Face this model isnt’ compatible with OpenAI or Gemini tokenizers and it appears to use it’s own custom tokenizer. Is it possible to configure discourse to use a cus…

TL;DR pick the closest tokenizer and set the maximum context to some thousands less to allow for the difference to not affect you.

Thanks. So I decided to enroll the services of ChatGPT, Gemini and Grok to help me decided which tokenizer to use; which would be the closest match to the Kimi Instruct TikToken/BPE tokenizer to generate the most accurate output from the model. I must say modern AI models are fairly representative …

Also worth noting … majority of coding agents these days don’t even bother with an accurate tokenizer like Discourse does. They just estimate at 4 letters per token. cl100k will be plenty fine for the vast majority of use cases on llms with slightly different tokenizers.

사용자 지정 토크나이저와 AI 모델 사용하기

지원

Falco (Falco) 12월 14, 2025, 12:02오전 2

TL;DR 가장 가까운 토크나이저를 선택하고, 최대 컨텍스트를 수천 토큰 정도 줄여서 그 차이가 영향을 주지 않도록 하세요.

글		댓글	조회수
Adding Semantic Search feature for our self-hosted discourse site Support ai , ai-search	9	377	3월 19, 2025
Frustrations on AI spam detector Support ai , spam	8	269	11월 21, 2025
Random discussion about Discourse LLM setup Support	21	124	8월 20, 2026
AI exceeds LLM token thresholds randomly and unpredictably Support ai	3	176	5월 6, 2026
Configuring OpenRouter language models Integrations ai	0	1934	12월 10, 2024

사용자 지정 토크나이저와 AI 모델 사용하기

Related topics