I’m trying to use the groq moonshotai/kimi-k2-instruct model. According to the documentation on moonshotai/Kimi-K2-Instruct · Hugging Face this model isnt’ compatible with OpenAI or Gemini tokenizers and it appears to use it’s own custom tokenizer. Is it possible to configure discourse to use a cus…

TL;DR pick the closest tokenizer and set the maximum context to some thousands less to allow for the difference to not affect you.

Thanks. So I decided to enroll the services of ChatGPT, Gemini and Grok to help me decided which tokenizer to use; which would be the closest match to the Kimi Instruct TikToken/BPE tokenizer to generate the most accurate output from the model. I must say modern AI models are fairly representative …

Also worth noting … majority of coding agents these days don’t even bother with an accurate tokenizer like Discourse does. They just estimate at 4 letters per token. cl100k will be plenty fine for the vast majority of use cases on llms with slightly different tokenizers.

Wie man AI-Modelle mit benutzerdefinierten Tokenizern verwendet

Unterstützung

Falco (Falco) 14. Dezember 2025 um 00:02 2

TL;DR wählen Sie den nächstgelegenen Tokenizer und setzen Sie den maximalen Kontext auf einige Tausend weniger, damit der Unterschied Sie nicht beeinträchtigt.

Thema		Antworten	Aufrufe
Adding Semantic Search feature for our self-hosted discourse site Support ai , ai-search	9	253	19. März 2025
Frustrations on AI spam detector Support spam , ai	9	137	21. Dezember 2025
Configuring OpenRouter language models Integrations ai	0	1033	10. Dezember 2024
How to implement Mistral with Embeddings Support related-topics , ai	6	228	11. Mai 2025
Inquiry About AI Plugin Options Support ai	7	133	24. November 2025

Wie man AI-Modelle mit benutzerdefinierten Tokenizern verwendet

Verwandte Themen