How to use AI models with custom tokenizers

תמיכה
2

TL;DR pick the closest tokenizer and set the maximum context to some thousands less to allow for the difference to not affect you.