Как использовать модели ИИ с пользовательскими токенизаторами

[Falco]

TL;DR pick the closest tokenizer and set the maximum context to some thousands less to allow for the difference to not affect you.