This may actually be a bug in the official tokenizer!!
Our counting totally lines up!
Also… looking at token counts, zalgo text is 1 hell of an attack on AI, given it inflates token counts for so little value.
@piffy any chance you can paste in the exact text you had into https://platform.openai.com/tokenizer to see if token counts line up with what the API says, there may be a repro for Open AI here.