Разочарования от детектора спама на базе ИИ

[NateDhaliwal]

Hi there, to answer your question:

@lilydjwg

What does the tokenizer do here?

From the docs:

@Discourse

What is tokenizer?

• The tokenizer translates strings into tokens, which is what a model uses to understand the input.