AI 스팸 탐지기에 대한 불만

[NateDhaliwal]

Hi there, to answer your question:

@lilydjwg

What does the tokenizer do here?

From the docs:

@Discourse

What is tokenizer?

• The tokenizer translates strings into tokens, which is what a model uses to understand the input.