I checked the source code. If I didn’t make mistakes, the tokenizer is used for two things: count tokens for statistics and price estimation, and truncate posts to the set limit. So it shouldn’t affect me a lot if a wrong one is used.
lilydjwg
9
Related topics
| Topic | Replies | Views | Activity | |
|---|---|---|---|---|
| AI powered Spam detection | 11 | 1189 | January 11, 2025 | |
| Random discussion about Discourse LLM setup | 21 | 88 | August 20, 2026 | |
| Discourse AI - Spam detection | 30 | 5062 | March 10, 2026 | |
| Introducing Discourse AI | 26 | 4199 | May 4, 2023 | |
| What's the cheapest/best AI to use for AI Spam? | 6 | 442 | March 18, 2025 |