I checked the source code. If I didn’t make mistakes, the tokenizer is used for two things: count tokens for statistics and price estimation, and truncate posts to the set limit. So it shouldn’t affect me a lot if a wrong one is used.
lilydjwg
9
Related topics
| Topic | Replies | Views | Activity | |
|---|---|---|---|---|
| AI powered Spam detection | 11 | 1266 | January 11, 2025 | |
| Random discussion about Discourse LLM setup | 21 | 189 | August 20, 2026 | |
| Introducing Discourse AI | 26 | 4317 | May 4, 2023 | |
| What's the cheapest/best AI to use for AI Spam? | 6 | 523 | March 18, 2025 | |
| Discourse AI spam detection replaces Akismet plugin | 13 | 1355 | November 4, 2025 |