Discourse AI - AI triage

[kuaza]

@pfaffman

The likelihood that ai can do this well is slim. If the person writing the topic doesn’t know enough about the community to know what topic it should go in, or what a reasonable title is, it’s unlikely that they be posting anything valuable to the community.

Where does it stop for you? Why make users write anything at all? Shouldn’t the AI generate the topics and the replies?

Users who will produce valuable content for the community can see these areas, but new members generally do not bother with titles, categories or tags. In a Q&A community, all that is expected is to ask questions and get answers (for example, like an AI bot). I want this request for users who are more novice and looking for answers to their urgent questions, but at the same time, these questions should be published generally. When AI is used, such topics do not benefit the community and remain in the background. It would be more useful to enrich the site content and to have them in general use for the same questions.

I also create titles and tags with AI and it is very successful in this regard. I am sure that with a good prompt, I think AI will give good results in terms of categories and tags. Of course, this does not mean leaving it like that, the authorities can re-check such topics and make the necessary arrangements, but we can save time by giving control of this to AI in the first place. Content that is not in the right category can be reported or an authority can correct it when they see it. The aim is to make this job easier.

Actually, I can do this with a simple webhook. A new topic is opened, the given page is pinged, the correct category (on the site) and tags are selected with artificial intelligence, and the topic is updated (with API). However, it would be logical to develop a feature that already exists.

---

@Jagster

For that is one answer: AI is bad at those.

You have somewhat romantic idea what AI is, what it can do, or how a human can interact with it.

But there is such semi-automatic function: AI-helper. You can try it. But there is a reason why even here is relatively little titles by AI

What it can do, again: relatively good, is proofreading and translating. Not so big suprise, because those are language models working with propabilities. But even then it struggles when it should guess used language.

Giving all semantic powers to AI would lead to more human job in background. I live in dream world in administration way. I don’t have problematic users and I don’t have spam. And yet I got false positives — quite rarely, but there is. And when trying to fix that, ”administrators or authorized users can edit it and give feedback to AI” is easier to write than do.

My way to write is sometimes hard to AI, that knows the best English. I try to keep structure of English right, but just how and in which order I should use places and times, is something I can’t. When an AI hits in a situation where I write Finnish using English words, it corrects me sometimes totally wrong and quite often misses the context and point.

I mean what are possibilities of an AI when it gets a sentence where is just hieroglyphs, aka. smileys and reactions? Or when a topic is just a title? Classification of such ones in right categories and tags is close to impossible to an AI.

AI works fine, if it succees rate is 85-90 %, and fixing those remaining 10-15% doesn’t cost too much. That is acceptable. But if a moderator do as many errors, admin will change moderation team quite fast. At same time it means that out of 1000 posts 100-150 is in wrong category with wrong tags and titles are after hard hallucination.

Chat GPT is more expensive AI tool that any of us uses in forums. It gave an solution to one question of Varnish Cache. It looked right, but it didn’t work. I asked help from Chat GPT in the first place only to save time, bacause I was too lazy to write needed if-structure by myself.

The reason was Chat GPT couldn’t see the difference between County-Code and Country-Code. Chatting with Chat GPT and then fixing the issue took more time than I would spend doing it from scratch by myself.

Why I use it? Because of that 85% success rate.

But when it can’t tell how many Rs are in word strawberry, we would like to use it working with words and contexts — because it can do it in the most of time. But when it fails, and it does mass job… fixing can and will be quite demanding, because we have other tech too.

AI messed my summaries, and at same time translates. Easy peasy to delete those records. But creating new ones costs me 100 bucks. What if it would, and it would if possible, put all topics in wrong categories with wrong tags? Fixable, but costly error.

AIs aren’t there yet, if ever, that those could do ”everything”. That is more or less scifi. But those can do some some jobs, when there is limits and well structured guidelines, relatively well. But dealing with hazy reality… no.

I like an idea of categoryless forum, where tags are logical based the human needs to find information, and everything is automated, though.

I agree with what you wrote, then let’s develop AI and run it like an agent and let the AI ​​make suggestions about opened or edited topics. For example, suitable categories, tags, etc. for an opened topic. If the only change that needs to be made is to approve it with a button and the topic is updated immediately. If not, make the edit and approve it quickly.

This is likely to be an extra cost, but it can be a good tool for administrators who want to make their jobs easier and not give up human control. An AI that monitors the activities on the forum and informs the administrator or the people in the selected group about them. Hmmmm, it doesn’t seem like a bad idea (a bit off topic of course)