I’m looking for ways to integrate AI into my Discourse forum to assist with moderation. I don’t need it to replace my moderators, I just need help catching things humans usually can’t see. Sometimes, it’s because these issues are literally invisible to a moderator (like a spammer who is creating multiple accounts from the same IP address). Other times, it is visible to a moderator, but it’s easy to get lazy and miss these things (like a topic that is posted in the wrong category, or a topic that is veering off course).
There are endless tasks an AI moderator could help with. Just a few ideas off the top of my head:
Monitoring all new posts to indicate the likelihood of whether they’re spammer or legitimate users.
Monitoring new users and their activity until they’ve reached a certain trust level.
Catching problem users making new accounts after being suspended.
Identifying topics that have been posted in the wrong category, and offering suggestions for which category they should be moved to.
Flagging and immediately removing NSFW content.
Identifying when the conversation in a topic is veering off-course or should be locked.
Identifying when a topic has already been covered and should be redirected.
Identifying when a user has created multiple accounts (multiple users logging in from the same IP address).
Identifying when a user is making a self-promotional or irrelevant post.
Not to mention (and this would be going in a slightly different direction), there are times when AI could even respond to certain topics with a clearly marked AI profile. For example, if someone posts a question about how to use the forum or where to find a certain feature (like how to update their profile), the bot could respond by identifying when it’s a question it could easily answer, and then it could jump in and explain how to do it.
I’m barely even scratching the surface here, but the underlying question is: Has anyone created an AI bot that can assist with these types of moderation tasks in Discourse?
If not, what’s holding this kind of innovation back? This seems like it would be insanely useful for forum admins, not to replace humans (although that may be possible in some cases), but to help humans do the job a lot better.
I don’t know but I would guess: AI is very unrealiable, and can be very fast very expensive.
Some of those options are already possible, though. And not in common use mainly because, well, AI is unrealiable and needs a human watching it.
Answering machine that is following every posts and jump in when triggered by content must be expensive in matter of hardware and pure money. But a model that answer on categorylevel to every topic starters is already possible.
Then there is something like watching IP is quite easy cover without AI but is really problematic. Having same IP is quite common.
Have you checked out Discourse AI and Discourse Chatbot?
I did just hear about this yesterday from @Jagster (thanks, Jakke, for pointing that out). I’ve been looking into this a bit more, the advanced version in particular… and if I’m reading it right, it looks like it’ll be somewhat expensive to implement this, either by having an Enterprise hosting account or a pretty beefy self-hosted server.
Either way, it’s good to know this already exists as an option. It looks like this checks some important boxes, but I can think of many more ways it can be utilized. I’m excited to see how this continues to develop in the months and years ahead. There’s a ton of potential for this kind of thing!
There are a lot of potential positive things for this, also a lot of risks and drawbacks.
Stack Exchange has an A.I. bot that reviews answers and will mention this to the author if their answer seems unclear:
“As it’s currently written your answer is unclear. Please edit to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center. -Community bot”
This kind of a prompt can be really helpful to inspire clearer explanations and avoid people becoming confused, frustrated, or annoyed with unclear answers.
It’s not resource intensive on your self-hosted instance because you can just use hosted models, e.g., openai. So you just pay for API calls for embeddings and chat.
I built a custom integration for a client using Discourse.
Collected past moderation activity and trained an NLP to flag topics and comments that needed attention.
Added a toxicity moderator also trained from their past moderation activity.
Added a sentiment integration to help quickly resolve comments.
Training is done on Google Colab and model loaded on GCP to server APIs from the discourse webhooks.
저는 포럼을 사용하는 것을 좋아합니다. 왜냐하면 우리는 모두 공통의 관심사나 목표를 가진 실제 사람들이기 때문입니다. 누군가가 질문에 대해 잘못된 답변을 하면, 반드시 다른 사용자가 나타나서 수정해 줍니다. AI가 잘못된 답변을 한 경우에도 같은 일이 일어날 수 있겠지만, 그 느낌은 전혀 다릅니다. 또한, 다른 사람들이 문제를 어떻게 접근하는지 읽는 것은 우리 자신의 사고 과정에도 도움이 됩니다. 저는 누군가의 잘 논리화된 답변을 읽으며 새로운 사고 방식을 발견하거나, 이미 잘 알고 있다고 생각했던 일을 새로운 방식으로 수행하는 법을 배우곤 했습니다.
또 다른 고려 사항은 오탐(false positives)의 가능성입니다. 이는 (그리고 실제로!) 사람들을 떠나게 만듭니다. 만약 제가 신규 사용자로 포럼을 방문했는데, 기계가 실수로 제 게시물을 플래그/표시하거나 계정을 정지시키거나 기타 조치를 취했다면, 그리고 그것이 명백히 일어나지 말았어야 할 일이었다면, 저는 아마도 다시 돌아오지 않을 것입니다. 사이트에서 벗어나 모든 것을 잊어버리거나, 수정을 요청하는 번거로움을 감수할 만큼 짜증이 날 것이기 때문입니다.
모더레이션에서 인간적인 요소를 제거하려는 충동은 잘못된 방향으로 가고 있다고 생각합니다. 모더레이션에는 때때로 예측 가능한 규칙이 있을 수 있습니다 - 예를 들어, 감시 단어 기능이나 IP 주소 매칭을 사용하여 그런 문제를 처리할 수 있습니다. 하지만 알고리즘으로 모호한 부분을 처리하는 것은 완벽한 알고리즘을 찾는 끝없는 추격전으로 끝나고, 근본적인 행동을 다룰 수 있는 더 건강한 커뮤니티를 구축하는 데서 주의를 분산시킵니다. 궁극적으로, 제 희망은 사용자들이 그들의 행동을 변화시키는 것입니다. 그리고 그들은 그것을 할 수 있다고 믿어야 합니다.
루틴한 질문들은 사람들이 인간적인 연결을 형성할 수 있는 기회이며, 이러한 접점은 포럼을 응원할 장기적인 사용자를 개발하는 데 필수적입니다. 쉬운 질문에 답하기 위해 나타나는 친근한 얼굴은 AI로는 결코 만들 수 없는 방식으로 환영하는 분위기를 조성합니다. 이는 커뮤니티 구축 측면에서 쉽게 얻을 수 있는 성과입니다!