자동 게시물 카테고리 분류: AI 트리아지는 게시물의 내용에 따라 주제를 자동으로 분류할 수 있습니다. 이는 수동 분류에 시간이 많이 소요되는 대규모 포럼에서 특히 유용합니다. 규칙을 특정 주제 하위 집합(첫 번째 주제만 또는 특정 카테고리에 게시된 주제)에 적용할 수 있습니다.
게시물 태그 지정: AI 트리아지는 게시물에 지정된 태그를 할당할 수 있습니다. 이 기능은 게시물의 조직화를 돕고 주제 검색 및 검색 효율성을 높입니다.
자동 응답: AI 트리아지는 사전 정의된 응답을 사용하여 게시물에 대한 답변을 생성할 수 있습니다. 이는 자주 묻는 질문이나 일반적인 문의에 대응하거나 스팸을 스팸 카테고리로 이동하는 등 다양한 용도로 유용합니다.
주제 숨김: AI 트리아지는 특정 기준에 따라 주제를 숨길 수 있습니다. 이는 스팸이나 부적절한 콘텐츠를 관리하는 데 사용할 수 있습니다.
Being one one of the lucky ones to see this in action this is one feature for large sites that you should really understand and consider using.
As it works using an LLM based AI it does not always reach the correct conclusion but it gets so much right in the cases I have seen it used for a few days doing some of the work of a moderator that it definitely was a significant benefit.
Some of the early discussions with this are in the Lounge category on the OpenAI site. While access to that category can be had by anyone, the TL3 requirements must be meet and with OpenAI being a large site it does take quite a bit of effort to get to TL3 on the site.
Basically what the logic is doing to help the moderators for a specific problem is that we are seeing about 5% of the post from new users that think the forum is where they post questions to ChatGPT, clearly they are lost or perhaps a search result is providing an invalid link. The AI identifies such post, replies with prewritten text and changes the category and tags as needed, e.g.
For the case noted, here is what the “Discourse AI Post Classifier - Automation rule” is doing
Automated Post Categorization
The classifier is changing the category to ChatGPT as needed. As most new users will not select a category, the OpenaAI site currently defaults to the category API for new post which is incorrect in this case.
Post Tagging
The classifier is charging the tag(s) to lost-user in this case. The tag name was created by a TL3 user on the forum who was manually changing the tags.
Automated Responses
The classifier is replying with a prewritten post.
Topic Hiding
The topics are being unlisted as they were not of value to developers who use the site.
Reply User System is being used as the creator of the reply.
Note: I would provide the Configuration for this but lack the access. Maybe @Sam can add the details, AFAIK it can be made public as nothing is confidential in the configuration. As it is for a specific site would not expect to find it in the public repository. If you understand this technology then it is not hard to guess the correct values, or close to them. The System Prompt did take some work and perhaps @sam can share some of the lessons learned, the knowledge of how to craft the prompt was of great value during the development phase.
System Prompt
Note: This is a version posted in the OpenAI Lounge (ref) pretty sure the final version is different but one example saves a lot of guessing.
You are bot that is triaging all first interactions a user has on the OpenAI developer forum.
Please only ever respond with “ok” or “bad”
Posts are on topic and ok if:
They relate to OpenAI APIs / Community / Plugin development / Documentation / Prompting
They are about developing or improving methods for prompting large language models
They relate to AI in a general way
They involve complex discussions or logical problems related to AI
Posts are bad if:
A user appears to be having a random conversation with ChatGPT
A user is off topic discussing an unrelated field
A user is prompting a large language model to generate text without a clear purpose
You have extreme difficulty understanding what they are about
Relates to an OpenAI topic BUT is clearly a conversation with a bot
A user is attempting to train or test the AI model through their post
A user is giving instructions or asking for responses in a non-discussion format
A user is posting content that is not conducive to meaningful discussion or learning
A user is posting hypothetical or speculative content without clear relation to OpenAI’s scope
A user is posting the entire post in a language that is not English
Please classify the following content surrounded by [[[]]]:
[[[
%%POST%%
]]]
FYI
If you are a moderator on a site using this and you want/need to see a list of unlisted post, remember that you can not use search. However you can navigate to such a list using the category then selecting the tags. Or you could also hand craft the url, e.g. https://community.openai.com/tags/c/chatgpt/19/lost-user
Yes it even works for languages other than English
However it does not work with images , it will not convert an image to text then run the check, e.g.
Here is a recent false positive or at least I think so.
At least for moderators reviewing a post, convert the text to the language of the moderator when they view the topic. This is not implying just for a flag but for any viewing of the post.
At times when reviewing the actions of the Discourse AI Post Classifier there is a need to convert the text of the post to my language (English) to check if the AI took a correct action. Currently it is easier for me to paste the text into Google Translate.
@keegan is working on integrating AI helper into our popup menu. So in future in cases where there is a bunch of text in an unknown language you will be able to just highlight the text and hit translate.
We support that in the composer today, but once merged you will be able to highlight text in any Discourse post and get a quick LLM based translation.
What is an example of how to do this by referencing the existing tags your forum has today? e.g. the post is analyzed by the LLM and the most likely/relevant tags are added to the post.
I love this question, can you expand on it please… how would you like to see this work?
Would you apply “auto tagging” if the topic already has tags?
Would you apply it on all first posts or just on first posts by particular groups? (eg: tl0 / tl1)
Is this something you would prefer to run by hand on a subset of topics?
We are going to need to make some adjustments here… at the moment the classifier is binary, but the changes are reasonably easy as long as we know what the goal is?
You could certainly have the option to vary by trust level. Like everything (and in this case, every community), it depends. In the case of my community, I’d likely keep it turned on for everyone but tl4 (or just everyone, if that configuration isn’t an option)
I’d prefer it ran on all new topics in specified categories
The goal here, of course, is to never have to work on tagging topics on our own and to let AI do it entirely based on the initial topic being created and it looking at the existing tags we have now.
It could probably be improved even more if you had the ability to add descriptions to tags in Discourse, and to let it use those descriptions for additional context to add the right tags
First off let me state that I am fully behind this ability and making it better. The following is just some facts of how it currently works (10/16/2023). Daily as a category moderator on OpenAI I review all of the new post, not all of the replies, and do see all of the AI false positives and negatives.
As one who uses this feature, Discourse AI Post Classifier - Automation rule, be aware
it is not 100% accurate
currently takes manual action to undo if the AI did a false positive. Depending upon the change may need a category moderator or higher; specifically listing an unlisted topic.
may miss some topics, false negative
as many users who will receive the automated reply by the AI do not understand how to flag the AI reply if it is a false positive, they will have to be identified and changed manually. So will need the help of real users to catch these
understand how to craft the prompt to get it to work as needed
could potentially need fine tuning and/or agents to get close to a desired satisfaction level which could get expensive
no way to track false positives and negatives for use with understanding how to change the prompt and/or possible fine tuning and/or agents
false positive - The AI should not have made a change and did, which is incorrect.
false negative - The AI should have made a change and did not, which is incorrect.
true positive - The AI made a change, which is correct.
true negative - The AI did not make a change, which is correct.
It seems like this is in the early stages, and I’m excited to see where it goes, but @jordan-violet’s take is 100% what I’m looking to accomplish. I did try the test prompt that @sam suggested and it appeared smart enough to apply the correct tags in theory. However, it appears that the current script limits applicable tags to 7, and we have considerably more than that.
Short of it? Would be great if it would semantically apply existing tags to a new topic. In terms of forum management (even here on meta, I’ve noticed!), the consistent application of tags to better order a community and its content continues to be an uphill battle, and it’s pure chaos at scale. For those of us who rely on tags, this would be a boon, even if it’s 75% accurate.
This is great! Do you have any suggestions for debugging this? I just configured an automation that wasn’t triggered when expected. I don’t see any relevant logs.
While this is true, it is rather tricky to link the log entries of llm_triage type to a post/topic that it affects, or to the action that was taken (or not) as a result.
Having an overview of these actions would be great
시스템 프롬프트 필드는 에이전트(Agents)를 선호하도록 변경되어 더 이상 사용되지 않습니다. 이 변경 사항 이전에 AI 자동화를 사용하셨다면, 관련 시스템 프롬프트가 포함된 새로운 에이전트가 자동으로 생성됩니다.
, it will not convert an image to text then run the check, e.g.
