"Needs Approval" review items from “Triage posts using AI” script of “Automation” plugin not visible to moderators

Bug
, ,
1

Priority/Severity:

Medium

Platform:

Discourse b66fca70d0e3d12ef930398289fac5269cd240c7

Description:

The official “Automation” plugin has a “Triage posts using AI” script. This script has the option to flag posts.

The type of flag generated by the automation (as well as accompanying actions) can be configured via the script’s “Flag type” setting:

  • “Add post to review queue”: causes flags to have the "Needs Approval reason
  • “Flag as spam”: causes flags to have the “Spam” reason

Since the separate Discourse AI spam detection feature already provides high quality spam detection, the “Triage posts using AI” script will often be used for tasks unrelated to spam. In this case, the “Add post to review queue” options will be most appropriate.

:bug: When an automation’s “Flag type” setting is set to one of the “Add post to review queue” options, the review items generated by the flags it raises are not visible to moderators. The burden of reviewing these flags will be solely on the forum administrators, even though this is a moderation task.

Reproducible steps:

  1. Log into a forum with a admin account.
  2. Open the Discourse AI > Personas admin page (/admin/plugins/discourse-ai/ai-personas)
  3. Click the “+ New persona” button.
    The persona configuration page will open.
  4. Add arbitrary content to the “Name” and “Description” fields in the persona configuration page.
  5. Define a prompt in the “System prompt” field that will be suitable to trigger the automation on a test post. For example:
    You will be given a piece of text, and your task is to determine whether it contains the string `foobarbazquxpippoplutopaperino`.

If the post contains that string respond `true`. If the post does not contain that string, respond `false`.

Your response must consist of the text `true`, or the text `false`, and nothing else.
  6. Select a model from the “Default language model” menu (e.g., “CDCK Hosted Small LLM”).
  7. Click the “Submit” button at the bottom of the page.
  8. If it isn’t already, enable the built-in “Automation” plugin.
  9. Navigate to the plugin’s “Automations” page (/admin/plugins/automation/automation).
  10. Click the “+ Add automation” button.
    The “Select a script” page will open.
  11. Add an arbitrary name to the “Name” fields in the automation configuration page.
  12. Select the “Triage posts using AI” script.
    The automation configuration page will open.
  13. Select “Post created/edited” from the When/What… > Trigger menu.
  14. Select the persona you created via the previous steps from the Script options > Persona menu.
  15. Type true in the Script options > Search for text field.
  16. Check the Script options > Flag post checkbox.
  17. Select Add post to review queue from the Script options > Flag type menu.
  18. Optionally, make adjustments to the automation’s other settings in order to facilitate the test.
  19. Click the “Update automation” button at the bottom of the automation configuration page.
    You will be returned to the “Automations” page.
  20. Set the “Enable automation” switch for the newly created automation to the “on” position.
  21. Compose a post or DM that will trigger the automation.
    ⓘ With the example system prompt provided above, this would be accomplished by including the text foobarbazquxpippoplutopaperino in the post.
  22. Publish the post or message.
  23. Using the admin account, navigate to the review queue (/review).
    :slightly_smiling_face: The automation generated a “Needs Approval” flag for the test post.
  24. Log into a forum with a non-admin account that has moderator permissions.
  25. Navigate to the review queue (/review).

:bug: The review item generated by the automation’s “Needs Approval” flag is not visible to the moderator account. Even if you navigate directly to the URL of the review item with that account, you will only get a 404 page.

Additional context:

The fault does not occur if I instead select Flag as spam and hide post from the Script options > Flag type menu.

Although this does offer a workaround, it result in a poor experience for the moderation staff in the use case where the script is being used to triage posts for reasons other than spam detection. We expect a flag to have a “Spam” reason only in the case where it is suspected of being spam.

The “Needs Approval” flags generated by the “Approve unless allowed groups” feature are visible to moderators. This leads me to believe that the lack of visibility of the flags from the “Triage posts using AI” script is caused by a bug rather than an (inexplicable) intentional design.

I don’t believe there is any valid reason for hiding these review items from moderators, but if there is one then this behavior should at least be made optional.

I am able to reproduce the fault on the forum.arduino.cc forum.