Priority/Severity:
Medium
Platform:
Discourse version b66fca70d0e3d12ef930398289fac5269cd240c7.
Description:
Discourse provides an automated spam detection system. The actions performed by the system when a post is detected as spam include:
- silence account of the post author
- flag the post
The flag brings the automated actions to the attention of the human staff for review. Since the automated system is prone to false positives, human review is essential.
If the author of the flagged post deletes it after the time of the flagging (which they may feel obligated to do after receiving the notification that their post was flagged), the review item is automatically marked resolved. The conclusion used in the resolution is “ignored”.
The user’s account remains in a silenced state. The human staff will not be aware of the action by the automated system and so will not perform the essential review of the silencing action.
Reproducible steps:
- Log into an account with admin privileges on a Discourse forum.
- Using the admin account, adjust the custom instructions on
/admin/plugins/discourse-ai/ai-spamto ensure a spam detection can be produced on a test post (e.g., by specifying in the instructions that posts containing a unique arbitrary keyword should always be considered spam). - Create a throwaway account on the forum.
ⓘ It is necessary to use a new account for this purpose because the Discourse AI Spam feature only scans the first three posts created by each user. - Use the throwaway account to create a post that will be detected as spam due to the custom instructions configured by the previous step.
- Wait for the throwaway account to receive a notification that the post was hidden.
- Use the throwaway account to delete the post.
- Using the admin account, navigate to the review queue, with the “Status” filter set to “Pending” (the default setting):
/review
There is no pending review item for the spam detector system’s action. - Using the admin account, navigate to the profile page of the throwaway user account.
The user’s account is permanently silenced, without any guarantee that the human staff reviewed this highly impactful action. A user whose post was subject to a false positive spam detection will be unfairly excluded from participation in the forum community.
Additional context:
I have reproduced the fault on the forum I manage forum.arduino.cc. Due to the requirement for administrative permissions, I have not been able to attempt to reproduce the fault on try.discourse.org.