Best practices dealing with Spam users and GPT reply posts

[jord8on (Jordan)]

Context: We started a community several years ago, right as the pandemic hit strong, then had to devote all resources to other initiatives for several years. We had a lot of participation and then when we stopped community programming and management, the forums slowed to a halt. We’re about to invest more time in implementing our community strategy and management, so this question is pressing the most!

---

Right now we are noticing that a lot of the users joining the community right now are either:

1. Joining to post a link back to their spammy affiliate keyword-focused website
2. Joining and posting what very much appears as an LLM / GPT generated reply/post

Here’s an example of a post that hit our moderation queue yesterday:

Note: The user just created their account that same day.

Click to view a screenshot of moderated post

Our Discourse system message was sent to the user:

Click to view a screenshot of canned SYSTEM message

This post from a new user was in response to this 3-year old post from Chris Anderson.

When you read the moderated post from the new user, it’s pretty clear that this message is disingenuous and a response that’s been generated by GPT or something similar.

Are there optimal ways to prevent these types of users from showing up in the community?

Are there ways to prevent users from entering LLM canned responses, for whatever self-serving or malicious reason they may want to be doing this?

Any input would be great!

[simon]

@jord8on

Joining and posting what very much appears as an LLM / GPT generated reply/post

How often is this happening? I wonder if your site is being specifically targeted for some reason.

@jord8on

for whatever self-serving or malicious reason they may want to be doing this

I’m taking a wild guess that it has something to do with one or both of these sites: http://bestpickleballpaddle.com/, https://thepickleballpaddles.com/. (Note, those are real sites, but let’s not give them too much traffic from here.)

The most obvious motivation I can think of for generating a post like the one in your screenshot is to gain some trust on the site in order to post spam links in the future. If that’s the issue, it might be worth changing the value of your site’s min trust to post links setting from the setting’s default value of trust level 0 to either trust level 1 or 2. If there are some domains that you’re happy to allow users of any trust level to post to, you could add those domains to the allowed link domains site setting.

I’m just spitballing here, but I’m wondering if instead of putting a lot of effort into trying to catch content generated by LLMs, it would be better to try to reduce the motivation for creating those types of posts in the first place. My concern is that LLM generated content will become more difficult to detect over time. Falsely identifying content as having been generated by an LLM could become an issue.

Another approach that might be useful for some cases would be to use membership in an organization, or activity on another platform as a criteria for creating a Discourse account, or as a criteria for gaining a trust level that allowed posting links on a Discourse site. For example, your signup form has an optional field asking for the user’s USAPA Rating Level. If membership in the USAPA gives members a profile page on their site, you could make membership in the USAPA a requirement for signing up for your site by having “USAPA Profile Link” as a required field on your signup form. You could then enable the Discourse must approve users site setting and only approve users after you had confirmed their USAPA membership. Another possibility would be that Discourse could add a feature that would allow a user’s trust level to be limited based on a custom criteria. For example, don’t allow a user to progress past trust level 1 until their USAPA membership has been confirmed.

I’m just using USAPA membership as an example here. I can see why you might not want to require USAPA membership for users on your site. The idea of establishing trust via membership or activity on another site is related to the plugin that’s being discussed here: $10k Bounty: Gitcoin Passport plug-in for Discourse.

[merefield (Robert)]

@simon

The most obvious motivation I can think of for generating a post like the one in your screenshot is to gain some trust on the site in order to post spam links in the future

Exactly this.

There almost certainly has to be some agenda, even if it is not yet obvious.

@simon

it might be worth changing the value of your site’s min trust to post links setting from the setting’s default value of trust level 0 to either trust level 1 or 2

I recently had a user that was quite convincing and made it all the way to Trust Level 2 before becoming blatant about their product promotion, having just hinted originally.

The trust level link setting is useful and dodgy accounts will start to suggest people search for xyz via google in lieu of being able to post links: another red flag.

Sometimes you just have to wait until the amount of evidence mounts up to be sufficiently sure to ban an account.

Unfortunately delaying action does mean the clean up operation is more work later on.

[merefield (Robert)]

@jord8on

Are there ways to prevent users from entering LLM canned responses, for whatever self-serving or malicious reason they may want to be doing this?

There is a cut and paste “typed suspiciously fast” check”? Unless the whole thing was a robot designed to type slowly to mimic a human …

[Bas (Bas van Leeuwen)]

So (if I understand things correctly) the good thing here is that this person was caught by the existing infrastructure. Bothering staff, but not users.

I think Simon hit the mark here:

@simon

reduce the motivation for creating those types of posts in the first place

I’d be interested to hear how much this is happening. The fact that this person set a relevant avatar, with a relevant username, on a relevant forum… leads me to assume this is partially a human thing.
Could just be a overzealous marketing intern?

[Dan_Eastwood]

I’ve had a similar problem. The offending party is currently suspended. There are web sites that can help to identify AI generated text, which is helpful but this is not a long-term solution. – The AI will keep getting better until there is no telling the difference.

However, Randall Munroe suggests it may be a self-solving problem.

[EricGT]

@Dan_Eastwood

The AI will keep getting better until there is no telling the difference.

FYI

I am a category moderator on the OpenAI Discourse forum (ref) (not an OpenAI employee) so see the feedback on the forum concerning ChatGPT changes over time (ref) and while in general your statement is correct just adding some clarity regarding ChatGPT.

From the time ChatGPT became available and for many months its abilities were improving, the completions were

• more coherent
• in agreement with the prompt
• become more concise, short in length

Then a change was made that resulted in completions that were

• less coherent
• less in agreement with the prompt
• less concise and longer in length

Many users on the forum noted that it was if the ChatGPT had been rolled back to an earlier version a few months ago. Those using ChatGPT to create code, a use case I also use ChatGPT, noticed it more pronounced as more syntax errors were appearing with the change.

So the thought to keep in mind is that if you are trying to identify ChatGPT generated content, it is a moving target and that it is not always getting better. As a programmer, if I were trying to identify ChatGPT generated content, this would be of value.

Also I do not have any inside knowledge by being a moderator on the site as I am not an OpenAI employee so if you hear of tools that embed info to make such ChatGPT content identifiable, I can not provide info for that.

---

---

For those that do not use ChatGPT here is simple example using the above text.

Example (Click triangle to expand)

ChatGPT July 20 Version
Prompt

Proofread, use nice tone, be concise.

```
FYI

I am a category moderator on the OpenAI Discourse forum ([ref](https://community.openai.com/about)) so see the feedback on the forum concerning ChatGPT changes over time ([ref](https://community.openai.com/t/gpt-4-has-been-severely-downgraded-topic-curation/304946)) and while in general your statement is correct just adding some clarity.

From the time ChatGPT became available and for many months its abilities were improving, the completions were 
* more coherent
* in agreement with the prompt 
* become more concise, short in length

Then a change was made that resulted in completions that were
* less coherent
* less in agreement with the prompt
* less concise and longer in length

Many users on the forum noted that it was if ChatGPT had been rolled back to an earlier version a few months ago. Those using ChatGPT to create code, a use case I also use ChatGPT, noticed it more pronounced as more syntax errors were appearing with the change.

So the thought to keep in mind is that if you are trying to identify ChatGPT generated content, it is a moving target and that it is not always getting better. As a programmer, if I were trying to identify ChatGPT generated content, this would be of value. 

Also I do not have any inside knowledge by being a moderator on the site as I am not an OpenAI employee so if you hear of tools that embed info to make such ChatGPT content identifiable, I can not provide info for that.
```

Completion

FYI,

I want to clarify some important points regarding the changes in ChatGPT over time. As a category moderator on the OpenAI Discourse forum, I’ve been closely observing the feedback from the community regarding ChatGPT’s development (ref).

Initially, after its release, ChatGPT showed consistent improvement in its abilities over several months. The completions were becoming more coherent, aligning better with the given prompts, and getting more concise and succinct.

However, there was a subsequent update that resulted in some setbacks. Users noticed that the completions became less coherent, less aligned with the prompts, and longer in length. This led to comparisons with a previous version of ChatGPT, which appeared to be a step backward. In particular, those who use ChatGPT for code generation experienced a more pronounced impact, as they noticed an increase in syntax errors with the change.

It’s essential to understand that identifying ChatGPT-generated content has become a moving target. While it was steadily improving before, it’s not always the case now. As a programmer myself, I find this information valuable, especially when trying to distinguish between human and AI-generated content.

I must clarify that, despite being a moderator on the site, I am not an OpenAI employee and do not possess any insider knowledge. Therefore, if there are any tools or methods developed to identify ChatGPT content, I am unable to provide information on that matter.

Thank you for your attention, and I hope this clarification helps in understanding the current state of ChatGPT’s development.

Best regards,

[Your Name]

[EricGT]

FYI

Just saw this OpenAI Quietly Shuts Down Its AI Detection Tool and confirmed it here

As of July 20, 2023, the AI classifier is no longer available due to its low rate of accuracy. We are working to incorporate feedback and are currently researching more effective provenance techniques for text, and have made a commitment to develop and deploy mechanisms that enable users to understand if audio or visual content is AI-generated.

[Dan_Eastwood]

@EricGT Excellent points! Yes, it is definitely a moving target. I’m not concerned about ChatGPT in particular. The cat is out of the bag, so to speak, and I think it’s only a matter before someone unscrupulous uses the tech in a deliberately harmful way.

[markersocial]

@simon

The most obvious motivation I can think of for generating a post like the one in your screenshot is to gain some trust on the site in order to post spam links in the future.

Yeah, this (including astroturf shilling without urls) as well as for the urls and advertising they can add to their profiles - either for direct response or SEO. In other cases, to farm accounts for DM spamming. These are the main cases afaik.