Im curious to hear from community members whether they are experiencing any or an uptake in AI powered spam
This would be specifically seeing answers to questions that look like they are ChatGPT based and seem either non-human like or have hallucinations (a common problem with LLMs)
I am experiencing AI based spam
Yes
No
0voters
If the answer is yes Im curious to hear…
How often this is happening?
How much of a problem is this creating within your community?
What are you currently doing about it?
If the answer is no Im curious to know…
How are you preventing this from happening?
Are there reasons as to why your community inherently doesn’t face this issue?
We just use AI as a tool to seek knowledge, maybe a little causal chat.
Perhaps our community is small, and has a common sense that hallucinations are BAD
I reckon the most effective way to stop anykind spamming is being member of very small and difficult language. It stops those clowns who are doing manual labour.
Well, we all know spammers aren’t that smart and automatic traffic doesn’t care of language, genre or even size. So, there must be another reason why some forums or sites are like honey pots for anykind trash and others live without drama.
For the reason why spammers can sign in there and can’t somewhere else when system and setup should be identical I don’t have answer. But one thing is sure: admin’s or other background force’s need to increase fast growth from global audience will lead to bot ans spam problems.
In the last two weeks or so, we have seen a spike on our site. We’re seeing typical spam with hidden links on new replies from new accounts. When we increased the reputation for creating new posts, we saw AI-generated responses increase, and it seemed the bots were trying to slowly increase their reputation on bogus accounts. These responses don’t have obvious bogus links, they just have generic AI text that doesn’t contribute to answering the question.
We got hit over a weekend with a large spike in spam posts, enough that someone created a new topic saying there was too much spam on our forum. Since then, admins need to check the site every day to clean up bogus AI posts. We’re also seeing AI posts on accounts that were created in the past and had no activity, which makes it seem like some spam bots had been seeding accounts for a while and letting them sit with no activity. Now they are trying to slowly get past the engagement limits so they can post new topics.
As noted above, we increased the trust levels for posting new topics. We also enabled akismet. But this hasn’t stopped the AI spam posts. Currently we need an admin/moderator to check the forum every day to review flagged posts and clean up. Some are challenging and look like they might be a person, so two people need to check.
We encouraged our users to help out and flag posts that look like AI and that has helped.
Our forum is fairly low volume and has run for years with very low admin clean-up and maintenance, but it seems the AI bots have found us. I’m thinking AI may be needed to stop AI?
It haven’t seen a lot of it yet, but my forum holds the first few posts in moderation, and I can usually tell if someone might be a spammer by certain clues. I lock the suspicious ones at TL0 until they post something that is clearly on topic.
It isn’t a “chat about random things” forum, so it’s usually possible to tell whether someone is faking interest by the first post.
Actually, I just stumbled on a user who slipped by and is posting with ChatGPT or other AI. There might be more spam accounts that I’ve missed.
Some ideas on how to fight it:
Make a database of VPN providers. This one’s IP address is from “M247 Europe SRL” which is a VPN service provider. I’ve always wanted some kind of notification that a new account is using a VPN. I have to do it manually at the moment.
Keep track of read time, days visited, topics/posts read. This user spent 8 minutes reading the site but posted 6 comments, and only visited 3 times on the day of their registration. The user is actually still TL0 naturally, because they haven’t really done anything except post comments.
I wonder if it’s possible to roughly classify users by the ratio of time spent on the site vs. number of words written, plus other signals like VPN, pasted content, injected content, etc. Suspect accounts could be marked for review.
Edit: this quick Data Explorer query turned up a few more, though some of them were already suspended.
SELECT
u.id,
u.created_at,
u.username,
u.trust_level,
us.time_read,
us.days_visited,
us.topics_entered,
us.post_count,
us.topic_count
FROM users u
LEFT JOIN user_stats us
ON us.user_id = u.id
WHERE u.trust_level < 1
AND u.created_at > '2023-01-01'
AND us.time_read < 1000 -- seconds
AND us.post_count > 1
One per day. Pattern on forums with 2000 or so users (500 per year new):
new user signs up with email domain from category “temporary domain” such as “cetnob.com”.
within a number of hours either creates a new topic or replies to an existing topic
creates text in English sections only (bilingual site, with over 90% NOT in English)
sometimes includes a URL in the text, sometimes not
text seems like an actual question or remark, using words uniquely used on the forums and relevant material
but text feels somewhat off-topic, but very good still: an inexperienced support engineer could not detect it
It highly resembles in pattern the pattern described in:
We are blocking already hotmail, gmail and other large consumer oriented domains using an explicit list, but there are at least 10.000 domains known to us that are used for this type of approach. In our own software we have an explicit list plus a real-time check on UserCheck (we use the free variant and only check on sign up on our own apps and cache, so 5000 lookups per month is sufficient).
From what I have seen, this specific behaviour can be tackled by automatically blocking temporary / spam email domains.
Blocking TL0 from using links is not really considered more viable than moderating all new requests, since many users post directly after first sign up, the site being a support portal.
I’m getting a lot of AI spammers lately, and it’s time-consuming to go through them.
With the current spammer I’m looking at, the text is written in perfect English, it’s a VPN, the email address is in StopForumSpam, and I can tell the content was copied/pasted because the dash character that was used doesn’t exist on keyboards. I had to check all of that manually though and still have several more to look at this morning.
Brainstorming another idea:
When a post is saved, Discourse could record extra data in a JSONB field on that post:
IP address
is_vpn? — a lookup in maxmind to find the org and see if it’s a VPN (e.g., PacketHub S.A.)
a quick lookup for the email address in StopForumSpam
A comparison of number of characters output into the editor vs. number of output-producing characters typed (excluding arrow keys, ctrl, etc.). For example, the user output 1,000 characters in the raw content, but only pressed output-producing keys 10 times (suggesting that the content was pasted and the user then might have edited a word).
Number of times content was copied or cut using keyboard shortcuts or right-click.
Number of times content was pasted using keyboard shortcuts or right-click. The difference in the copy/paste numbers would provide another clue.
Moderators could view that data on posts in a small table. Unusual values could be highlighted so suspicious posts would stand out.
There probably isn’t a perfect method to automate the detection, but having more information would speed up the moderation process.
I haven’t used AI in my forums because it’s expensive. I’m also not convinced AI would be able to solve this problem, because the content looks normal. I’d still have to manually investigate every suspicious post.
I haven’t had a problem with NSFW content.
My problem isn’t that there is anything wrong with the content. The only suspicious things about it are that new users don’t write posts like that within a few minutes of registering, and the content is also somewhat vague. My main forum has a very specific topic, and if a first post doesn’t say something specific about the person’s relationship with that topic, I start the investigation process. Otherwise I might not notice their posts.
Here are a couple of examples. The content is just vague enough for me to start the investigation process, but it’s time consuming, because I have to do it manually.
I can’t ban this user based on content alone. It’s the other clues that tell me it’s a spammer.
This IP address was a VPN in Norway, and the content was too vague. I was able to confirm it because the email address was in StopForumSpam with an IP address of Germany:
I’m just brainstorming out loud here, but it would be faster to moderate these users if there were a small table on posts that said something like:
location
Oslo, Norway [from maxmind]
organization
PacketHub S.A. [from maxmind]
is_vpn
true
email
whatever@example [sometimes this provides clues]
stopforumspam
true [link]
characters_output
1,234
characters_output_pressed
10 [this doesn’t match the number of chars in the post, so it’s a clue]
num_cut_or_copy
0 [didn’t copy text from editor]
num_paste
1 [did make one paste]
seconds_editor_open
20 [suspicious for a post of that length]
Maybe the table could be collapsed unless there is a suspicious value and/or moderators could flag a specific user as “probably not spam” which would then collapse the table on all their posts or stop future lookups for that user. Or the user could automatically be marked as safe when they reach TL2.
It’s a combination of things:
VPN or IP address tends to be in a small number of countries where there are a lot of SEO companies (India, Pakistan, Ukraine, Vietnam, Bangladesh).
Sometimes the email addresses are in StopForumSpam.
A lot of the content is pasted in to the editor, but probably not all of it.
The email addresses often don’t match the username. E.g., the username will be “Bob Smith” and the email address will be something different like stevenjohnee1234@example.
The raw content sometimes uses formatted punctuation characters like smart quotes or mdash, suggesting that the content wasn’t written in the Discourse editor.
Take a look at post 1622105 here in this forum. It was posted 3 minutes after registering, hand-edited to change the link from Quora to stackexchange, and the English is good, but it’s talking about technology that isn’t relevant to Discourse. I don’t want to link to it because it would notify the poster.
That’s the kind of post where it would be useful to see the data I mentioned above right in the post.
A lot of this feels like staff experience vs AI. Will call the right people into the topic.
I do think it sound compelling to try to bulk up some of the “fast typing” detection we already have. Having SFS integrated into core may also be compelling.
There is of course the deep philosophical question:
Is it spam if it adds value to the forum (even if it is AI generated)
Should it be removed from the forum if it adds no value to the forum (even it if is human generated)
I leave a few spammer posts online when they provoke discussion, but most of them get deleted. The quality is very low, and it’s often easy to tell when something is written by AI. If I feel like something I’m reading is AI, I start losing trust in the source. I’m not an AI Luddite, but I don’t want to read AI-generated content unless I know it’s generated by AI.
If I see that someone is using AI in the forum, I immediately put a stop to it because trustworthy content is one of the forum’s most important assets.
Also, what looks passable to humans in 2024 might be easy for people to detect as AI in 2034, kind of like how movie effects that once looked realistic decades ago are now immediately detectable as fake. I think AI-generated content from 2024 is going to look dated eventually.
