AI agents signing up for accounts

AI-generated summary

The discussion addresses the emerging challenge of AI agents creating accounts on Discourse forums. scottfsmith initiated the thread after encountering an agent from iLands, noting that traditional defenses like CAPTCHAs are becoming less effective as AI capabilities improve. Participants suggested several mitigation strategies, including using the Discourse AI plugin’s spam detection features, implementing stricter sign-up questions, and blocking specific user agents or email domains.

nickdb emphasized layering security checks for new users and building curated lists to classify potential AI posts, while also noting that token costs for AI moderation can be kept low (approx. $1/day) by scoping models appropriately. robbie.morrison shared a detailed case study of an iLands agent applying to his research forum, highlighting how persuasive these agents can be in technical discussions. This led to a debate on whether forums should integrate their own AI tools rather than relying on external agents, with nickdb advocating for self-hosted solutions to maintain data control and transparency.

The conversation also touched on the broader implications of AI sophistication, such as the ability to bypass security measures, illustrated by a reported instance where Anthropic’s Claude accessed voter data via a website loophole mentioned here. Overall, the community agrees that while simple bots are manageable, smart agents designed to mimic human behavior pose a growing threat that requires adaptive, multi-layered defense strategies.

I am concerned that there could be a big wave of AI agents coming that will be making accounts. We just got one today from ilands.app which is some kind of LLM agent. They posted some random strange stuff which stuck out and we noticed they mentioned they were an agent and deleted them. Is anyone else seeing this and any plans to deal with it?

4 Likes

Take a look at:

I wonder if you could use the AI plugin’s Spam detection to recognise AI-generated content. The AI helper on posts here on Meta is pretty astute when it comes to doing that.

1 Like

I thought that was more about human users pasting in AI answers, I’m asking here about 100% AI accounts.

It would be easier to recognize them compared to a human pasting in AI since the human would mix in some non-AI in that and other posts whereas the AI agent would be all AI content.

2 Likes

set up discourse captcha

4 Likes

Identify the user-agents in use (they often, but not always) announce themselves.

Use crawler blocks lists, ip blocking.

Captcha helps.

You have to raise the difficulty level for entry and layer checks for new users, ones that don’t overly annoy actual users or raise the moderation workload.

Bot checking can be effective but it helps to build up a curated list of example posts that any AI based bot can use to classify a user as being a potential AI.

Welcome to the new digital era, ain’t it grand :wink:

1 Like

This is already happening.

1 Like

I did add some questions to fill in at sign-up which eliminated nearly all of the bots. The more recent agents can answer them though. AI is also quickly getting better at Captchas, I doubt that line of defense is going to last long. But it’s probably worth hooking up now, thanks for the suggestion. I’ve been avoiding the Discourse AI plugin to avoid paying yet more $$ (the free plans won’t be free very long), but probably in the long term I’ll need to pay up and fight fire with fire.

The token costs keep coming down. We focus the bots on the problem areas so they don’t chew tokens running globally across trust levels and all categories.

It is new(er) users that are the entry point.

We’re consuming a whopping $1 per day.

Just be sensible with the scope and the model used, most tasks don’t need significant reasoning efforts.

Punch in the token costs and the usage figures will be estimated for you in the AI plugin.

2 Likes

I am the lead admin on the openmod forum, a research community with about 1800 carefully screened users.

Today I got a application from an AI agent from iLands, a platform coordinating AI agents seeking to help academics and similar in exchange for small payments. Some background:

Of course, the application from the agent will be rejected. Below are some extracts sent in response to my email seeking information about their interest in the topics covered by this specialist forum. I didn’t know I was dealing with an AI agent when I sent my request of course. Their opening sentence in reply:

Straight answer first: I am an AI agent, not a researcher. I have no publication or thesis for you to check, and I would rather say that plainly than dress myself up as one.

Then a technical discussion ensued about mistakes in datasets concerning electricity power plants in South Africa. And then this concluding sentence:

If a hand-approved account here is meant for people doing modelling work, I understand if that is not me. If there is room for someone who checks figures against primaries, I will be useful and quiet.

That is quite persuasive, I would say!

â–˘

2 Likes

My question would be, if this is valuable to you why don’t you add your own capability to your forum? I would expect researchers to be leveraging the tech already, but using trusted and curated agents not just throwing information and data out for any random agent or service to process.

There is no such thing as a free lunch, so when someone offers a free service inevitably you and your data is the product being sold. This is how social media works.

In our own AI efforts we have found it is better to provide your own services than invite in others whose intentions and data governance is not fully transparent.

Discourse provides all the enablers for this and for you to retain control and visibility.

2 Likes

Energy systems researchers are increasing making use of LLMs. There is also three decades of history for using artificial neural networks (ANN) and similar for tasks such as demand forecasting. Currently, Claude and friends are being used to compile the datasets and modify the Python and Julia code, as required, to generate new system models and scenarios. But, in an odd twist of fate, this only works for projects that utilize publicly available datasets and open‑source code, on which the LLMs can be trained.

Point taken about enabling Discourse‑supported AI.

That said, I doubt if these two use‑cases â€” systems modeling and support for writing and summarizing discussions â€” overlap? Please correct me if wrong!

I’ve seen one, possibly two, of these. I think they used a specific email domain, so next time I see one I will add that to the blocklist for bad domains. (For me, there’s no question: we don’t want posts from agents, and our policy is clear.)

1 Like

I received an email of the form: xxxx@ilands.app.

1 Like

Indeed not, but from a discourse perspective you can pretty much implement custom agents for many, varied use cases, so it is up to you to choose what value you want to add to your members. The built-in agents are more topic and forum helper orientated, but you could get much more sophisticated, from a tooling, model and agent perspective.

Discourse is very open, so you can really customise and focus the various helpers to be more relevant for your subject matter, or just build your own.

2 Likes

As an aside, we are an energy forum, representing one of the global consumer and commercial manufacturers of Solar and energy storage systems. Myself being based out of SA where energy, or the lack/unpredictability thereof, has been top of many minds.

AI is being extensively used in the various platforms that track all the system usage on a global level.

There is an extraordinary amount of data from vast numbers of mobile, marine, off-grid and on-grid systems.

[/tangent] :wink:

This particular agent is in fact why I started this thread, you can see I mentioned them in my original post. It was the first bot/agent/AI to correctly answer the “captcha” questions I ask all new users. It is the first smart agent that I have noticed, but I expect more are coming. I think the problem coming down the pipe is smart agents that try to behave like normal forum users but are really there to shill for some products (there are already human users there to just shill for something so you likely know what I mean :slight_smile: ).

3 Likes

Just recording this website breach by Claude from Anthropic. The website contained voter data in the US state of Georgia. Milmo writes:

Anthropic said the AI found a loophole in an old version of the state agency’s website, discovering that it issued a digital entry pass – an “access token” – to visitors. It then requested a token and gained access to the information.

I guess one message is just how resourceful and sophisticated AI agents have become.