A side observation the poses a question could using Cloudflare also help with google indexing?
I noticed some improvements yesterday, positive indexing and impression trend but it could just be coincidental. 
Noticed a big surge toward end of October. These ones must be getting past CF Block Ai Bots mitigation feature.
CF now suggesting to use Ai Labyrinth:
AI Labyrinth modifies your web pages by adding nofollow links that contain AI-generated content to disrupt bots ignoring crawling standards. The nofollow links added do not alter the contents of your web pages and are only visible to bots.
Fight AI with AI? 
AI Arms race much?
1 Like
It is an arms race and, as with spammers, I suspect it always will be.
What is the option sans Cloudflare (fallout bunker) as an in the middle-man to mitigate this, or is it the only option?
It seems itās all or nothing, and nothing = AI traffic that behaves like DDOS! 
1 Like
This surge is a real PITA, the most effective measure has been total geo block on traffic from every other geo-zone except the geo-zone where the majority of traffic comes from presently.
There are so many ASNās potentially involved in this surge picking them out is a slow painful process.
This makes things overall very non-viable in the mid to long run.
1 Like
For me, what matters most is the experience of real human users - whether members or lurkers. So long as my forum responsiveness is OK, and I donāt get charged for extra traffic, or have my storage fill up with logs, this extra unwanted traffic isnāt really a concern.
It is of course possible that my costs will rise if I need to add RAM or storage or CPUs, or if I need to pay for excessive traffic. It hasnāt happened yet, for me.
2 Likes
Yes users experience is paramount, the whole point of this topic. We only exist because āusersā but sometimes I wonder do dev and the like forget about this. 
These AI things load the server so much that you start to get 50x errors. Especially if you try a search. The platform can take a lot of traffic and hold up but itās get a bit janky.
I wonder what spec you are dealing with and is it behind CF or other WAF?
Iāve seen this Ai surge hit 4vCPU 8Gb 160+Gb HD droplets, that easily manages 10ās of unique users hit in a real unique visitor surge, start to really struggle under the Ai sucking motion.
Running discourse on an entry level droplet, letāsā say 2GB droplet very quickly the platform will start to buckle and collapse, if not behind something like CF.
This factor did not happen before Ai, unless you had a hot topic. I have seen that happen but only after the fact, because there was no noticeable difference in the performance, discourse handled the serving no problem.
Looking at CF avg data served per day is around 2Gb p/day over a month, when this swarm hit it jumped to 14GB with sign of stopping, until some WAF rules put manners on the assault.
Having to intensify entrance friction means users who really want to get in get in, you have to value those, but more or less the extra organic traffic and so on, well that is going to not bother.
As I said mid to long term viability here poses a real serious problem.
It now requires resources of time and also money one or both. I have no idea how to get around this any other way, than current posture and then there is the nuclear option, pull the plug. I have seen sites shut down for less.
The open internet was not built with this kind of game in mind.
For sure, if you find youāre serving 50x errors, thatās something dramatic.
I used to run behind Cloudflare and I would do so again if I had a problem: my thinking is that they have a central view of all malicious traffic and they can respond rapidly and centrally. I believe they have specific offerings to stop bad behaved AI crawlers. I wouldnāt expect local methods to work very well in this arms race.
Just as I pay a small fee to mailgun to handle mail, which in some ideal world I wouldnāt need to, Iād accept I might need to pay a small fee to someone to block this unwanted traffic.