Genuine question: why should I let AI crawlers flood my server? Of course, the article makes it clear that the choice will always be mine, but, from a commercial perspective and using Reddit as an example of how they handle AI scraping, what would the benefits be here?
Recently, I saw that Google is going to create personalized pages based on users’ history, meaning fewer clicks for webmasters and more money for Alphabet. So, again, what’s the point here?
Currently, I allow search engines and cache indexers, such as the Wayback Machine, to read and cache my content, although I can’t see anything other than providing my users’ content for Alphabet and others to monetize without my community benefiting in any way, not to mention legal issues like LGPD in my country or GDPR in Europe.