Good thoughts.
There is no one size fits all to stop rouge and malicious bots; and each site must evaluate which controls work best for them.
On a similar note…
Sites which rely mainly on blacklists and spam or rogue bot databases can also have problems because let’s say someone does not like site www.our-arch-rival.com because that site is a competitor (or simply made us angry or offended us). Some people will then submit the site www.our-arch-rival.com to a blacklist or DB and then other sites will filter a legitimate site because of this kind of “bad consequence” from the method of blacklisting DBs.
Then, of course advocates of blacklists will stay, “you can go to the blacklisting sites and submit a report and asked to be removed”, but for many busy, long standing sites, that can be a time killer.
Generally, it is important to analyze the problem and create a mitigation strategy based on the scenario because every “adversary” is different. It’s the old “Know Your Enemy” from Sun Tzu and the Art of War. Every situation is a bit different in the real-world and unfortunately, it does take analysis skills for sys admins to create optimal mitigation strategies against malicious or unwanted cyber activity.
This is also a good reason to run Discourse behind a reverse proxy because the reverse proxy is a good place for analyzing, classifying and controlling malicious activity before this traffic hits the Discourse app.
It can be a full time job in the year 2020 trying to control and mitigate rouge bots and other malicious activity in cyberspace. As soon as admins come up with one good detection and mitigation strategy, the spammers and scrappers will adjust and find ways around it. I tend to advise people to oversize their servers to insure they have enough headroom since these kinds of problems in cyberspace are only going to get worse over time, not better.
Ready Player One!