About 1/3 of our traffic is from crawlers (about 250K last month). Is there a way to block these but allow Google’s crawlers?
现在可以通过站点设置实现此功能，搜索“crawler”，具体为 blocked crawler user agents。如果您希望非常严格，仅允许特定的爬虫，还可以使用 allowed crawler user agents 来禁止其他所有爬虫。
12 个赞