About 1/3 of our traffic is from crawlers (about 250K last month). Is there a way to block these but allow Google’s crawlers?
AI 生成的摘要
A user, Roy_ILGM, is seeking a way to block crawlers while allowing Google’s crawlers on their site, which receives a significant amount of crawler traffic. codinghorror suggests using site settings, where you can find options to block crawler user agents, or specify allowed crawler user agents to only permit specific crawlers, such as Google’s.
现在可以通过站点设置实现此功能，搜索“crawler”，具体为 blocked crawler user agents。如果您希望非常严格，仅允许特定的爬虫，还可以使用 allowed crawler user agents 来禁止其他所有爬虫。
12 个赞