# 更智能地处理随机爬虫流量

**URL:** <https://meta.discourse.org/t/smarter-handling-of-random-crawler-traffic/78227>\
**Category:** Feature\
**Created:** [2018年一月16日 23:17 UTC](https://meta.discourse.org/t/smarter-handling-of-random-crawler-traffic/78227 "2018-01-16T23:17:49Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![sam](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/sam/32/102149_2.png) [@sam](https://meta.discourse.org/u/sam)\
**Post date:** [2018年三月29日 03:51 UTC](https://meta.discourse.org/t/smarter-handling-of-random-crawler-traffic/78227/2 "2018-03-29T03:51:12Z")

</div>

This is now complete thanks to @neil!

You can see “top crawlers” at `https://yoursite/admin/reports/page_view_crawler_reqs`

You can blacklist bad crawlers by setting `blacklisted crawler user agents`

Alternatively, if you wish to only allow particular crawlers, you can set the whitelist with the setting `whitelisted crawler user agents`

---

_[View the full topic](https://meta.discourse.org/t/smarter-handling-of-random-crawler-traffic/78227)._