# Too many Crawlers, is that a problem?
**URL:** https://meta.discourse.org/t/too-many-crawlers-is-that-a-problem/148550
**Category:** Data & reporting
**Created:** [April 19, 2020, 8:18pm UTC](https://meta.discourse.org/t/too-many-crawlers-is-that-a-problem/148550 "2020-04-19T20:18:37Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@salmanfarisvp](https://meta.discourse.org/u/salmanfarisvp)
#### Post date: [April 19, 2020, 8:18pm UTC](https://meta.discourse.org/t/too-many-crawlers-is-that-a-problem/148550/1 "2020-04-19T20:18:37Z")
Hi, In our forum we got too much Too many Crawlers, it’ like 81k to 90k as average! So Is that will cause any problems? if yes how can I disable Bad Crawlers from crawling?
I listed blacklisted crawler user agents as
- mauibot
- semruhbot
- ahrefsbot
- blexbot
- SEO spider
also slow down crawler user agents as
- bingbot
Thanks.
---
### Author:  [@Johani](https://meta.discourse.org/u/Johani)
#### Post date: [April 19, 2020, 8:28pm UTC](https://meta.discourse.org/t/too-many-crawlers-is-that-a-problem/148550/2 "2020-04-19T20:28:40Z")
You can see detailed numbers on a per-crawler basis if you visit
`your.site.com/admin/reports/web_crawlers`
If you see an unusual crawler that’s misbehaving, you can add a unique string from its user agent to the `blacklisted_crawler_user_agents` setting and it will be blocked.
---
### Author:  [@salmanfarisvp](https://meta.discourse.org/u/salmanfarisvp)
#### Post date: [April 20, 2020, 7:06am UTC](https://meta.discourse.org/t/too-many-crawlers-is-that-a-problem/148550/3 "2020-04-20T07:06:12Z")
Thanks for the quick replay. I found some crawler has 1695285 views, is that normal or not?
also on more question how can add the **solution** button in my forum, Do I need to install any plugins?
Thanks.
---
### Author:  [@Johani](https://meta.discourse.org/u/Johani)
#### Post date: [April 21, 2020, 1:22am UTC](https://meta.discourse.org/t/too-many-crawlers-is-that-a-problem/148550/4 "2020-04-21T01:22:07Z")
> [@salmanfarisvp](#):
>
> I found some crawler has 1695285 views, is that normal or not?
Well, that depends on how large your site is and the user agent for that crawler. You don’t want to block Googlebot, for example.
> [@salmanfarisvp](#):
>
> how can add the **solution** button in my forum
Have a look here
> [@Discourse Solved](https://meta.discourse.org/t/discourse-solved-accepted-answer-plugin/30155):
>
> discourse2Summary Discourse Solved allows users to accept solutions on topics in designated categories.open_bookInstall Guide This plugin is bundled with Discourse core. There is no need to install the plugin separately. Enabling Solved The Solved plugin can be enabled from its settings, accessed from your admin/plugins page: After enabling the plugin, to switch on Solved for all categories use the allow solved on all topics site setting. Alternati…
---
### Author:  [@salmanfarisvp](https://meta.discourse.org/u/salmanfarisvp)
#### Post date: [June 24, 2020, 8:58am UTC](https://meta.discourse.org/t/too-many-crawlers-is-that-a-problem/148550/5 "2020-06-24T08:58:54Z")
> [@Johani](#):
>
> You don’t want to block Googlebot, for example.
What are the bot , that shouldn’t block? and Is there any way find the bots name from the consolidated page view dashviews?
---
### Author:  [@Johani](https://meta.discourse.org/u/Johani)
#### Post date: [June 24, 2020, 11:22pm UTC](https://meta.discourse.org/t/too-many-crawlers-is-that-a-problem/148550/6 "2020-06-24T23:22:10Z")
You should only block bots that are misbehaving and leave everything else unblocked.
> [@salmanfarisvp](#):
>
> Is there any way find the bots name from the consolidated page view dashviews?
Like I mentioned in my previous post. If you go
```plaintext
your.site.com/admin/reports/web_crawlers
```
You’ll get a full list of all the crawlers on your site and the number of pageviews per crawler. Can you take a screenshot of that page and post it here?
---
### Author:  [@salmanfarisvp](https://meta.discourse.org/u/salmanfarisvp)
#### Post date: [June 25, 2020, 6:20am UTC](https://meta.discourse.org/t/too-many-crawlers-is-that-a-problem/148550/7 "2020-06-25T06:20:42Z")
> [@Johani](#):
>
> Can you take a screenshot of that page and post it here?
Here it is!
