Too many Crawlers, is that a problem?

Data & reporting
1

Hi, In our forum we got too much Too many Crawlers, it’ like 81k to 90k as average! So Is that will cause any problems? if yes how can I disable Bad Crawlers from crawling?

I listed blacklisted crawler user agents as

  • mauibot
  • semruhbot
  • ahrefsbot
  • blexbot
  • SEO spider

also slow down crawler user agents as

  • bingbot

Thanks.

1 Like
2

You can see detailed numbers on a per-crawler basis if you visit

your.site.com/admin/reports/web_crawlers

If you see an unusual crawler that’s misbehaving, you can add a unique string from its user agent to the blacklisted_crawler_user_agents setting and it will be blocked.

4 Likes
3

Thanks for the quick replay. I found some crawler has 1695285 views, is that normal or not?

also on more question how can add the solution button in my forum, Do I need to install any plugins?

Thanks.

3 Likes
4

Well, that depends on how large your site is and the user agent for that crawler. You don’t want to block Googlebot, for example.

Have a look here

3 Likes
5

What are the bot , that shouldn’t block? and Is there any way find the bots name from the consolidated page view dashviews?

6

You should only block bots that are misbehaving and leave everything else unblocked.

Like I mentioned in my previous post. If you go

your.site.com/admin/reports/web_crawlers

You’ll get a full list of all the crawlers on your site and the number of pageviews per crawler. Can you take a screenshot of that page and post it here?

1 Like
7

Here it is!

Capture
Capture990×551 28.6 KB