Hi, In our forum we got too much Too many Crawlers, it’ like 81k to 90k as average! So Is that will cause any problems? if yes how can I disable Bad Crawlers from crawling?
I listed blacklisted crawler user agents as
also slow down crawler user agents as
Thanks.
You can see detailed numbers on a per-crawler basis if you visit
your.site.com/admin/reports/web_crawlers
If you see an unusual crawler that’s misbehaving, you can add a unique string from its user agent to the blacklisted_crawler_user_agents setting and it will be blocked.
Thanks for the quick replay. I found some crawler has 1695285 views, is that normal or not?
also on more question how can add the solution button in my forum, Do I need to install any plugins?
Thanks.
Well, that depends on how large your site is and the user agent for that crawler. You don’t want to block Googlebot, for example.
Have a look here
What are the bot , that shouldn’t block? and Is there any way find the bots name from the consolidated page view dashviews?
You should only block bots that are misbehaving and leave everything else unblocked.
Like I mentioned in my previous post. If you go
your.site.com/admin/reports/web_crawlers
You’ll get a full list of all the crawlers on your site and the number of pageviews per crawler. Can you take a screenshot of that page and post it here?
