Too many Crawlers, is that a problem?
salmanfarisvp is concerned about the large number of crawlers (81k-90k) on their forum and wonders if it will cause problems. They’ve listed some blacklisted crawler user agents, including mauibot and ahrefsbot, and slow-down crawler user agents, including bingbot.
Johani suggests visiting your.site.com/admin/reports/web_crawlers to see detailed numbers on a per-crawler basis and to block misbehaving crawlers. They also advise not to block Googlebot, as an example.
salmanfarisvp found a crawler with 1,695,285 views and asks if it’s normal. They also inquire about adding a solution button to their forum, which Johani responds to by suggesting the Discourse Solved/Accepted Answer plugin.
Johani reiterates that only misbehaving bots should be blocked and suggests reviewing the list of crawlers on the your.site.com/admin/reports/web_crawlers page.
salmanfarisvp provides a screenshot of the page, which Johani requested to help identify the bots.
Hi, In our forum we got too much Too many Crawlers, it’ like 81k to 90k as average! So Is that will cause any problems? if yes how can I disable Bad Crawlers from crawling?
I listed blacklisted crawler user agents as
also slow down crawler user agents as
Thanks.
You can see detailed numbers on a per-crawler basis if you visit
your.site.com/admin/reports/web_crawlers
If you see an unusual crawler that’s misbehaving, you can add a unique string from its user agent to the blacklisted_crawler_user_agents setting and it will be blocked.
Thanks for the quick replay. I found some crawler has 1695285 views, is that normal or not?
also on more question how can add the solution button in my forum, Do I need to install any plugins?
Thanks.
Well, that depends on how large your site is and the user agent for that crawler. You don’t want to block Googlebot, for example.
Have a look here
What are the bot , that shouldn’t block? and Is there any way find the bots name from the consolidated page view dashviews?
You should only block bots that are misbehaving and leave everything else unblocked.
Like I mentioned in my previous post. If you go
your.site.com/admin/reports/web_crawlers
You’ll get a full list of all the crawlers on your site and the number of pageviews per crawler. Can you take a screenshot of that page and post it here?
