Hi, In our forum we got too much Too many Crawlers, it’ like 81k to 90k as average! So Is that will cause any problems? if yes how can I disable Bad Crawlers from crawling? I listed blacklisted crawler user agents as mauibot semruhbot ahrefsbot blexbot SEO spider also slow down crawler user agen…

You can see detailed numbers on a per-crawler basis if you visit your.site.com/admin/reports/web_crawlers If you see an unusual crawler that’s misbehaving, you can add a unique string from its user agent to the blacklisted_crawler_user_agents setting and it will be blocked.

Thanks for the quick replay. I found some crawler has 1695285 views, is that normal or not? also on more question how can add the solution button in my forum, Do I need to install any plugins? Thanks.

[圖片] salmanfarisvp: I found some crawler has 1695285 views, is that normal or not? Well, that depends on how large your site is and the user agent for that crawler. You don’t want to block Googlebot, for example. [圖片] salmanfarisvp: how can add the solution button in my forum Have a l…

[圖片] Johani: You don’t want to block Googlebot, for example. What are the bot , that shouldn’t block? and Is there any way find the bots name from the consolidated page view dashviews?

You should only block bots that are misbehaving and leave everything else unblocked. [圖片] salmanfarisvp: Is there any way find the bots name from the consolidated page view dashviews? Like I mentioned in my previous post. If you go your.site.com/admin/reports/web_crawlers You’ll get a full…

[圖片] Johani: Can you take a screenshot of that page and post it here? Here it is! [Capture]

爬虫太多，会有问题吗？

社区数据与报告

Johani (Joe) 2020年四月21日 01:22 4

这取决于你的网站规模以及该爬虫的用户代理。例如，你不应该屏蔽 Googlebot。

请查看此处

話題		回覆	觀看
Controlling Web Crawlers For a Site Site Management how-to	10	2751	2025年七月19日
MegaIndex bot did about 4,000 pageviews on one day Community Building	40	4985	2023年十二月2日
Smarter handling of random crawler traffic Feature	1	3620	2018年三月29日
Spiders are making up the majority of our traffic! Support	3	155	2024年八月15日
Very high user profile views Support	5	955	2020年四月30日

爬虫太多，会有问题吗？

相關主題