Hi, In our forum we got too much Too many Crawlers, it’ like 81k to 90k as average! So Is that will cause any problems? if yes how can I disable Bad Crawlers from crawling? I listed blacklisted crawler user agents as mauibot semruhbot ahrefsbot blexbot SEO spider also slow down crawler user agen…

You can see detailed numbers on a per-crawler basis if you visit your.site.com/admin/reports/web_crawlers If you see an unusual crawler that’s misbehaving, you can add a unique string from its user agent to the blacklisted_crawler_user_agents setting and it will be blocked.

Thanks for the quick replay. I found some crawler has 1695285 views, is that normal or not? also on more question how can add the solution button in my forum, Do I need to install any plugins? Thanks.

[이미지] salmanfarisvp: I found some crawler has 1695285 views, is that normal or not? Well, that depends on how large your site is and the user agent for that crawler. You don’t want to block Googlebot, for example. [이미지] salmanfarisvp: how can add the solution button in my forum Have a…

[이미지] Johani: You don’t want to block Googlebot, for example. What are the bot , that shouldn’t block? and Is there any way find the bots name from the consolidated page view dashviews?

You should only block bots that are misbehaving and leave everything else unblocked. [이미지] salmanfarisvp: Is there any way find the bots name from the consolidated page view dashviews? Like I mentioned in my previous post. If you go your.site.com/admin/reports/web_crawlers You’ll get a ful…

[이미지] Johani: Can you take a screenshot of that page and post it here? Here it is! [Capture]

크롤러가 너무 많으면 문제가 되나요?

커뮤니티 구축 데이터 및 보고

Johani (Joe) 4월 21, 2020, 1:22오전 4

글쎄요, 그것은 사이트의 크기와 해당 크롤러의 사용자 에이전트(user agent)에 따라 달라집니다. 예를 들어 Googlebot은 차단하면 안 됩니다.

여기를 참고해 보세요.

글		댓글	조회수
Controlling Web Crawlers For a Site Site Management how-to	10	2803	7월 19, 2025
MegaIndex bot did about 4,000 pageviews on one day Community Building	40	5063	12월 2, 2023
Smarter handling of random crawler traffic Feature	1	3637	3월 29, 2018
Spiders are making up the majority of our traffic! Product Support	3	164	8월 15, 2024
Very high user profile views Product Support	5	977	4월 30, 2020

크롤러가 너무 많으면 문제가 되나요?

Related topics