I was checking the Google Search Console coverage report and found that lots of our forum pages are blocked by robots.txt. So I went ahead and checked the robots.txt. Then I found that semrushbot and ahrefsbot are blocked by default: [image] I know these are two widely used SEO tools, why blocking…

[画像] Jamie_Liu1: I know these are two widely used SEO tools, why blocking their bots? Because they suck? They add a lot of server load for no discernable benefit, and our customers do have pageview limits on their plans.

Why semrushbot and ahrefsbot are blocked by default?

サポート

Johani (Joe) 2020 年 12 月 2 日午後 4:34 11

429 errors mean that those crawlers are getting rate-limited. Discourse has some throttling enabled by default to prevent abuse. You can read more about this here.

「いいね！」 3

トピック		返信	表示
Handling Bingbot Feature	29	7373	2020 年 11 月 20 日
MegaIndex bot did about 4,000 pageviews on one day Community	40	4473	2023 年 12 月 2 日
Issues Google Search Console is throwing at me for wrong discourse structure (or some for wrong administration of my site) Support	18	150	2024 年 12 月 18 日
Why there are lots of Disallow rule in robots.txt? Support	34	4528	2020 年 12 月 22 日
How to protect myself from bots crawling my Discourse instance? Support	6	1588	2022 年 1 月 17 日

Why semrushbot and ahrefsbot are blocked by default?

関連トピック