I was checking the Google Search Console coverage report and found that lots of our forum pages are blocked by robots.txt. So I went ahead and checked the robots.txt. Then I found that semrushbot and ahrefsbot are blocked by default: [image] I know these are two widely used SEO tools, why blocking…

[image] Jamie_Liu1: I know these are two widely used SEO tools, why blocking their bots? Because they suck? They add a lot of server load for no discernable benefit, and our customers do have pageview limits on their plans.

Why semrushbot and ahrefsbot are blocked by default?

support

osioke (Osioke Itseuwa) July 14, 2020, 12:35pm 4

This was fixed recently with

https://github.com/discourse/discourse/commit/13f229808a22db9e1032832a313ab701b66614c8

Can you update your Discourse and reverify?

1 Like

Topic		Replies	Views
Why there are lots of Disallow rule in robots.txt? support	34	4221	December 22, 2020
Google changed how they process robots.txt in Discourse? support	20	1462	December 22, 2020
Generic rules in "robots.txt" not picked up by Googlebot support	6	829	April 2, 2022
Google complaining – Indexed, though blocked by robots.txt support	24	2229	September 28, 2023
Pages listed in the robots.txt are crawled and indexed by Google support	19	2785	July 30, 2019

Why semrushbot and ahrefsbot are blocked by default?

Related Topics