I was checking the Google Search Console coverage report and found that lots of our forum pages are blocked by robots.txt. So I went ahead and checked the robots.txt. Then I found that semrushbot and ahrefsbot are blocked by default: [image] I know these are two widely used SEO tools, why blocking…

[صورة] Jamie_Liu1: I know these are two widely used SEO tools, why blocking their bots? Because they suck? They add a lot of server load for no discernable benefit, and our customers do have pageview limits on their plans.

Why semrushbot and ahrefsbot are blocked by default?

الدعم

osioke (Osioke Itseuwa) 14 يوليو 2020، 12:35م 4

This was fixed recently with

Can you update your Discourse and reverify?

إعجاب واحد (1)

الموضوع		الردود	مرات العرض
Handling Bingbot Feature	29	7414	20 نوفمبر 2020
MegaIndex bot did about 4,000 pageviews on one day Community	40	4526	2 ديسمبر 2023
Issues Google Search Console is throwing at me for wrong discourse structure (or some for wrong administration of my site) Support	18	190	18 ديسمبر 2024
Why there are lots of Disallow rule in robots.txt? Support	34	4562	22 ديسمبر 2020
How to protect myself from bots crawling my Discourse instance? Support	6	1607	17 يناير 2022

Why semrushbot and ahrefsbot are blocked by default?

الموضوعات ذات الصلة