I was checking the Google Search Console coverage report and found that lots of our forum pages are blocked by robots.txt. So I went ahead and checked the robots.txt. Then I found that semrushbot and ahrefsbot are blocked by default: [image] I know these are two widely used SEO tools, why blockin…

[이미지] Jamie_Liu1: I know these are two widely used SEO tools, why blocking their bots? Because they suck? They add a lot of server load for no discernable benefit, and our customers do have pageview limits on their plans.

[이미지] Jamie_Liu1: I know these are two widely used SEO tools, why blocking their bots? Because those bots are “resource sucking bot hogs” which provide very little value to sites compared to the amount of resources these bots consume. Of course, you can customize the Discourse robots.txt fil…

Thanks so much for the information! I found another issue and maybe you can share your insight on it as well. :slight_smile: I know Discourse has blocked user pages by default, but in my Google Search Console coverage report, there are still some user pages indexed, which is an issue in Google’s e…

This was fixed recently with https://github.com/discourse/discourse/commit/13f229808a22db9e1032832a313ab701b66614c8 Can you update your Discourse and reverify?

@osioke Thanks for your reply! I believe our installed version already has the feature? Because I noticed that the fix was committed in Jan. [image] Could you please verify if I need to upgrade to the latest version to have this feature?

It doesn’t hurt to update IMO, but yes, that fix should be in your installed version. I would try updating and reverifying unless you don’t want to update for some other reason.

Sounds good. We are updating now. Hope things will work out after the update. I’ll get back and keep you informed. :slight_smile: Thanks!

Just to clarify, is there no way to unblock semrushbot and seo spider? We need them for SEO audit. Tried removing both from /admin/customize/robots (also tried Allow: ) but we get 429 error in Screaming Frog. Or is this 429 error a separate issue? Your insights are highly appreciated.

429 errors mean that those crawlers are getting rate-limited. Discourse has some throttling enabled by default to prevent abuse. You can read more about this here. [이미지] Available settings for global rate limits and throttling Self-Hosting Discourse ships with…

[이미지] trying2survive: Just to clarify, is there no way to unblock semrushbot and seo spider? Did you try this (but use your container name)? Note: you can also configure this in the Admin UI: # docker exec -it socket-only bash root@socket-only:/# rails c [1] pry(main)> SiteSetting.blocked_c…

왜 semrushbot과 ahrefsbot이 기본적으로 차단되나요?

제품 지원

codinghorror (Jeff Atwood) 7월 15, 2020, 9:41오후 8

그냥 별로라서요? 아무런 이점도 없이 서버 부하를 많이 유발하고, 우리 고객님들은 요금제에 페이지뷰 한도가 있기 때문입니다.

AI Citations - Does Discourse allow AI bot crawling?

글		댓글	조회수
Handling Bingbot Feature	28	7762	11월 20, 2020
MegaIndex bot did about 4,000 pageviews on one day Community Building	40	5064	12월 2, 2023
Issues Google Search Console is throwing at me for wrong discourse structure (or some for wrong administration of my site) Product Support	18	403	12월 18, 2024
Why there are lots of Disallow rule in robots.txt? Product Support	34	4958	12월 22, 2020
How to protect myself from bots crawling my Discourse instance? Product Support	4	1718	1월 21, 2019

왜 semrushbot과 ahrefsbot이 기본적으로 차단되나요?

Related topics