Why semrushbot and ahrefsbot are blocked by default?

429 errors mean that those crawlers are getting rate-limited. Discourse has some throttling enabled by default to prevent abuse. You can read more about this here.

3 Likes