The discussion revolves around the handling of crawler throttling in Discourse. vchuravy pointed out that Discourse throttling is implemented by throwing an error, which returns a generic 505 Internal Server Error, instead of the recommended HTTP 429 (Too many requests) or 503 (Site unavailable) status code. This has caused issues with Google Search Console, which reports 5xx errors and drastically reduced indexing and impressions. Falco clarified that the error is rescued by a global controller, which sets the status code. However, vchuravy and vchuravy further explained that Google Search Console still reports 5xx errors, even for the sitemap.xml file, which is problematic and likely caused a gap in coverage.
My Ruby on Rails days are long behind me, but I am assuming that this raises a generic 505?
The Google crawler doesn’t quite understand discourse’s throttling and in Google Search Console I can see that our indexing and therefore impressions drastically reduced after throttling was implemented, but not due to throttling, but due to 5xx server errors.
I understand that throttling instances may be sometimes necessary if they cause to much traffic, but I was expecting that discourse reports a HTTP 429, instead of serving the crawler a 505 Internal Error.
Thanks! That’s reassuring, but doesn’t quite explain why Google Search Console is reporting 5xx errors that correlate with the moment throttling was implemented.
