I had a general question how crawler throttling is implemented. According to Reduce Google Crawl Rate | Google Crawling Infrastructure | Crawling infrastructure | Google for Developers the recommended HTTP status is 429 (Too many requests) or 503 (Site unavailable). But reading through the source code it looks like throttling is implemented by throwing an error: discourse/lib/rate_limiter.rb at 85fddf58bc1e751d0ac5b8192a630c59a34aed7d · discourse/discourse · GitHub My Ruby on Rails days are long behind me, but I am assuming that this raises a generic 505? The Google crawler doesn’t quite understand discourse’s throttling and in Google Search Console I can see that our indexing and therefore impressions drastically reduced after throttling was implemented, but not due to throttling, but due to 5xx server errors. I understand that throttling instances may be sometimes necessary if they cause to much traffic, but I was expecting that discourse reports a HTTP 429, instead of serving the crawler a 505 Internal Error.

[image] Valentin Churavy: My Ruby on Rails days are long behind me, but I am assuming that this raises a generic 505? I think what you looking for is app/controllers/application_controller.rb 85fddf58b rescue_from RateLimiter::LimitExceeded do |e| retry_time_in_seconds = e&.available_in response_headers = { "Retry-After": retry_time_in_seconds.to_s } response_headers["Discourse-Rate-Limit-Error-Code"] = e.error_code if e&.error_code with_resolved_locale do render_json_error( e.description, type: :rate_limit, status: 429, extras: { wait_seconds: retry_time_in_seconds, time_left: e&.time_left, }, headers: response_headers, ) end end Which is the “global” controller rescue for that error that sets the status code.

Thanks! That’s reassuring, but doesn’t quite explain why Google Search Console is reporting 5xx errors that correlate with the moment throttling was implemented. [image] [image] It even reports that it couldn’t fetch the discourse sitemap.xml

In particular throttling the sitemap.xml seems problematic. [image] I assume that’s what caused the gap in coverage. I could believe Google misreporting 429 as 5xx.

Handling of crawler throttling

Product Support

vchuravy (Valentin Churavy) September 12, 2023, 6:34pm 4

In particular throttling the sitemap.xml seems problematic.

I assume that’s what caused the gap in coverage. I could believe Google misreporting 429 as 5xx.

Topic		Replies	Views
Discourse API Generating 429 Product Support	6	764	March 28, 2023
429 too many requests Self-hosting	4	2898	September 13, 2019
The only solution I have found to workaround «429 Too Many Requests» failure from rubygems.org Product Support	4	2639	October 27, 2017
User_10_secs_limit Development	6	388	July 10, 2025
Understanding /logs/report_js_error 429 Product Support	2	1065	February 16, 2020

Handling of crawler throttling

Related topics