Chunked backup uploader exceeds Discourse request rate limit

Contribute Bug
1

One of our customers ran into a reproducible issue when uploading a local Discourse backup through the admin UI.

The backup uploader uses fixed 5 MB chunks and, on a sufficiently fast connection, can generate enough requests to exceed Discourse’s own application-level request rate limit. When that happens, the server returns HTTP 429, but the chunked uploader does not retry 429 responses or honor the Retry-After header. As a result, the entire backup upload fails.

Non-2xx responses are converted to an error:

if (ev.target.status < 200 || ev.target.status >= 300) {
  const error = new Error("Non 2xx");
  error.source = ev.target;
  reject(error);
  return;
}

The retry logic is:

_shouldRetry(err) {
  if (err.source && typeof err.source.status === "number") {
    const { status } = err.source;

    return (
      status === 0 ||
      status === 409 ||
      status === 423 ||
      (status >= 500 && status < 600)
    );
  }

  return false;
}

HTTP 429 is therefore explicitly not retryable.

One 429 response is enough for the chunk upload promise to fail, eventually triggering upload-error for the entire backup upload.

There is also no handling of the Retry-After response header.

I found this topic from 2021 which derailed pretty quickly but seems to be the exact same thing

3

thanks for the report @RGJ :+1: will be fixed by

3 Likes