Chunked backup uploader exceeds Discourse request rate limit

zogstrip · September 8, 2026, 8:03am

thanks for the report @RGJ will be fixed by

github.com/discourse/discourse

FIX: Retry rate limited backup upload chunks (#43373)

feature/rate-limit-client-helpers ← fix/retry-rate-limited-backup-chunks

merged 10:52AM - 15 Sep 26 UTC

ZogStriP

+228 -49

Stacked on #43372. Fixes the report in https://meta.discourse.org/t/chunked-back…up-uploader-exceeds-discourse-request-rate-limit/411386. Uploading a local backup through the admin UI fails outright on a fast connection. The uploader sends one POST per chunk, so throughput alone decides the request rate, and a large backup over a quick link trips the per-user request limiter. The server answers 429 and the upload dies, discarding however many gigabytes had already been sent. There is no resume, so a retry starts from chunk 1. `_shouldRetry` allowed status 0, 409, 423 and 5xx. That list came from uppy's S3 multipart plugin, where it is correct — S3 does not answer 429 with a `Retry-After`. Discourse does, so the one status worth retrying was the one treated as fatal, and none of the four configured retry delays was ever consumed. ### Why three changes and not one Adding 429 to the allowlist alone makes it worse. The limiters run in aggressive mode, so a rejected request re-arms the window; retrying on the existing sub-second ladder turns a fast failure into a permanent block. Measured against a persistent 429 on a 6-chunk file: | | POSTs of the failing chunk | |---|---| | allowlist only | 15 in 10s, still fails | | \+ honour `Retry-After` | 6 | | \+ keep the chunk reserved | 2 — correct | The third is needed because clearing `busy` before rejecting let every sibling completion re-pick the same chunk and start a second retry chain that POSTed immediately, defeating the backoff. ### Chunk size stays at 5 MB An earlier revision of this PR raised it with the ladder S3 multipart has used since #22061. @Ethsim12 caught that this regresses stock installs, and reproduced it: these chunks go through the site's own nginx, whose shipped `client_max_body_size` is `10m`, and the multipart envelope puts a 10 MB chunk 962 bytes over that. nginx answers 413 before Rails sees the request, and 413 is not retryable — so every backup of 100 MB or more would have died on chunk 1. S3 parts have no proxy in front of them, which is why the ladder is correct there and stays there. ### Latent defects fixed alongside - Registering the abort listener called `xhr.abort()` immediately and registered its return value, so aborting was a no-op and in-flight chunks kept running after a failure was reported. - `chunkSize` was only assigned when no `getChunkSize` was supplied, so any caller providing one produced NaN bounds, a single empty chunk, and a silently truncated backup. - The pause plumbing called a method the class never defined; nothing could reach it, so it is gone. ### Not a regression from the Uppy migration Worth recording, since the linked topic asks. The 2021 report predates Uppy becoming the default backup uploader by two days — that reporter was on Resumable.js, which classified 429 as retryable but retried immediately, up to 100 times, against a limiter that was already aggressive. 429 has never been survivable here. What the port changed was the failure mode, from slow-and-noisy to instant-and-silent. The capability actually lost was resume, which this PR does not restore.

Topic		Replies	Views
Backup uploads are not protected against 429s Bug backups	2	192	June 30, 2025
Rate limiter issues when uploading a backup file / can't disable rate limiter Self-hosting	4	912	March 31, 2022
Error 404 when uploading a backup Support	2	491	May 18, 2018
Increase chunksize for Discourse backups using Scaleway S3 Support backups , s3	0	63	January 14, 2025
Backup uploads hang at 2% complete Support backups	1	1529	February 5, 2015

Chunked backup uploader exceeds Discourse request rate limit

Related topics