Rate limiter issues when uploading a backup file / can't disable rate limiter

I’m moving an existing Discourse instance from its current home to an AWS EC2 VM.

The site sits behind Amazon load balancers. I’ve already successfully tweaked app.yml with the appropriate set_real_ip_from directives so NGINX knows about the load balancer IPs. I know this is working because I can see my real IP when I check my user’s last IP.

However, when I attempt to upload the 375 MB backup from the old server, the new site starts emitting 429 errors when the file is about 35% uploaded and the upload process fails. The headers in the 429 responses identify “discourse-rate-limit-error-code: id_10_secs_limit”.

I found this surprising. I had my browser’s Dev Tools window open on the “Network” tab and saw lots of small chunks being uploaded (I think they were 5 MB) fairly quickly. I’m on a fast 200mbps Internet connection so maybe it’s simply too fast for the default rate limiter settings? Or maybe the expectation was that admin tasks would usually be done from a private net (which isn’t possible with my AWS setup)?

But wait, there’s more! I tried to disable the rate limiter by commenting out the “templates/web.ratelimited.template.yml” line from app.yml and rebuilding the app but that didn’t work. I still got 429 errors when my file was about 35% uploaded.

So, I quickly added the below env variables to the app.yml file, rebuilt and was finally able to upload the backup to be restored.

DISCOURSE_MAX_REQS_PER_IP_MODE: none
DISCOURSE_MAX_REQS_PER_IP_PER_10_SECONDS: 1000

I believe these are rate limiter settings so it was odd to see something responding to these settings when I the rate limiter was supposedly disabled.

Bottom line - I’d love some guidance on:

  1. Should the rate limiter be blocking backup uploads like this?
  2. Why wasn’t the rate limiter disabled when I commented out that line and rebuilt the app?

Thanks!

If you’re trying to upload a backup then use scp or put it on S3.

By “use scp” do you mean transfer the file directly to the VM and then place it in a certain folder where Discourse can find it? I haven’t tried this, but it looks like that certain folder would be /var/discourse/shared/standalone/backups/default?

And would using S3 really bypass the Amazon load balancers?

Thanks!

1 Like

If that folder exists, then that’s probably the place.

It would.

1 Like

Default restore guide calls for creating the default folder so even if it doesn’t exist, it can be created manually.

Ref: Restore a backup from command line

2 Likes