I am now taking a look at this again as we have launched a Discourse integration and we want to make sure we don’t run into any issues related to rate limiting.
I tested it with a new key to make sure it isn’t limited in any way. To be clear, what exactly do you mean by an admin API key?
It says, “API key has no restriction and all endpoints are accessible.”
I am testing this by making API requests from a local Python shell, so they are coming from the same IP address. We also ran into the rate limits when running a script on our server. In that case, all requests came from the same IP address.
I confirmed that the rate limit is hit with the following code:
async def get_topic_post_stream(topic_id):
url = f"{DISCOURSE_URL}/t/{topic_id}"
async with httpx.AsyncClient(headers=HEADERS) as client:
topic = await client.get(url)
return topic.status_code
async def get_topic_post_streams(topic_ids):
tasks = [functools.partial(get_topic_post_stream, topic_id) for topic_id in topic_ids]
topics = await aiometer.run_all(
tasks,
# max_per_second=1,
)
return topics
# Just get a slice of 15 of the topics in topic_ids for testing.
topics = asyncio.run(get_topic_post_streams(topic_ids[:15]))
Note that the max_per_second parameter is commented out, which results in no limits on the number of requests.
This completes in 2.05 s and 2 out of the 15 requests return 429.
When I run it with max_per_second=1, everything completes successfully.
Let me know if I can provide any more details. Thanks!
It seems to me that I should not be getting these 429s regardless of any settings mentioned in that post. In the example I provided, I sent 15 requests which is under all of the default API limits. I did this using an admin API key and username.
The example doesn’t exceed the following per-IP defaults:
It doesn’t even exceed the non-admin limits:
Changing DISCOURSE_MAX_REQS_PER_IP_MODE to warn or none did not help.
Am I missing something?
BTW, I changed the settings by editing app.yml and running ./launcher destroy app && ./launcher start app.
I can see in /var/log/nginx/access.log that the IP address is correct, so I don’t think Discourse considers all requests to be coming from the same IP.
EDIT: I just checked the response contents of one of the failed requests and noticed that it mentioned nginx:
<html>\r\n<head><title>429 Too Many Requests</title></head>\r\n<body>\r\n<center><h1>429 Too Many Requests</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n
I’ll do some more investigating on the topics that mention nginx.
location @discourse {
add_header Strict-Transport-Security 'max-age=31536000'; # remember the certificate for a year and automatically connect to HTTPS for this domain
limit_conn connperip 20;
limit_req zone=flood burst=12 nodelay;
limit_req zone=bot burst=100 nodelay;
proxy_set_header Host $http_host;
proxy_set_header X-Request-Start "t=${msec}";
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $thescheme;
proxy_pass http://discourse;
}
}
My remaining questions now are:
Should I edit both sections in order to match my Discourse settings? Or just the values for location @discourse?
What’s the correct way to modify these values and persist the changes across rebuilds?
I assume that I can edit the nginx config directly in the container then stop/start the container. But it looks like these values originally came from templates/web.ratelimited.template.yml and may be overwritten on a rebuild?
Oof, now we’re getting outside of my comfort zone I’m afraid.
If you are being rate-limited by nginx, then yes, fiddling with those settings and making them less restrictive makes sense. I’m not sure if Nginx can whitelist IP addresses?
