429 too many requests

Product Support Self-hosting

AI-generated summary

izzy reports intermittent 429 “Too Many Requests” errors in their Discourse instance, despite low server load and no obvious misconfiguration. The errors consistently trace back to a large JS file and coincide with a traffic spike resembling a DDoS attack. codinghorror asks if the site is behind a reverse proxy affecting IP tracking. izzy confirms it’s behind an AWS ELB and notes logs show normal IP capture, questioning why rate limiting affects all users rather than just high-traffic ones. Falco suggests the issue may stem from Discourse’s internal nginx using the ELB’s IP instead of the client’s IP for rate limiting. izzy confirms the ELB setup and seeks guidance on whether this is a misconfiguration and how to resolve it with their ops team.

1

Hey everyone, I know there are “too many requests” topics before, but this one seems to not 100% fit the bill.

I’m getting intermittent 429 errors in discourse (and everything’s fairly slow in general) with this backtrace

Error: Too Many Requests
    at s (https://discourse.looker.com/assets/ember_jquery-d430881a3fb1403871256e5a02423c4b20a78793685e92088613ca9a701baf88.js:9:9188)
    at a (https://discourse.looker.com/assets/ember_jquery-d430881a3fb1403871256e5a02423c4b20a78793685e92088613ca9a701baf88.js:9:9045)
    at o (https://discourse.looker.com/assets/ember_jquery-d430881a3fb1403871256e5a02423c4b20a78793685e92088613ca9a701baf88.js:9:8936)
    at Object.trigger (https://discourse.looker.com/assets/ember_jquery-d430881a3fb1403871256e5a02423c4b20a78793685e92088613ca9a701baf88.js:18:7223)
    at https://discourse.looker.com/assets/ember_jquery-d430881a3fb1403871256e5a02423c4b20a78793685e92088613ca9a701baf88.js:18:9212
    at t.invoke (https://discourse.looker.com/assets/ember_jquery-d430881a3fb1403871256e5a02423c4b20a78793685e92088613ca9a701baf88.js:16:9729)
    at e.t.flush (https://discourse.looker.com/assets/ember_jquery-d430881a3fb1403871256e5a02423c4b20a78793685e92088613ca9a701baf88.js:16:8732)
    at e.t.flush (https://discourse.looker.com/assets/ember_jquery-d430881a3fb1403871256e5a02423c4b20a78793685e92088613ca9a701baf88.js:16:10782)
    at e.n._end (https://discourse.looker.com/assets/ember_jquery-d430881a3fb1403871256e5a02423c4b20a78793685e92088613ca9a701baf88.js:16:15440)
    at e.n.end (https://discourse.looker.com/assets/ember_jquery-d430881a3fb1403871256e5a02423c4b20a78793685e92088613ca9a701baf88.js:16:12110)

It’s always that .js link, which is a pretty impressive block of JS that doesn’t really mean anything to me. The instance running discourse seems to be under minimal load— 10% of CPU, everything else seems fairly nominal, so it’s a bit confusing to me that I’d be getting 429 errors.

Are there settings I can up somewhere to change global rate limiting? The instance can handle much more load than discourse seems to think, or I’m missing something greater caused by a plugin or bug.

Thanks!

2

Is your site behind a reverse proxy or anything else that would mess with real incoming IP addresses?

3

No, I don’t believe so (and the logs capture IP addresses normally) but it does sit behind an ELB. It definitely did correlate with a significant increase in traffic (looks kind of like a DDOS attack or something)

image
image388×420 2.78 KB

But if my understanding of rate limiting is correct, that shouldn’t have affected every user— Only the user trying to visit a million times, right?

I’ll verify re: the network architecture. Thanks!

4

I believe the nginx inside the Discourse container is using the ELB IP for the rate limiting bucket instead of the original client IP.

5

That sounds quite possible. Just verified that we do have an ELB setup in aws with nothing particularly special about it— Is is the result of something that I have misconfigured on my end?

Not 100% sure what next steps would be. If you point me in the right direction, I can probably work with my ops team to figure it out. Thanks much!