429 too many requests

تثبيت
1

Hey everyone, I know there are “too many requests” topics before, but this one seems to not 100% fit the bill.

I’m getting intermittent 429 errors in discourse (and everything’s fairly slow in general) with this backtrace

Error: Too Many Requests
    at s (https://discourse.looker.com/assets/ember_jquery-d430881a3fb1403871256e5a02423c4b20a78793685e92088613ca9a701baf88.js:9:9188)
    at a (https://discourse.looker.com/assets/ember_jquery-d430881a3fb1403871256e5a02423c4b20a78793685e92088613ca9a701baf88.js:9:9045)
    at o (https://discourse.looker.com/assets/ember_jquery-d430881a3fb1403871256e5a02423c4b20a78793685e92088613ca9a701baf88.js:9:8936)
    at Object.trigger (https://discourse.looker.com/assets/ember_jquery-d430881a3fb1403871256e5a02423c4b20a78793685e92088613ca9a701baf88.js:18:7223)
    at https://discourse.looker.com/assets/ember_jquery-d430881a3fb1403871256e5a02423c4b20a78793685e92088613ca9a701baf88.js:18:9212
    at t.invoke (https://discourse.looker.com/assets/ember_jquery-d430881a3fb1403871256e5a02423c4b20a78793685e92088613ca9a701baf88.js:16:9729)
    at e.t.flush (https://discourse.looker.com/assets/ember_jquery-d430881a3fb1403871256e5a02423c4b20a78793685e92088613ca9a701baf88.js:16:8732)
    at e.t.flush (https://discourse.looker.com/assets/ember_jquery-d430881a3fb1403871256e5a02423c4b20a78793685e92088613ca9a701baf88.js:16:10782)
    at e.n._end (https://discourse.looker.com/assets/ember_jquery-d430881a3fb1403871256e5a02423c4b20a78793685e92088613ca9a701baf88.js:16:15440)
    at e.n.end (https://discourse.looker.com/assets/ember_jquery-d430881a3fb1403871256e5a02423c4b20a78793685e92088613ca9a701baf88.js:16:12110)

It’s always that .js link, which is a pretty impressive block of JS that doesn’t really mean anything to me. The instance running discourse seems to be under minimal load— 10% of CPU, everything else seems fairly nominal, so it’s a bit confusing to me that I’d be getting 429 errors.

Are there settings I can up somewhere to change global rate limiting? The instance can handle much more load than discourse seems to think, or I’m missing something greater caused by a plugin or bug.

Thanks!

إعجابَين (2)
2

Is your site behind a reverse proxy or anything else that would mess with real incoming IP addresses?

إعجاب واحد (1)
3

No, I don’t believe so (and the logs capture IP addresses normally) but it does sit behind an ELB. It definitely did correlate with a significant increase in traffic (looks kind of like a DDOS attack or something)
image

But if my understanding of rate limiting is correct, that shouldn’t have affected every user— Only the user trying to visit a million times, right?

I’ll verify re: the network architecture. Thanks!

4

I believe the nginx inside the Discourse container is using the ELB IP for the rate limiting bucket instead of the original client IP.

إعجاب واحد (1)
5

That sounds quite possible. Just verified that we do have an ELB setup in aws with nothing particularly special about it— Is is the result of something that I have misconfigured on my end?

Not 100% sure what next steps would be. If you point me in the right direction, I can probably work with my ops team to figure it out. Thanks much!

إعجاب واحد (1)