Troubleshooting a 429 (rate limit)


(AstonJ) #1

One of my forums went down yesterday, first with a blank page then displaying 429’s. Sam thinks it may be possible that my proxy is not configured correctly - anyone know what the best way is to ascertain whether IP addresses are being handed over correctly to Discourse?

I’m pretty sure it is set up correctly (as I have option forwardfor set in HAProxy and users’ IP addresses correctly show up in the admin CP) but I’d like to check Discourse’s end just in case.


(Matt Palmer) #2

That’s the important bit. If the Discourse admin panel is listing users’ IP addresses correctly, then the IP addresses are getting into Discourse.


(AstonJ) #3

Thanks Matt.

I wonder what caused the issues yesterday then - reading this it appears that rate limits are applied per user or per IP and so it was odd that the site was inaccessible to everyone (I also tried using various IPs).

The forum has been getting busier lately - we’re now serving around 600K pages a month, but I wouldn’t have thought that would trigger this in itself.

Would the rate limiter only show a 429 to the IPs affected, or everyone?

Any other ideas what it might have been or how to troubleshoot this?


(Sam Saffron) #4

Some of the throttling happens in NGINX though not in Discourse. So you need NGINX to have a set_real_ip_from going.


(Matt Palmer) #5

If nginx’s set_real_ip_from is misconfigured, though, won’t Discourse not be able to get the real IP, either?


(Sam Saffron) #6

It actually will, cause as long as Discourse sees the HTTP X-Forwarded-For header it is fine to set the IP. So you can have a state where IP looks good in Discourse, but NGINX is not “setting real IP” for rate limiting purposes.


(AstonJ) #7

Thanks Sam.

So do I basically need to do what’s in this post? Or is there a simpler way?

If following that post, I guess for me that file would read something like the following?

run:
  - replace:
     filename: "/etc/nginx/conf.d/discourse.conf"
     from: /^add_header Strict-Transport-Security 'max-age=31536000';$/
     to: |
       add_header Strict-Transport-Security 'max-age=31536000';

       # Cloudflare
       set_real_ip_from   my.server.ip;
       real_ip_header     CF-Connecting-IP;

(It’s a dedicated server running a single IP address.)


(Matt Palmer) #8

If nginx isn’t stripping untrusted XFF, and Discourse is seeing a request from 127.0.0.1 and saying “I trust that IP to give me legit XFF headers”, doesn’t that imply that source IP can be spoofed?


(Sam Saffron) #9

Possibly, we should test, maybe our nginx template is lacking


#10

i actually just got this error. working through it now…


we’re back up. that seemed totally random.