Troubleshooting a 429 (rate limit)

[mpalmer (Matt Palmer)]

users’ IP addresses correctly show up in the admin CP

That’s the important bit. If the Discourse admin panel is listing users’ IP addresses correctly, then the IP addresses are getting into Discourse.

[AstonJ]

Thanks Matt.

I wonder what caused the issues yesterday then - reading this it appears that rate limits are applied per user or per IP and so it was odd that the site was inaccessible to everyone (I also tried using various IPs).

The forum has been getting busier lately - we’re now serving around 600K pages a month, but I wouldn’t have thought that would trigger this in itself.

Would the rate limiter only show a 429 to the IPs affected, or everyone?

Any other ideas what it might have been or how to troubleshoot this?

[sam (Sam Saffron)]

Some of the throttling happens in NGINX though not in Discourse. So you need NGINX to have a set_real_ip_from going.

[mpalmer (Matt Palmer)]

If nginx’s set_real_ip_from is misconfigured, though, won’t Discourse not be able to get the real IP, either?

[sam (Sam Saffron)]

It actually will, cause as long as Discourse sees the HTTP X-Forwarded-For header it is fine to set the IP. So you can have a state where IP looks good in Discourse, but NGINX is not “setting real IP” for rate limiting purposes.

[AstonJ]

Thanks Sam.

So do I basically need to do what’s in this post? Or is there a simpler way?

If following that post, I guess for me that file would read something like the following?

run:
  - replace:
     filename: "/etc/nginx/conf.d/discourse.conf"
     from: /^add_header Strict-Transport-Security 'max-age=31536000';$/
     to: |
       add_header Strict-Transport-Security 'max-age=31536000';

       # Cloudflare
       set_real_ip_from   my.server.ip;
       real_ip_header     CF-Connecting-IP;

(It’s a dedicated server running a single IP address.)

[mpalmer (Matt Palmer)]

If nginx isn’t stripping untrusted XFF, and Discourse is seeing a request from 127.0.0.1 and saying “I trust that IP to give me legit XFF headers”, doesn’t that imply that source IP can be spoofed?

[sam (Sam Saffron)]

Possibly, we should test, maybe our nginx template is lacking

[8BIT]

i actually just got this error. working through it now…

---

we’re back up. that seemed totally random.

[AstonJ]

Coming back to this Topic as the other one is slightly different.

Do we still need to be doing something like in the post above?

Looking through the /etc/nginx/conf.d/discourse.conf file there seems to be no mention of set_real_ip_from (searching the Discourse repo for the same yields no results either). From the discourse.conf file these seem most relevant:

(mentions of IP)

    # This big block is needed so we can selectively enable
    # acceleration for backups and avatars
    # see note about repetition above
    location ~ ^/(letter_avatar/|user_avatar|highlight-js|stylesheets|favicon/proxied|service-worker) {
      proxy_set_header Host $http_host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Request-Start "t=${msec}";
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $thescheme;

    # we need buffering off for message bus
    location /message-bus/ {
      proxy_set_header X-Request-Start "t=${msec}";
      proxy_set_header Host $http_host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $thescheme;
      proxy_http_version 1.1;
      proxy_buffering off;
      proxy_pass http://discourse;
      break;
    }

    # auth_basic on;
    # auth_basic_user_file /etc/nginx/htpasswd;

    location ~* (assets|plugins|uploads)/.*\.(eot|ttf|woff|woff2|ico)$ {
      expires 1y;
      add_header Cache-Control public,immutable;
      add_header Access-Control-Allow-Origin *;
     }

    location = /srv/status {
      access_log off;
      log_not_found off;
      proxy_set_header Host $http_host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Request-Start "t=${msec}";
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $thescheme;
      proxy_pass http://discourse;
      break;
    }

Which appear to be setting the correct remote address

For completeness I received a couple of 429s after a peak in traffic a couple of days ago, and looking at my app.yml I found that the set_real_ip_from my.server.ip; and real_ip_header CF-Connecting-IP; lines were commented out… but it doesn’t look like they do anything anyway? Can you remember what the outcome of your investigations were after?

Currently I am recommending adding those lines to the app.yml in the following guide: How to set up Discourse on a server with existing Apache sites (and so may be giving bad advice to others?)

[AstonJ]

Some of the throttling happens in NGINX though not in Discourse. So you need NGINX to have a set_real_ip_from going.

Hi Sam, what’s the best way tot set (or check) this? It seems the other things I’ve tried in this Topic may not be working - we had an announcement posted on Twitter today and people are experiencing 429s

Dennis Tel (dennistel90)

I would not even class this as a huge amount of traffic (only had 25K views reported in the ACP up till now).

Any idea what’s going on?

[sam (Sam Saffron)]

Any idea what’s going on?

Yeah very likely NGINX thinks everyone has the same IP, so your rate limiting is kind of nonsense, to quickly mitigate I would recommend removing the rate limiting template and rebuilding. But longer term you are going to need to learn a bit of NGINX and figure out how to teach NGINX that everyone has different IPs using the set real ip extension

[AstonJ]

Thanks Sam.

Should this work?

run:
  - replace:
     filename: "/etc/nginx/conf.d/discourse.conf"
     from: /^add_header Strict-Transport-Security 'max-age=31536000';$/
     to: |
       add_header Strict-Transport-Security 'max-age=31536000';

       # IP
       set_real_ip_from   my.server.ip;

Only thing is I already have that, and can’t actually see anything called set_real_ip_from in /etc/nginx/conf.d/discourse.conf (as detailed in this post)

Any pointers would be greatly appreciated

[pfaffman (Jay Pfaffman)]

You can look at the logs (e.g., shared/standalone/log/var-log/nginx/access.log and see what IP numbers are getting through. You should be able to, for example, see your own IP when you load the site.

Do you have something external to the Discourse container doing a reverse proxy?

Are you using some external that’s doing a reverse proxy (e.g., CloudFlare)?

[AstonJ]

There’s HAProxy on the front, set up as per this guide:

How to set up Discourse on a server with existing Apache sites

It’s not as daunting as it sounds! As you probably know, the only supported installs of Discourse are the Docker installs. While this may sound a bit intimidating at first, it’s really not all that bad. You basically need to do two things: Install HAProxy (or an alternative) which will take over port 80 and then divert your Discourse traffic to your docker container, and all your other sites to your usual Apache set-up. Let Apache know which port to listen for. While this is only a rou…

When looking in the admin control panel, all user’s IPs are showing correctly (which I believe is because option forwardfor is set in HAProxy).

Other than that there is no cloudfare - however the datcentre does offer DDOS protection as standard (but don’t think that would make any difference) : /

[AstonJ]

Looking at other topics here, looks like I may need to do this?

- replace:
    filename: /etc/nginx/conf.d/discourse.conf
    from: "types {"
    to: |
      set_real_ip_from 10.0.0.0/24;
      set_real_ip_from 172.17.0.0/24;
      real_ip_header X-Forwarded-For;
      real_ip_recursive on;
      types {

From this post which was linked to from this one by Sam (and you (Jay) commented on it - mentioning HAProxy).

[pfaffman (Jay Pfaffman)]

Look at the nginx logs and see what IP addresses are there. Are they all the IP address of the magic DDOS protector that the data center provides?

That snippet might be a solution. Before fooling with doing it that way, you might go into the container and do it by hand to see if it works. (Or, if that doesn’t make sense to you, just blindly copy it and see if it works )

[AstonJ]

shared/standalone/log/var-log/nginx/access.log

All of the IPs are 172.17.0.1

So I’d guess I would just need the following?

- replace:
    filename: /etc/nginx/conf.d/discourse.conf
    from: "types {"
    to: |
      set_real_ip_from 172.17.0.0/24;
      real_ip_header X-Forwarded-For;
      real_ip_recursive on;
      types {

Tho not sure about these two:

      real_ip_header X-Forwarded-For;
      real_ip_recursive on;

you might go into the container and do it by hand to see if it works

Do you mean editing the /etc/nginx/conf.d/discourse.conf file directly inside the container then restarting Nginx?

[AstonJ]

Yay!! It’s done! IP’s are now showing correctly in the access log

For some reason the spacing messed up in the snippet above, here it is for anyone who might need it in future:

run:
  - replace:
      filename: /etc/nginx/conf.d/discourse.conf
      from: "types {"
      to: |
        set_real_ip_from 172.17.0.0/24;
        real_ip_header X-Forwarded-For;
        real_ip_recursive on;
        types {

Thank you @pfaffman and @sam and @mpalmer for your help