# Troubleshooting a 429 (rate limit)
**URL:** https://meta.discourse.org/t/troubleshooting-a-429-rate-limit/81060
**Category:** Self-hosting
**Created:** [20.Февраль.2018 18:20:41 UTC](https://meta.discourse.org/t/troubleshooting-a-429-rate-limit/81060 "2018-02-20T18:20:41Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@AstonJ](https://meta.discourse.org/u/AstonJ)
#### Post date: [20.Февраль.2018 18:20:41 UTC](https://meta.discourse.org/t/troubleshooting-a-429-rate-limit/81060/1 "2018-02-20T18:20:41Z")
One of my forums went down yesterday, first with a blank page then displaying 429’s. Sam thinks it may be possible that my proxy is not configured correctly - anyone know what the best way is to ascertain whether IP addresses are being handed over correctly to Discourse?
_I’m pretty sure it is set up correctly (as I have `option forwardfor` set in HAProxy and users’ IP addresses correctly show up in the admin CP) but I’d like to check Discourse’s end just in case._
---
### Author:  [@mpalmer](https://meta.discourse.org/u/mpalmer)
#### Post date: [20.Февраль.2018 21:33:38 UTC](https://meta.discourse.org/t/troubleshooting-a-429-rate-limit/81060/2 "2018-02-20T21:33:38Z")
> [@AstonJ](#):
>
> users’ IP addresses correctly show up in the admin CP
That’s the important bit. If the Discourse admin panel is listing users’ IP addresses correctly, then the IP addresses are getting into Discourse.
---
### Author:  [@AstonJ](https://meta.discourse.org/u/AstonJ)
#### Post date: [20.Февраль.2018 23:43:05 UTC](https://meta.discourse.org/t/troubleshooting-a-429-rate-limit/81060/3 "2018-02-20T23:43:05Z")
Thanks Matt.
I wonder what caused the issues yesterday then - reading [this](https://meta.discourse.org/t/global-rate-limits-and-throttling-in-discourse/78612) it appears that rate limits are applied per user or per IP and so it was odd that the site was inaccessible to everyone (I also tried using various IPs).
The forum has been getting busier lately - we’re now serving around 600K pages a month, but I wouldn’t have thought that would trigger this in itself.
Would the rate limiter only show a 429 to the IPs affected, or everyone?
Any other ideas what it might have been or how to troubleshoot this?
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [20.Февраль.2018 23:46:04 UTC](https://meta.discourse.org/t/troubleshooting-a-429-rate-limit/81060/4 "2018-02-20T23:46:04Z")
Some of the throttling happens in NGINX though not in Discourse. So you need NGINX to have a `set_real_ip_from` going.
---
### Author:  [@mpalmer](https://meta.discourse.org/u/mpalmer)
#### Post date: [20.Февраль.2018 23:56:34 UTC](https://meta.discourse.org/t/troubleshooting-a-429-rate-limit/81060/5 "2018-02-20T23:56:34Z")
If nginx’s `set_real_ip_from` is misconfigured, though, won’t Discourse not be able to get the real IP, either?
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [20.Февраль.2018 23:58:55 UTC](https://meta.discourse.org/t/troubleshooting-a-429-rate-limit/81060/6 "2018-02-20T23:58:55Z")
It actually will, cause as long as Discourse sees the HTTP X-Forwarded-For header it is fine to set the IP. So you can have a state where IP looks good in Discourse, but NGINX is not “setting real IP” for rate limiting purposes.
---
### Author:  [@AstonJ](https://meta.discourse.org/u/AstonJ)
#### Post date: [21.Февраль.2018 01:08:46 UTC](https://meta.discourse.org/t/troubleshooting-a-429-rate-limit/81060/7 "2018-02-21T01:08:46Z")
Thanks Sam.
So do I basically need to do what’s in [this](https://meta.discourse.org/t/config-nginx-to-receive-real-ip-when-using-cloudflare-for-noobs/22645/15) post? Or is there a simpler way?
If following that post, I guess for me that file would read something like the following?
```plaintext
run:
- replace:
filename: "/etc/nginx/conf.d/discourse.conf"
from: /^add_header Strict-Transport-Security 'max-age=31536000';$/
to: |
add_header Strict-Transport-Security 'max-age=31536000';
# Cloudflare
set_real_ip_from my.server.ip;
real_ip_header CF-Connecting-IP;
```
(It’s a dedicated server running a single IP address.)
---
### Author:  [@mpalmer](https://meta.discourse.org/u/mpalmer)
#### Post date: [21.Февраль.2018 01:54:46 UTC](https://meta.discourse.org/t/troubleshooting-a-429-rate-limit/81060/8 "2018-02-21T01:54:46Z")
If nginx isn’t stripping untrusted XFF, and Discourse is seeing a request from 127.0.0.1 and saying “I trust that IP to give me legit XFF headers”, doesn’t that imply that source IP can be spoofed?
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [21.Февраль.2018 02:31:49 UTC](https://meta.discourse.org/t/troubleshooting-a-429-rate-limit/81060/9 "2018-02-21T02:31:49Z")
Possibly, we should test, maybe our nginx template is lacking
---
### Author:  [@8BIT](https://meta.discourse.org/u/8BIT)
#### Post date: [22.Февраль.2018 03:21:51 UTC](https://meta.discourse.org/t/troubleshooting-a-429-rate-limit/81060/10 "2018-02-22T03:21:51Z")
i actually just got this error. working through it now…
* * *
we’re back up. that seemed totally random.
---
### Author:  [@AstonJ](https://meta.discourse.org/u/AstonJ)
#### Post date: [19.Сентябрь.2018 23:22:39 UTC](https://meta.discourse.org/t/troubleshooting-a-429-rate-limit/81060/11 "2018-09-19T23:22:39Z")
_Coming back to this Topic as the [other one](https://meta.discourse.org/t/intermittent-performance-issues/97391/) is slightly different._
Do we still need to be doing something like in [the post](https://meta.discourse.org/t/troubleshooting-a-429-rate-limit/81060/7) above?
Looking through the `/etc/nginx/conf.d/discourse.conf` file there seems to be no mention of `set_real_ip_from` (searching the Discourse repo for the same yields no results either). From the `discourse.conf` file these seem most relevant:
(mentions of `IP`)
```plaintext
# This big block is needed so we can selectively enable
# acceleration for backups and avatars
# see note about repetition above
location ~ ^/(letter_avatar/|user_avatar|highlight-js|stylesheets|favicon/proxied|service-worker) {
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Request-Start "t=${msec}";
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $thescheme;
```
```plaintext
# we need buffering off for message bus
location /message-bus/ {
proxy_set_header X-Request-Start "t=${msec}";
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $thescheme;
proxy_http_version 1.1;
proxy_buffering off;
proxy_pass http://discourse;
break;
}
```
```plaintext
# auth_basic on;
# auth_basic_user_file /etc/nginx/htpasswd;
location ~* (assets|plugins|uploads)/.*\.(eot|ttf|woff|woff2|ico)$ {
expires 1y;
add_header Cache-Control public,immutable;
add_header Access-Control-Allow-Origin *;
}
location = /srv/status {
access_log off;
log_not_found off;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Request-Start "t=${msec}";
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $thescheme;
proxy_pass http://discourse;
break;
}
```
Which appear to be setting the correct remote address 😕
For completeness I received a couple of `429s` after a peak in traffic a couple of days ago, and looking at my app.yml I found that the `set_real_ip_from my.server.ip;` and `real_ip_header CF-Connecting-IP;` lines were commented out… but it doesn’t look like they do anything anyway? Can you remember what the [outcome](https://meta.discourse.org/t/troubleshooting-a-429-rate-limit/81060/8) of your [investigations](https://meta.discourse.org/t/troubleshooting-a-429-rate-limit/81060/9) were after?
_Currently I am recommending adding those lines to the app.yml in the following guide: [Set up Discourse on a server with existing Apache sites](https://meta.discourse.org/t/how-to-set-up-discourse-on-a-server-with-existing-apache-sites/30013) (and so may be giving bad advice to others?)_
---
### Author:  [@AstonJ](https://meta.discourse.org/u/AstonJ)
#### Post date: [09.Октябрь.2018 22:01:03 UTC](https://meta.discourse.org/t/troubleshooting-a-429-rate-limit/81060/12 "2018-10-09T22:01:03Z")
> [@sam](#):
>
> Some of the throttling happens in NGINX though not in Discourse. So you need NGINX to have a `set_real_ip_from` going.
Hi Sam, what’s the best way tot set (or check) this? It seems the other things I’ve tried in this Topic may not be working - we had an announcement posted on Twitter today and people are experiencing 429s
>
I would not even class this as a huge amount of traffic (only had 25K views reported in the ACP up till now).
Any idea what’s going on?
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [09.Октябрь.2018 22:03:23 UTC](https://meta.discourse.org/t/troubleshooting-a-429-rate-limit/81060/13 "2018-10-09T22:03:23Z")
> [@AstonJ](#):
>
> Any idea what’s going on?
Yeah very likely NGINX thinks everyone has the same IP, so your rate limiting is kind of nonsense, to quickly mitigate I would recommend removing the rate limiting template and rebuilding. But longer term you are going to need to learn a bit of NGINX and figure out how to teach NGINX that everyone has different IPs using the set real ip extension
---
### Author:  [@AstonJ](https://meta.discourse.org/u/AstonJ)
#### Post date: [09.Октябрь.2018 22:10:48 UTC](https://meta.discourse.org/t/troubleshooting-a-429-rate-limit/81060/14 "2018-10-09T22:10:48Z")
Thanks Sam.
Should this work?
```plaintext
run:
- replace:
filename: "/etc/nginx/conf.d/discourse.conf"
from: /^add_header Strict-Transport-Security 'max-age=31536000';$/
to: |
add_header Strict-Transport-Security 'max-age=31536000';
# IP
set_real_ip_from my.server.ip;
```
Only thing is I already have that, and can’t actually see anything called `set_real_ip_from` in `/etc/nginx/conf.d/discourse.conf` (as detailed in [this post](https://meta.discourse.org/t/troubleshooting-a-429-rate-limit/81060/11))
Any pointers would be greatly appreciated 😊
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [09.Октябрь.2018 22:47:51 UTC](https://meta.discourse.org/t/troubleshooting-a-429-rate-limit/81060/15 "2018-10-09T22:47:51Z")
You can look at the logs (e.g., `shared/standalone/log/var-log/nginx/access.log` and see what IP numbers are getting through. You should be able to, for example, see your own IP when you load the site.
Do you have something external to the Discourse container doing a reverse proxy?
Are you using some external that’s doing a reverse proxy (e.g., CloudFlare)?
---
### Author:  [@AstonJ](https://meta.discourse.org/u/AstonJ)
#### Post date: [09.Октябрь.2018 22:54:29 UTC](https://meta.discourse.org/t/troubleshooting-a-429-rate-limit/81060/16 "2018-10-09T22:54:29Z")
There’s HAProxy on the front, set up as per this guide:
> [@Set up Discourse on a server with existing Apache sites](https://meta.discourse.org/t/how-to-set-up-discourse-on-a-server-with-existing-apache-sites/30013):
>
> It’s not as daunting as it sounds! While this may sound a bit intimidating at first, it’s really not all that bad. You basically need to do two things: Install HAProxy (or an alternative) which will take over port 80 and then divert your Discourse traffic to your docker container, and all your other sites to your usual Apache set-up. Let Apache know which port to listen for. While this is only a rough guide, it should do a good job of pointing you in the right direction. Let’s get star…
When looking in the admin control panel, all user’s IPs are showing correctly (which I believe is because `option forwardfor` is set in HAProxy).
Other than that there is no cloudfare - however the datcentre does offer DDOS protection as standard (but don’t think that would make any difference) : /
---
### Author:  [@AstonJ](https://meta.discourse.org/u/AstonJ)
#### Post date: [09.Октябрь.2018 22:58:33 UTC](https://meta.discourse.org/t/troubleshooting-a-429-rate-limit/81060/17 "2018-10-09T22:58:33Z")
Looking at other topics here, looks like I may need to do this?
```plaintext
- replace:
filename: /etc/nginx/conf.d/discourse.conf
from: "types {"
to: |
set_real_ip_from 10.0.0.0/24;
set_real_ip_from 172.17.0.0/24;
real_ip_header X-Forwarded-For;
real_ip_recursive on;
types {
```
From [this post](https://meta.discourse.org/t/last-ip-address-and-action-dispatch-trusted-proxies/50098/3) which was linked to from this one by Sam (and you (Jay) [commented](https://meta.discourse.org/t/haproxy-and-discourse-ip-issue/92387/3) on it - mentioning HAProxy).
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [09.Октябрь.2018 23:04:03 UTC](https://meta.discourse.org/t/troubleshooting-a-429-rate-limit/81060/18 "2018-10-09T23:04:03Z")
Look at the nginx logs and see what IP addresses are there. Are they all the IP address of the magic DDOS protector that the data center provides?
That snippet **might** be a solution. Before fooling with doing it that way, you might go into the container and do it by hand to see if it works. (Or, if that doesn’t make sense to you, just blindly copy it and see if it works 🙂 )
---
### Author:  [@AstonJ](https://meta.discourse.org/u/AstonJ)
#### Post date: [09.Октябрь.2018 23:21:54 UTC](https://meta.discourse.org/t/troubleshooting-a-429-rate-limit/81060/19 "2018-10-09T23:21:54Z")
> [@pfaffman](#):
>
> shared/standalone/log/var-log/nginx/access.log
All of the IPs are 172.17.0.1 😲
So I’d guess I would just need the following?
```plaintext
- replace:
filename: /etc/nginx/conf.d/discourse.conf
from: "types {"
to: |
set_real_ip_from 172.17.0.0/24;
real_ip_header X-Forwarded-For;
real_ip_recursive on;
types {
```
Tho not sure about these two:
```plaintext
real_ip_header X-Forwarded-For;
real_ip_recursive on;
```
> [@pfaffman](#):
>
> you might go into the container and do it by hand to see if it works
Do you mean editing the `/etc/nginx/conf.d/discourse.conf` file directly inside the container then restarting Nginx?
---
### Author:  [@AstonJ](https://meta.discourse.org/u/AstonJ)
#### Post date: [10.Октябрь.2018 00:51:45 UTC](https://meta.discourse.org/t/troubleshooting-a-429-rate-limit/81060/20 "2018-10-10T00:51:45Z")
Yay!! It’s done! IP’s are now showing correctly in the access log 😃
For some reason the spacing messed up in the snippet above, here it is for anyone who might need it in future:
```plaintext
run:
- replace:
filename: /etc/nginx/conf.d/discourse.conf
from: "types {"
to: |
set_real_ip_from 172.17.0.0/24;
real_ip_header X-Forwarded-For;
real_ip_recursive on;
types {
```
Thank you @pfaffman and @sam and @mpalmer for your help 💙
[Следующая страница](https://meta.discourse.org/t/troubleshooting-a-429-rate-limit/81060.md?page=2)