One of my forums went down yesterday, first with a blank page then displaying 429’s. Sam thinks it may be possible that my proxy is not configured correctly - anyone know what the best way is to ascertain whether IP addresses are being handed over correctly to Discourse?
I’m pretty sure it is set up correctly (as I have option forwardfor set in HAProxy and users’ IP addresses correctly show up in the admin CP) but I’d like to check Discourse’s end just in case.
That’s the important bit. If the Discourse admin panel is listing users’ IP addresses correctly, then the IP addresses are getting into Discourse.
Thanks Matt.
I wonder what caused the issues yesterday then - reading this it appears that rate limits are applied per user or per IP and so it was odd that the site was inaccessible to everyone (I also tried using various IPs).
The forum has been getting busier lately - we’re now serving around 600K pages a month, but I wouldn’t have thought that would trigger this in itself.
Would the rate limiter only show a 429 to the IPs affected, or everyone?
Any other ideas what it might have been or how to troubleshoot this?
Some of the throttling happens in NGINX though not in Discourse. So you need NGINX to have a set_real_ip_from going.
If nginx’s set_real_ip_from is misconfigured, though, won’t Discourse not be able to get the real IP, either?
It actually will, cause as long as Discourse sees the HTTP X-Forwarded-For header it is fine to set the IP. So you can have a state where IP looks good in Discourse, but NGINX is not “setting real IP” for rate limiting purposes.
Thanks Sam.
So do I basically need to do what’s in this post? Or is there a simpler way?
If following that post, I guess for me that file would read something like the following?
run:
- replace:
filename: "/etc/nginx/conf.d/discourse.conf"
from: /^add_header Strict-Transport-Security 'max-age=31536000';$/
to: |
add_header Strict-Transport-Security 'max-age=31536000';
# Cloudflare
set_real_ip_from my.server.ip;
real_ip_header CF-Connecting-IP;
(It’s a dedicated server running a single IP address.)
If nginx isn’t stripping untrusted XFF, and Discourse is seeing a request from 127.0.0.1 and saying “I trust that IP to give me legit XFF headers”, doesn’t that imply that source IP can be spoofed?
Possibly, we should test, maybe our nginx template is lacking
i actually just got this error. working through it now…
we’re back up. that seemed totally random.
Coming back to this Topic as the other one is slightly different.
Do we still need to be doing something like in the post above?
Looking through the /etc/nginx/conf.d/discourse.conf file there seems to be no mention of set_real_ip_from (searching the Discourse repo for the same yields no results either). From the discourse.conf file these seem most relevant:
(mentions of IP)
# This big block is needed so we can selectively enable
# acceleration for backups and avatars
# see note about repetition above
location ~ ^/(letter_avatar/|user_avatar|highlight-js|stylesheets|favicon/proxied|service-worker) {
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Request-Start "t=${msec}";
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $thescheme;
# we need buffering off for message bus
location /message-bus/ {
proxy_set_header X-Request-Start "t=${msec}";
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $thescheme;
proxy_http_version 1.1;
proxy_buffering off;
proxy_pass http://discourse;
break;
}
# auth_basic on;
# auth_basic_user_file /etc/nginx/htpasswd;
location ~* (assets|plugins|uploads)/.*\.(eot|ttf|woff|woff2|ico)$ {
expires 1y;
add_header Cache-Control public,immutable;
add_header Access-Control-Allow-Origin *;
}
location = /srv/status {
access_log off;
log_not_found off;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Request-Start "t=${msec}";
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $thescheme;
proxy_pass http://discourse;
break;
}
Which appear to be setting the correct remote address 
For completeness I received a couple of 429s after a peak in traffic a couple of days ago, and looking at my app.yml I found that the set_real_ip_from my.server.ip; and real_ip_header CF-Connecting-IP; lines were commented out… but it doesn’t look like they do anything anyway? Can you remember what the outcome of your investigations were after?
Currently I am recommending adding those lines to the app.yml in the following guide: How to set up Discourse on a server with existing Apache sites (and so may be giving bad advice to others?)
Hi Sam, what’s the best way tot set (or check) this? It seems the other things I’ve tried in this Topic may not be working - we had an announcement posted on Twitter today and people are experiencing 429s
I would not even class this as a huge amount of traffic (only had 25K views reported in the ACP up till now).
Any idea what’s going on?
Yeah very likely NGINX thinks everyone has the same IP, so your rate limiting is kind of nonsense, to quickly mitigate I would recommend removing the rate limiting template and rebuilding. But longer term you are going to need to learn a bit of NGINX and figure out how to teach NGINX that everyone has different IPs using the set real ip extension
Thanks Sam.
Should this work?
run:
- replace:
filename: "/etc/nginx/conf.d/discourse.conf"
from: /^add_header Strict-Transport-Security 'max-age=31536000';$/
to: |
add_header Strict-Transport-Security 'max-age=31536000';
# IP
set_real_ip_from my.server.ip;
Only thing is I already have that, and can’t actually see anything called set_real_ip_from in /etc/nginx/conf.d/discourse.conf (as detailed in this post)
Any pointers would be greatly appreciated 
You can look at the logs (e.g., shared/standalone/log/var-log/nginx/access.log and see what IP numbers are getting through. You should be able to, for example, see your own IP when you load the site.
Do you have something external to the Discourse container doing a reverse proxy?
Are you using some external that’s doing a reverse proxy (e.g., CloudFlare)?
There’s HAProxy on the front, set up as per this guide:
When looking in the admin control panel, all user’s IPs are showing correctly (which I believe is because option forwardfor is set in HAProxy).
Other than that there is no cloudfare - however the datcentre does offer DDOS protection as standard (but don’t think that would make any difference) : /
Looking at other topics here, looks like I may need to do this?
- replace:
filename: /etc/nginx/conf.d/discourse.conf
from: "types {"
to: |
set_real_ip_from 10.0.0.0/24;
set_real_ip_from 172.17.0.0/24;
real_ip_header X-Forwarded-For;
real_ip_recursive on;
types {
From this post which was linked to from this one by Sam (and you (Jay) commented on it - mentioning HAProxy).
Look at the nginx logs and see what IP addresses are there. Are they all the IP address of the magic DDOS protector that the data center provides?
That snippet might be a solution. Before fooling with doing it that way, you might go into the container and do it by hand to see if it works. (Or, if that doesn’t make sense to you, just blindly copy it and see if it works 
)
All of the IPs are 172.17.0.1 
So I’d guess I would just need the following?
- replace:
filename: /etc/nginx/conf.d/discourse.conf
from: "types {"
to: |
set_real_ip_from 172.17.0.0/24;
real_ip_header X-Forwarded-For;
real_ip_recursive on;
types {
Tho not sure about these two:
real_ip_header X-Forwarded-For;
real_ip_recursive on;
Do you mean editing the /etc/nginx/conf.d/discourse.conf file directly inside the container then restarting Nginx?
Yay!! It’s done! IP’s are now showing correctly in the access log 
For some reason the spacing messed up in the snippet above, here it is for anyone who might need it in future:
run:
- replace:
filename: /etc/nginx/conf.d/discourse.conf
from: "types {"
to: |
set_real_ip_from 172.17.0.0/24;
real_ip_header X-Forwarded-For;
real_ip_recursive on;
types {
