Rate Limiting when behind Nginx Proxy

unsupported-install

(Nkansah Rexford) #1

2016/10/01 09:17:03 [error] 56#56: *161 limiting requests, excess: 12.668 by zone "flood", client: 178.62.3.120, server: _, request: "POST /mini-profiler-resources/results HTTP/1.1", host: "yologhana.khophi.co", referrer: "https://yologhana.khophi.co/latest"

I get this error few minutes into starting the discourse app.

I read to understand it is because the request hitting the inner nginx appears to come from ONLY one IP address, which is the outer nginx, therefore the inner nginx thinks it is one person trying to do all those requests.

Of course, disabling the rate limting yml template might solve it, but it ain’t the wisest thing to do. Any workaround?

I saw this, but doesn’t work for me:


(Felix Freiberger) #2

Does your (outer) nginx config contain a line like this?

    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

(Nkansah Rexford) #3

This is my outer nginx config:

server {
    listen 80; listen [::]:80;
    server_name yologhana.khophi.co;

    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl http2;  
    server_name yologhana.khophi.co;

    include /etc/nginx/ssl/globalssl.conf;

    location / {
        proxy_pass http://yologhana.khophi.co:25654/;
        proxy_read_timeout 90;
        proxy_set_header Host $http_host;
        proxy_http_version 1.1;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_redirect http://yologhana.khophi.co:25654/ https://yologhana.khophi.co;
    }
}

(Felix Freiberger) #4

Looks good to me :confused:

Can you check the last seen IP and registration IP of a few users? Are they plausible or local?


(Nkansah Rexford) #5

The last seen IP is me, and no registered user at the moment, except those I invited to join as moderators. The instance keeps spiking out, crippling my entire server, to the extent ssh doesn’t even work.

I disabled the rate limiting bot inner and outer nginx, yet, Discourse keeps refusing connection coming from outer nginx, and the logs in /var/discourse/shared/standalone/log/var-log/nginx/error.log does not give me anything to make sense from except something like this:

GET / HTTP/1.1", upstream: "http://127.0.0.1:3000/", host: "yologhana.khophi.co"
2016/10/01 17:54:18 [error] 50#50: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 178.62.3.120, server: _, request: "POST /message-bus/feaf6454acd54378a7e3086935c5293a/poll?dlp=t HTTP/1.1", upstream: "http://127.0.0.1:3000/message-bus/feaf6454acd54378a7e3086935c5293a/poll?dlp=t", host: "yologhana.khophi.co", referrer: "https://yologhana.khophi.co/"
2016/10/01 18:03:09 [error] 50#50: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 178.62.3.120, server: _, request: "POST /message-bus/feaf6454acd54378a7e3086935c5293a/poll?dlp=t HTTP/1.1", upstream: "http://127.0.0.1:3000/message-bus/feaf6454acd54378a7e3086935c5293a/poll?dlp=t", host: "yologhana.khophi.co", referrer: "https://yologhana.khophi.co/"

I will be honest here. My experience with Discourse has been a horrible one. Maybe because I’m dumb. Maybe it is because the moving parts are an overkill. The stack is just ridiculous, making it hard to get down to what might be the culprit.

Plus because serving Discourse behind a proxy is considered an ‘unsupported install’, I guess not much has been written to cover that aspect.

I’m currently stuck with connection refused with any explicit error anywhere. Should I consider it a dead end?


(Felix Freiberger) #6

Hm, strange.

Oh, you had rate limiting set up in both? Was the first log entry in your initial post from the inner or outer nginx?

Is this IP a local IP, or your actual internet-facing public IP?

That’s bad, is your instance out of memory? Can you post an output of free -h?

Of course, you’re entitled to your opinion. However, …


(Paolo G. Giarrusso) #7

FWIW, when I hit the original problem with rate limiting, I never saw anything like that. I second @fefrei’s suggestion: An out-of-memory can kill random processes, leaving to undefined results. And “connection refused” does suggest a missing server. Please don’t forget you need at least 2GB of RAM for Discourse alone. An OOM would also show up in your (host) kernel logs.

Sadly Ruby on Rails applications don’t get much simpler than this. At least Discourse has an (usually working) Docker container—when I installed Gitlab (also using Ruby on Rails) things were much worse.

But surely an undiagnosed OOM would be annoying as hell.


(Paolo G. Giarrusso) #8

Just found Forcefully install Discourse on 256 or 512 RAM machine. You probably need a bigger swap size. (You don’t need to repartition, you can also use swap files). And quoting:

To sum up, it sounds like random OOMs got in the way of debugging rate limiting. For extra complication, the rate limiting configuration I proposed and you linked to does use some extra RAM—just a couple MB IIRC, but that can’t help. (I admit I have more free RAM so I wasn’t as concerned).


(Jeff Atwood) #9

I believe this user wanted to run on 512mb even when advised it is against the required minimum.


(Nkansah Rexford) #10

I’m not a Discourse rebel. As per the recommendation, I upgraded my droplet to 1 Gig RAM on DO. Plus I have a 4 Gig Swap in place.

Knowing the minimum an app can take helps me. In this case, Discourse has rigged the whole platform to not entertain anything less than 1 Gig RAM. That’s fine.


(Nkansah Rexford) #11

No, it was only on the inner nginx.

My internet-facing public IP

Below my free -m

var/discourse$ free -m
              total        used        free      shared  buff/cache   available
Mem:            992         739          72          33         181          80
Swap:          4095         566        3529

And htop

A two-year old kid today can install WordPress instances and manage even a cluster of them. I am afraid I can’t say so for Discourse. Discourse isn’t for kids, obviously.

Because a lot of people are using it doesn’t mean you’ve made the process seamless. Perhaps they’re forced to because there’s no competition.

Please, don’t point to the first link. It is 2 years old, and the solution is scattered across tens of posts if not hundreds. Try using that tutorial to deploy something today and lemme know if it works.

I see many keep pointing to that outdated document claiming it as if it is a fresh approach. 2014 till now, a lot has happened. And so, yes, hosting behind a proxy is indeed an unsupported install


(Felix Freiberger) #12

The last edit to the first post, which contains all steps, is less than half a year old.

Done! All my three installs work fine behind their nginx proxies, installed using the official install guide followed by this howto.

Did you set up swap before or after installing Discourse? Either way, please rebuild the container:

/var/discourse/launcher rebuild app

(Nkansah Rexford) #13

Yeah, I’ve had swap of 4 Gig since I created my droplet, about 2 years ago.

Glad to hear it worked for you seamlessly. I wish it did for me initially.