All of my internal users show as coming from!

(Michael Brown) #1

After setting up a Discourse instance, all the users from my internal network show as coming from instead of their real IP. What gives eh?

Thanks to @sam I learned that the Rails middleware is treating all internal IP addresses as proxies and stripping them from the X-Forwarded-For list. If nothing is left, then REMOTE_ADDR is used which is usually (nginx).

This behaviour can be overridden so that only is stripped by adding the following to config/environments/production.rb:

class ActionDispatch::RemoteIp
  self.send :remove_const, "TRUSTED_PROXIES"

(Sam Saffron) #3

@supermathie also, this fix is a hack, can you open this up with Rails so a proper fix can be made to ActionDispatch (an accessor to change it for example)

(Michael Brown) #5

looks like it should fix this. I’ve set:

config.action_dispatch.trusted_proxies = /^(127\.0\.0\.1|::1)$/

in my config/environments/production.rb but it didn’t make a difference…

(Nathan Broadbent) #6

In Rails 4, this will work.

However, in Rails 3.2, it looks like config.action_dispatch.trusted_proxies is only added to TRUSTED_PROXIES with Regexp.union: rails/remote_ip.rb at v3.2.12 · rails/rails · GitHub

So it looks like @supermathie’s TRUSTED_PROXIES override is the only way to do this for Rails 3.2

(Andrew Meyer) #7

In Rails 4, this will work.

So if this is fixed in Rails 4, and Discourse is now running on Rails 4 (see Gemfile.lock) should the link to this topic in the Ubuntu install docs be removed?

(Michael Brown) #8

We still need a nice way to override this setting for two reasons:

  • IPv6
  • front end proxy addr is IPv4 globally routable

Not everybody uses the hack that is PNAT.

Config Nginx to receive real IP when using Cloudflare for noobs
(Binoj David) #10

Still getting for internal users.

(Chris Saenz) #11

Same here. Did this ever get fixed?

(Jack Coulter) #12

Same here as well - all users are reported as

Running tcpdump within the container and inspecting the packets between nginx and Discourse itself, the X-Forwarded-For and X-Real-IP headers both have the IP address of the docker host, and not the real source IP, suggesting that this is the value $remote_addr returns in the nginx configuration file.While this is of course, incorrect, this raises further questions as to why Discourse shows rather than the value in X-Forwarded-For / X-Real-IP.

What could cause this? The configuration of this instance has not been customised aside from the basics, and enabling SSL support.

(Kane York) #13

Is there another proxy in front of the container, as in this topic?

The topic received an update, which adds the directives to correctly pass the IP through.

(Jack Coulter) #14

In our case, there’s no other web servers on the machine, nor is there any reverse proxy in front of it. Port 443 of nginx within Discourse’s Docker container is directly exposed.

(Jack Coulter) #15

This certainly seems to be a NAT issue. Looking at the iptables rules set up on the host by Docker, it seems the DNAT rules are never hit, and the userspace relay, docker-proxy is handling all the packet forwarding to the container (hence the source address on all packets received in the container being the host IP, which ends up as the value for X-Forwarded-For, which I imagine Discourse is ignoring because it’s a private IP)

From iptables -t nat -nvL DOCKER, showing the DNAT rules are never hit:

Chain DOCKER (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DNAT       tcp  --  !docker0 *              tcp dpt:443 to:
    0     0 DNAT       tcp  --  !docker0 *              tcp dpt:80 to:

(Sam Saffron) #16

Something here is odd, on multiple Docker digital ocean setups I am seeing IPs correctly reported (Just checked Jeff’s blog which is stock standard)

Something about your setup is off, Rails respects X-Forwarded-For if it is reasonable and deals with chains even.

(Jack Coulter) #17

Would I be correct to assume that Rails would ignore an X-Forwarded-For header when it’s value consists only of a single private IP?

This is what I see, running tcpdump inside the container, inspecting traffic to port 3000:

X-Forwarded-Proto: https
Connection: close
Content-Length: 159
accept: application/json, text/javascript, */*; q=0.01
accept-encoding: gzip, deflate
accept-language: en-GB,en-US;q=0.8,en;q=0.6
content-type: application/x-www-form-urlencoded; charset=UTF-8

This is what i see doing tcpdump on eth0 inside the container, which seems to indicate that the original source address is being lost somewhere:

09:26:46.933458 IP > Flags [.], ack 10524, win 2715, options [nop,nop,TS val 69777543 ecr 69777533], length 0
09:26:47.118145 IP > Flags [P.], seq 19974:21368, ack 10524, win 2715, options [nop,nop,TS val 69777589 ecr 69777533], length 1394
09:26:47.122555 IP > Flags [P.], seq 21368:21721, ack 10524, win 2715, options [nop,nop,TS val 69777590 ecr 69777533], length 353

I’m confused as to just how packets are making it to the container at all, considering the DNAT rules are never hit (the DOCKER chain is only referenced from the OUTPUT chain). I would guess, that it’s all going through docker-proxy, which (again, a guess) I imagine is why the source address is the docker host.

We haven’t customised anything aside from enabling SSL. The host has UFW, but only a few, very simple rules:

Status: active

To                         Action      From
--                         ------      ----
22                         ALLOW       Anywhere
25                         ALLOW       Anywhere 995/tcp on docker0 ALLOW       Anywhere
80                         ALLOW       Anywhere
443                        ALLOW       Anywhere
22 (v6)                    ALLOW       Anywhere (v6)
25 (v6)                    ALLOW       Anywhere (v6)
80 (v6)                    ALLOW       Anywhere (v6)
443 (v6)                   ALLOW       Anywhere (v6)

I’m a bit lost as to what’s happening here, as we also have another instance which seems to have the same firewall rules, yet it is working correctly (and the DNAT rules on that instance show many packets hitting them, as expected).

(Sam Saffron) #18

Can you expand a bit on your setup? Host you are using? Docker version? Host os version?

(Jack Coulter) #19

Sure, the host is Digital Ocean, the OS is Ubuntu 14.04.2 LTS, and Docker is version 1.5.0, build a8a31ef.

(Jens Maier) #20

Do you start services in parallel during boot or in a fixed order? Are the two hosts starting services in the same order? If docker starts before the firewall sets its rules, the firewall could be replacing some or all of docker’s rules… or add rules in different order.

(Jack Coulter) #21

I tried temporarily disabling UFW, unfortunately it made no difference - The source address of incoming HTTPS connections within the container is still incorrect.

(Jens Maier) #22

Temporarily disabling UFW would not restore rules it’s already flushed from iptables. You could try to disable the firewall and restart dockerd – or just reboot without the firewall script…

(Jack Coulter) #23

Thanks! Stopping UFW, restarting dockerd, and starting UFW again fixed it. Thanks to everyone here for all the help :slight_smile: