IPv6 Address isn't forwarded

docker

#1

I’m seeing some weird behavior on my instance for IPv6 only traffic, for some reason the X-Forwarded-For header isn’t being set right.

If I do a tcpdump inside the container and do a curl (ensuring that it resolves my forum to the IPv6 address) I can see its just being set to the docker interface:

Host: talk.configdroid.com
X-Real-IP: 172.17.42.1
X-Forwarded-For: 172.17.42.1
X-Forwarded-Proto: https
Connection: close
User-Agent: curl/7.35.0
Accept: */*

Whats weird is that if I go via CloudFlare (cdn-talk.configdroid.com) and resolve via IPv6 everything goes fine and the IPv6 address resolves. I’ve tried this via various VPS’s as I thought it might have been some weird routing but it isn’t that.

Anyone got any ideas?


(Matt Palmer) #2

I’m inclined to think the problem is going to be related to “trusted forwarders” not being set correctly for IPv6 addresses, or something of that nature. Without knowing all of the details of your setup, though, it’s hard to be sure. Can you give a complete request flow diagram, and pcaps on either side of each point? That’ll at least help to narrow down where everything’s going horribly, horribly wrong.


#3

Main Discourse instance:

Both of the above IP’s are on the same interface, eth0.

CDN on CloudFlare at cdn-talk.configdroid.com which just goes to the above two IP’s.

Other than that its a stock standard docker Discourse install.

tcpdump’ing on the Discourse end on port 3000 returns this when doing a cURL to https://talk.configdroid.com:

GET / HTTP/1.0
Host: talk.configdroid.com
X-Real-IP: 45.32.247.194
X-Forwarded-For: 45.32.247.194
X-Forwarded-Proto: https
Connection: close
User-Agent: curl/7.35.0
Accept: */*

And the response:

Date: Fri, 09 Oct 2015 04:56:58 GMT
Status: 200 OK
Connection: close
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
X-Discourse-Route: list/latest
Content-Type: text/html; charset=utf-8
Content-Length: 9949
Set-Cookie: _forum_session=<blah>; path=/; HttpOnly
Set-Cookie: __profilin=<blah>; path=/
X-Request-Id: da75304f-edcb-4bf1-91be-40fa8e3d8ed3
X-Runtime: 0.094071

And our IPv6 results done with this cURL curl -g -6 -vso/dev/null -H "Host: talk.configdroid.com" https://[2400:6180:0:d0::84:2002]/ --insecure from the same host above:

GET / HTTP/1.0
Host: talk.configdroid.com
X-Real-IP: 172.17.42.1
X-Forwarded-For: 172.17.42.1
X-Forwarded-Proto: https
Connection: close
User-Agent: curl/7.35.0
Accept: */*

And the response:

Date: Fri, 09 Oct 2015 05:00:05 GMT
Status: 200 OK
Connection: close
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
X-Discourse-Route: list/latest
Content-Type: text/html; charset=utf-8
Content-Length: 9949
X-Discourse-Cached: true
X-Request-Id: 3ea85842-5b04-4110-bbec-780c5b4ac1b7
X-Runtime: 0.004462
Set-Cookie: __profilin=<blah>; path=/

A cURL via CloudFlare sets the IPv6’s fine though, this is likely because I’m using the cloudflare.template.yml template which sets the real IP’s via CloudFlare headers.

So I guess you’re right, it is something to do with the “trusted forwarders” but its weird its only happening on IPv6?


(Matt Palmer) #4

How far into the network is the IPv6 traffic going, before it becomes IPv4? Docker isn’t particularly IPv6-transparent – your IPv6 traffic is almost certainly getting turned into IPv4 traffic somewhere, and thus losing important IPv6 source address information. I strongly suspect that the reason why it appears to be working through CloudFlare is that they set the X-Forwarded-For header, and your system is trusting that, while docker-proxy isn’t.

Verifying this hypothesis (that it isn’t IPv6 traffic all the way in) is the purpose of the pcaps – we don’t just want to see the L7 traffic, we need to see all the way down to L3, at each point in the network where transformation can occur.


(Kane York) #5

Do you have an extra nginx in front of the Docker container? It needs to get the Cloudflare X-Forwarded-For rules, too. I’m not sure where else the 172.17.42.1 would be coming from…

If not, try checking the Cloudflare config for IPV6-related settings?

If so, put this inside the server block:

set_real_ip_from 103.21.244.0/22; set_real_ip_from 103.22.200.0/22; set_real_ip_from 103.31.4.0/22; set_real_ip_from 104.16.0.0/12; set_real_ip_from 108.162.192.0/18; set_real_ip_from 141.101.64.0/18; set_real_ip_from 162.158.0.0/15; set_real_ip_from 172.64.0.0/13; set_real_ip_from 173.245.48.0/20; set_real_ip_from 188.114.96.0/20; set_real_ip_from 190.93.240.0/20; set_real_ip_from 197.234.240.0/22; set_real_ip_from 198.41.128.0/17; set_real_ip_from 199.27.128.0/21; set_real_ip_from 2400:cb00::/32; set_real_ip_from 2405:8100::/32; set_real_ip_from 2405:b500::/32; set_real_ip_from 2606:4700::/32; set_real_ip_from 2803:f800::/32;

NOTE to anyone coming by later: do not copy the above listing, instead generate it yourself:

#!/bin/bash -e
# Download list of CloudFlare ips
wget https://www.cloudflare.com/ips-v4 -O - > /tmp/cloudflare-ips
echo >> /tmp/cloudflare-ips
wget https://www.cloudflare.com/ips-v6 -O - >> /tmp/cloudflare-ips
# Make into nginx commands and escape for inclusion into sed append command
CONTENTS=$(</tmp/cloudflare-ips sed 's/^/set_real_ip_from /' | sed 's/$/;/' | tr '\n' '\\' | sed 's/\\/\\n/g')

echo CloudFlare IPs:
echo $(echo | sed "/^/a $CONTENTS")
# Insert into discourse.conf
sed -i "/sendfile on;/a $CONTENTS\nreal_ip_header CF-Connecting-IP;" /etc/nginx/conf.d/discourse.conf
# Clean up
rm /tmp/cloudflare-ips

#6

I’ve already got the CloudFlare rules in place, that’s why going via them works :smile:.

172.17.42.1 is the docker interface on my box, so for some reason the nginx that’s running discourse isn’t trusting the header IP’s for IPv6, IPv4 is fine.

I don’t have anything in front of the discourse instance. I’m still working on getting the packet captures done.


#7

So I’ve done some packet dumps and it turns out IPv6 curls make it to the host, but they don’t make it into the container. They turn into IPv4 addresses at this stage. I’m not really sure where to go from here.


(Matt Palmer) #8

I’d be inclined to swear heartily at Docker for not properly supporting IPv6. There really isn’t any excuse in 2015 for having no coherent IPv6 support story.

The only thing that comes to mind as a “quick fix” is to put an nginx instance in the host, have it receive the requests and add an X-Forwarded-For header if one doesn’t already exist (as added by CF) and then pass the request off to the container. That’ll capture the original (IPv6) source address for processing by the nginx in the container. You’ll have to adjust the set_real_ip_from parameter in the container to be the IP of the host, and use set_real_ip_from with the CF IPs in the host’s config.

Ugly, but short of fixing Docker’s IPv6 support, it’s probably the only coherent solution available.


#9

Actually, looking at the docs here it seems it might actually be possible by just giving the docker daemon some more args. By default docker doesn’t give containers IPv6 routeable addresses.

I think it’s going to be easier if I switch to using the socket template and putting a HAProxy instance in front of the discourse nginx or another nginx instance with some decent keep alive time outs. I’ll have a go today and get back to you.


(Ricardonacif) #10

Any solution here? I’m facing the same issue with my currently setup using nginx proxy and my discourse nginx is returning:

2016/02/11 21:15:02 [error] 50#50: *246261 connect() to [2400:cb00:2048:1::6819:f117]:443 failed (101: Network is unreachable) while connecting to upstream, client: 52.35.11.252, server: _, request: "GET /letter_avatar_proxy/v2/letter/a/ad7895/64.png HTTP/1.0", upstream: "https://[2400:cb00:2048:1::6819:f117]:443/v2/letter/a/ad7895/64.png", host: "community.vangoart.co", referrer: "https://www.vangoart.co/community/t/all-work-not-showing/154"

(Matt Palmer) #11

This doesn’t seem like the same problem as was discussed previously. That just looks like your nginx is trying to use IPv6 to talk to the backend, but can’t because IPv6 networking isn’t configured correctly. Regular network troubleshooting methods (pcaps, examining routing tables / firewalls, etc) should solve the problem. This isn’t a Discourse-specific problem.