Could not resolve host: github.com for SamSaffron/pups.git

Support
1

Hi all -

We’re enjoying discourse thus far. I’m having an issue with running a rebuild of the docker container. The container won’t rebuild because for some reason it can’t resolve github.com. I know the host system (debian jessie) can resolve at that time via dig && ping. However if I issue a systemctl restart docker the issue goes away.

I turned to the goog or alphabet machine which suggested starting the docker daemon using the goog’s dns which I’ve done and confirmed that it seems to have taken via a systemctl status docker.

I’ve been able to reproduce on a debian droplet and a virtual box vm.

Also I’ve been able to successfully pull a Ubuntu image, and run the container.

I’m not sure if there’s something in launcher or any other suggestions? Working off of git master, and is up to date.

shortlist:

  • Debian Jessie Kernel 3.16
  • Docker version 1.6.2, build 7c8fca2

launch rebuild app

root@alpha:/var/discourse# ./launcher rebuild app
Ensuring discourse docker is up to date
Fetching origin
Discourse Docker is up-to-date
Stopping old container
+ /usr/bin/docker stop -t 10 app
app
cd /pups && git pull && /pups/bin/pups --stdin
fatal: unable to access 'https://github.com/SamSaffron/pups.git/': Could not resolve host: github.com
210c421dbe03e0339d110f1eec9eb3c985b0a0d4e32709d569c9cd1f235a7964
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one

systemctl status - shows --dns

root@alpha:/var/discourse# systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled)
   Active: active (running) since Thu 2015-08-13 03:42:17 UTC; 26min ago
     Docs: http://docs.docker.com
 Main PID: 22232 (docker)
   CGroup: /system.slice/docker.service
           └─22232 /usr/bin/docker -d -H fd:// --dns 8.8.8.8 --dns 8.8.4.4

Aug 13 04:03:40 alpha docker[22232]: time="2015-08-13T04:03:40Z" level=info msg="+job wait(7d602b0addff82e7626b659c5e30131a11ab7f35034198983a19e2b8861111b5)"
Aug 13 04:03:40 alpha docker[22232]: time="2015-08-13T04:03:40Z" level=info msg="-job wait(7d602b0addff82e7626b659c5e30131a11ab7f35034198983a19e2b88... = OK (0)"
Aug 13 04:03:40 alpha docker[22232]: time="2015-08-13T04:03:40Z" level=info msg="DELETE /v1.18/containers/7d602b0addff82e7626b659c5e30131a11ab7f3503...8861111b5"
Aug 13 04:03:40 alpha docker[22232]: time="2015-08-13T04:03:40Z" level=info msg="+job rm(7d602b0addff82e7626b659c5e30131a11ab7f35034198983a19e2b8861111b5)"
Aug 13 04:03:40 alpha docker[22232]: time="2015-08-13T04:03:40Z" level=info msg="+job log(destroy, 7d602b0addff82e7626b659c5e30131a11ab7f35034198983...e:1.0.12)"
Aug 13 04:03:40 alpha docker[22232]: time="2015-08-13T04:03:40Z" level=info msg="-job log(destroy, 7d602b0addff82e7626b659c5e30131a11ab7f35034198983... = OK (0)"
Aug 13 04:03:40 alpha docker[22232]: time="2015-08-13T04:03:40Z" level=info msg="-job rm(7d602b0addff82e7626b659c5e30131a11ab7f35034198983a19e2b8861... = OK (0)"
Aug 13 04:04:15 alpha docker[22232]: time="2015-08-13T04:04:15Z" level=info msg="GET /v1.18/containers/json"
Aug 13 04:04:15 alpha docker[22232]: time="2015-08-13T04:04:15Z" level=info msg="+job containers()"
Aug 13 04:04:15 alpha docker[22232]: time="2015-08-13T04:04:15Z" level=info msg="-job containers() = OK (0)"
Hint: Some lines were ellipsized, use -l to show in full.

docker info

root@alpha:/var/discourse# docker info
Containers: 3
Images: 18
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 24
 Dirperm1 Supported: true
Execution Driver: native-0.2
Kernel Version: 3.16.0-4-amd64
Operating System: Debian GNU/Linux 8 (jessie)
CPUs: 1
Total Memory: 2.879 GiB
Name: alpha
ID: WF4H:KABY:JPTX:RW2U:FWCT:PNKZ:DR2P:2ZFE:ZLUD:AB4R:D3CH:EXCA
WARNING: No memory limit support
WARNING: No swap limit support

edit: formatting fix code block

1 Like
2

This is very likely the solution

3 Likes
3

Hi @sam thanks for taking the time to review.
I have that option uncommented in /etc/defaults/docker, and I’ve restarted the docker service many times.

To clarify, after restarting the docker service the laucher rebuild will work once, after that I have to restart the service.

You can see this via my systemd status.

CGroup: /system.slice/docker.service
           └─22232 /usr/bin/docker -d -H fd:// --dns 8.8.8.8 --dns 8.8.4.4
1 Like
4

Same issue, I went to add the cloudflare template and when rebuilding it could not resolve github.com.

I have already tried the solution of using googles DNS servers in dockers configuration file with no success.

doing ping github.com works but again, it cant seem to resolve when rebuilding.

Here is the error:

Here is me pinging github.com
https://i.imgur.com/oUjgq8p.png

Interestingly though, when I ping6 github.com I of course do not recieve a response because github does not have AAAA records i believe.
https://i.imgur.com/Qu9OJg9.png

Any idea how to fix this?

1 Like
5

This isn’t a Discourse problem, it’s a Docker problem. You’ll probably get more knowledgeable help at the Docker support forums (which, helpfully, also run Discourse).

5 Likes
6

I figured out the issue and solution to this problem at least for me.

The issue was that Docker uses 8.8.8.8 and 8.8.4.4 as its DNS and I had an OVH network based firewall blocking ALL IPv4 traffic that did not match port 80, 443, and a few other specific rules. I had to authorize IPv4 traffic for 8.8.8.8 and 8.8.4.4

7

I’m experiencing a similar issue on the latest discourse git. This started happening after performing a system update and/or discourse update via the web UI. Now when I try to rebuild the app, I get:

[root@forum /var/discourse]# ./launcher rebuild app
Ensuring launcher is up to date
Fetching origin
Launcher is up-to-date
Stopping old container
+ /usr/bin/docker stop -t 60 app
app
cd /pups && git pull && /pups/bin/pups --stdin
fatal: unable to access 'https://github.com/discourse/pups.git/': Could not resolve host: github.com
8f082ebcb977f9efafbdbff15ab69e8d06c0a7e2cb99410f85e1f90b03ae733b
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.

I can ping github.com and 8.8.8.8, and I’ve tried editing /etc/default/docker to toggle the DNS option: DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4"

Where should I continue investigating this?

This is Ubuntu 18.04 with latest updates applied.

4 Likes
8

I am getting the same error as gekkonen after having attempted to upgrade to the latest Discourse version.

It appears Docker can no longer access the Internet at all from the default bridge network:

docker run -it --net=bridge --rm busybox ping -c 1 github.com
ping: bad address 'github.com'

If I force it to use the host network, it can access the Internet:

docker run -it --net=host --rm busybox ping -c 1 github.com
PING github.com (140.82.112.4): 56 data bytes
64 bytes from 140.82.112.4: seq=0 ttl=53 time=6.902 ms

--- github.com ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 6.902/6.902/6.902 ms

I am also running Ubuntu 18.04.4 LTS, having upgraded all packages and installed all security upgrades just prior to following the Discourse upgrade sequence.

3 Likes
9

@gekkonen I got mine running again. As a workaround, to force the Internet to work from Docker, you can configure the docker0 network to use “promiscuous” mode:

[root@forum /var/discourse]# ifconfig docker0 promisc

You should then be able to use the launcher to rebuild the app, and the Discourse forum should work fine. The caveat is that you probably will have to issue the command again after each reboot.

Let me know if it works for you.

I still don’t know the root cause of the problem.

3 Likes
10

Yes, that seemed to get past the issue with the DNS for now. Thank you for the suggestion.

1 Like
11

On CentOS8 this error is due to the RHEL7 docker rpms (there are no RHEL8 ones yet) not understanding the new nftables based firewall. One has to setup masquerading by hand for the docker0 interface

https://serverfault.com/questions/987686/no-network-connectivity-to-from-docker-ce-container-on-centos-8

1 Like
12

I’d like to confirm this solution works. I am on CentOS 8, with Hetzner.

To be specific I ran these commands in order like listed below. (source) - thanks to @paulraines68

# Masquerading allows for docker ingress and egress (this is the juicy bit)
firewall-cmd --zone=public --add-masquerade --permanent

# Specifically allow incoming traffic on port 80/443 (nothing new here)
firewall-cmd --zone=public --add-port=80/tcp
firewall-cmd --zone=public --add-port=443/tcp

# Reload firewall to apply permanent rules
firewall-cmd --reload