After upgrade, docker cannot communicate with the outside world

Looks like google

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
# 127.0.0.53 is the systemd-resolved stub resolver.
# run "systemd-resolve --status" to see details about the actual nameservers.

nameserver 8.8.8.8
nameserver 8.8.4.4

@supermathie Not sure if this has anything to do with it but in ./launcher logs app I see

[Tue 01 Dec 2020 07:07:13 PM UTC] Please refer to https://curl.haxx.se/libcurl/c/libcurl-errors.html for error code: 6
[Tue 01 Dec 2020 07:07:13 PM UTC] Can not init api.
[Tue 01 Dec 2020 07:07:13 PM UTC] Domains not changed.
[Tue 01 Dec 2020 07:07:13 PM UTC] Skip, Next renewal time is: Sat Dec 26 00:31:17 UTC 2020
[Tue 01 Dec 2020 07:07:13 PM UTC] Add '--force' to force to renew.
[Tue 01 Dec 2020 07:07:13 PM UTC] Installing key to:/shared/ssl/community.acescentral.com.key
[Tue 01 Dec 2020 07:07:13 PM UTC] Installing full chain to:/shared/ssl/community.acescentral.com.cer
[Tue 01 Dec 2020 07:07:13 PM UTC] Run reload cmd: sv reload nginx
fail: nginx: runsv not running
[Tue 01 Dec 2020 07:07:13 PM UTC] Reload error for :
[Tue 01 Dec 2020 07:07:33 PM UTC] Please refer to https://curl.haxx.se/libcurl/c/libcurl-errors.html for error code: 6
[Tue 01 Dec 2020 07:07:34 PM UTC] Can not init api.
[Tue 01 Dec 2020 07:07:34 PM UTC] Domains not changed.
[Tue 01 Dec 2020 07:07:34 PM UTC] Skip, Next renewal time is: Sat Dec 26 00:31:20 UTC 2020
[Tue 01 Dec 2020 07:07:34 PM UTC] Add '--force' to force to renew.
[Tue 01 Dec 2020 07:07:34 PM UTC] Installing key to:/shared/ssl/community.acescentral.com_ecc.key
[Tue 01 Dec 2020 07:07:34 PM UTC] Installing full chain to:/shared/ssl/community.acescentral.com_ecc.cer
[Tue 01 Dec 2020 07:07:34 PM UTC] Run reload cmd: sv reload nginx
fail: nginx: runsv not running
[Tue 01 Dec 2020 07:07:34 PM UTC] Reload error for :
Started runsvdir, PID is 458
chgrp: invalid group: ‘syslog’
ok: run: redis: (pid 473) 0s
ok: run: postgres: (pid 470) 0s
rsyslogd: imklog: cannot open kernel log (/proc/kmsg): Operation not permitted.
rsyslogd: activation of module imklog failed [v8.1901.0 try https://www.rsyslog.com/e/2145 ]
supervisor pid: 471 unicorn pid: 497

What’s your docker version and how did you install it? And what Ubuntu are you running?

Client:
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.13.8
 Git commit:        afacb8b7f0
 Built:             Wed Oct 14 19:43:43 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          19.03.8
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.8
  Git commit:       afacb8b7f0
  Built:            Wed Oct 14 16:41:21 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.3.3-0ubuntu2.1
  GitCommit:
 runc:
  Version:          spec: 1.0.1-dev
  GitCommit:
 docker-init:
  Version:          0.18.0
  GitCommit:

Frankly I don’t remember … it’s been here forever. I did update Ubuntu this morning as described above

Ubuntu 20.04.1 LTS

After restarting docker, does the ping command work?

No, same error.

ping: github.com: Temporary failure in name resolution

Another random point …

My discourse server is at community.mydomain.com.
I have a wordpress server at mydomain.com.
Yesterday I setup a rocketchat server at chat.mydomain.com.

I registered chat.mydomain.com with let’s encrypt.
The wordpress and rocketchat servers are working fine.

I’m not sure this would have anything to do with this issue but just wanted to make sure I’m not overlooking something.

I might try purging and reinstalling docker; it’s hard to guess exactly what’s going on here and why docker broke.

Before you do anything though, ensure you have a backup of your data (either a Discourse backup, or at least postgres & uploads). Image the VM if you can.

1 Like

Completely removed docker and reinstalled.

Problem persists.

This is frustrating.

Can your server reach Google DNS, aka ping 8.8.8.8 ?

No issues from the server

ping -c 1 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=119 time=2.72 ms

--- 8.8.8.8 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 2.720/2.720/2.720/0.000 ms

No luck from docker

docker run --rm -i debian ping -c 1 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.

--- 8.8.8.8 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

Honestly, I think the best approach at this point is to reinstall and copy your data over.

You have a broken system for Mysterious Reasons, and the installation process is rather quick. It’s likely the best use of your time.

4 Likes

Reinstall and recover from a backup was the only solution.

No idea what changed in the middle of the night but it’s done now.

Thanks for the support

2 Likes

This issue is not new. It sounds like you solved it with a backup, re-installation, and restore.

For what it’s worth, I worked around the problem as described here: Could not resolve host: github.com for SamSaffron/pups.git

Just as a note, I have this same problem but @rcauvin 's suggestion did not work for me.

See: ./discourse-setup: could not resolve host: github.com for my topic on the same issue

I am experiencing the same issue but then with a clean digital ocean droplet, following this tutorial
. Docker is installed by the discourse installer. I have no idea how to solve this. The above solution also did not work for me.firewall-cmd --zone=public --add-masquerade --permanent

edit: after I applied the solution from @rcauvin followed by firewall-cmd --zone=public --add-masquerade --permanent from this post it works

1 Like

One thing that I’ve noticed recently in Ubuntu 20 on DigitalOcean is that the host machine doesn’t make its domain name servers available to Docker. This is not an issue with the firewall because Ubuntu 20 doesn’t come with the firewall enabled.

To fix, first find the DO name servers with cat /run/systemd/resolve/resolv.conf. You’ll see something like this…

nameserver 1.2.3.4
nameserver 9.8.7.6

Next, edit the daemon.json file with nano /etc/docker/daemon.json to make those nameservers, and optionally, the Google DNS (8.8.8.8) are available to Docker.

{
    "dns": ["<ip1 from above>", "<ip2 from above>", "8.8.8.8"]
}

Then restart Docker and rebuild.

systemctl restart docker
/var/discourse/launcher rebuild app
2 Likes

That file does not exist on my installation. Did you create it from scratch, or did you already have that file in your installation?

The only file I have in that location is key.json.

I created the file from scratch.

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.