After upgrade, docker cannot communicate with the outside world

Installation
1

I woke up this morning to reports that my discourse server was “down”.

I was able to ssh into the server without a problem, noticed some security updates needed to be applied, applied them with apt full-upgrade, and restarted the server with reboot.

Took a little while to come back up but when it did I checked that the discourse docker container was running. Oddly, no problem.

Tried stopping and restarting with ./launcher app stop and ./launcher app start. Didn’t help.

Tried doing a ./launcher app rebuild and got the following error.

fatal: unable to access 'https://github.com/discourse/pups.git/': Could not resolve host: github.com
d1412324832190f43a2d51b5f10c53d6fa671056f91d0be2178d17a5ba1ab692

I know this isn’t exactly a lot to go on but I’m stumped. Any suggestions would be greatly appreciated.

Thanks

1 Like
2

That suggests some kind of networking error. Maybe docker can’t access the outside world?

3

Any thoughts on how to verify?

4

Try:

docker run --rm -i debian ping -c 1 github.com

You should get e.g.:

○ → docker run --rm -i debian ping -c 1 github.com
PING github.com (140.82.113.3) 56(84) bytes of data.
64 bytes from lb-140-82-113-3-iad.github.com (140.82.113.3): icmp_seq=1 ttl=49 time=44.8 ms

--- github.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 44.797/44.797/44.797/0.000 ms
3 Likes
5

yeah … that’s returning
ping: github.com: Temporary failure in name resolution

Not sure what changed. I have no idea how to fix this.

6

Does that work from outside docker?

1 Like
7

I can reach github from outside of docker.

ping github.com
PING github.com (192.30.255.113) 56(84) bytes of data.
64 bytes from lb-192-30-255-113-sea.github.com (192.30.255.113): icmp_seq=1 ttl=52 time=23.3 ms
64 bytes from lb-192-30-255-113-sea.github.com (192.30.255.113): icmp_seq=2 ttl=52 time=23.3 ms
8

Try:

  • restarting docker
  • docker run --rm -i debian cat /etc/resolv.conf to see what’s there
2 Likes
9

Looks like google

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
# 127.0.0.53 is the systemd-resolved stub resolver.
# run "systemd-resolve --status" to see details about the actual nameservers.

nameserver 8.8.8.8
nameserver 8.8.4.4
10

@supermathie Not sure if this has anything to do with it but in ./launcher logs app I see

[Tue 01 Dec 2020 07:07:13 PM UTC] Please refer to https://curl.haxx.se/libcurl/c/libcurl-errors.html for error code: 6
[Tue 01 Dec 2020 07:07:13 PM UTC] Can not init api.
[Tue 01 Dec 2020 07:07:13 PM UTC] Domains not changed.
[Tue 01 Dec 2020 07:07:13 PM UTC] Skip, Next renewal time is: Sat Dec 26 00:31:17 UTC 2020
[Tue 01 Dec 2020 07:07:13 PM UTC] Add '--force' to force to renew.
[Tue 01 Dec 2020 07:07:13 PM UTC] Installing key to:/shared/ssl/community.acescentral.com.key
[Tue 01 Dec 2020 07:07:13 PM UTC] Installing full chain to:/shared/ssl/community.acescentral.com.cer
[Tue 01 Dec 2020 07:07:13 PM UTC] Run reload cmd: sv reload nginx
fail: nginx: runsv not running
[Tue 01 Dec 2020 07:07:13 PM UTC] Reload error for :
[Tue 01 Dec 2020 07:07:33 PM UTC] Please refer to https://curl.haxx.se/libcurl/c/libcurl-errors.html for error code: 6
[Tue 01 Dec 2020 07:07:34 PM UTC] Can not init api.
[Tue 01 Dec 2020 07:07:34 PM UTC] Domains not changed.
[Tue 01 Dec 2020 07:07:34 PM UTC] Skip, Next renewal time is: Sat Dec 26 00:31:20 UTC 2020
[Tue 01 Dec 2020 07:07:34 PM UTC] Add '--force' to force to renew.
[Tue 01 Dec 2020 07:07:34 PM UTC] Installing key to:/shared/ssl/community.acescentral.com_ecc.key
[Tue 01 Dec 2020 07:07:34 PM UTC] Installing full chain to:/shared/ssl/community.acescentral.com_ecc.cer
[Tue 01 Dec 2020 07:07:34 PM UTC] Run reload cmd: sv reload nginx
fail: nginx: runsv not running
[Tue 01 Dec 2020 07:07:34 PM UTC] Reload error for :
Started runsvdir, PID is 458
chgrp: invalid group: ‘syslog’
ok: run: redis: (pid 473) 0s
ok: run: postgres: (pid 470) 0s
rsyslogd: imklog: cannot open kernel log (/proc/kmsg): Operation not permitted.
rsyslogd: activation of module imklog failed [v8.1901.0 try https://www.rsyslog.com/e/2145 ]
supervisor pid: 471 unicorn pid: 497
11

What’s your docker version and how did you install it? And what Ubuntu are you running?

12 
Client:
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.13.8
 Git commit:        afacb8b7f0
 Built:             Wed Oct 14 19:43:43 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          19.03.8
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.8
  Git commit:       afacb8b7f0
  Built:            Wed Oct 14 16:41:21 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.3.3-0ubuntu2.1
  GitCommit:
 runc:
  Version:          spec: 1.0.1-dev
  GitCommit:
 docker-init:
  Version:          0.18.0
  GitCommit:

Frankly I don’t remember … it’s been here forever. I did update Ubuntu this morning as described above

Ubuntu 20.04.1 LTS

13

After restarting docker, does the ping command work?

14

No, same error.

ping: github.com: Temporary failure in name resolution

15

Another random point …

My discourse server is at community.mydomain.com.
I have a wordpress server at mydomain.com.
Yesterday I setup a rocketchat server at chat.mydomain.com.

I registered chat.mydomain.com with let’s encrypt.
The wordpress and rocketchat servers are working fine.

I’m not sure this would have anything to do with this issue but just wanted to make sure I’m not overlooking something.

16

I might try purging and reinstalling docker; it’s hard to guess exactly what’s going on here and why docker broke.

Before you do anything though, ensure you have a backup of your data (either a Discourse backup, or at least postgres & uploads). Image the VM if you can.

1 Like
17

Completely removed docker and reinstalled.

Problem persists.

This is frustrating.

18

Can your server reach Google DNS, aka ping 8.8.8.8 ?

19

No issues from the server

ping -c 1 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=119 time=2.72 ms

--- 8.8.8.8 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 2.720/2.720/2.720/0.000 ms

No luck from docker

docker run --rm -i debian ping -c 1 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.

--- 8.8.8.8 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
20

Honestly, I think the best approach at this point is to reinstall and copy your data over.

You have a broken system for Mysterious Reasons, and the installation process is rather quick. It’s likely the best use of your time.

4 Likes