After upgrading Ubuntu, a user experienced issues with Docker containers unable to resolve external hostnames like github.com, resulting in failed pups builds. The user could ping external IPs (e.g., 8.8.8.8) from the host but not from within Docker, indicating a DNS resolution problem. Despite restarting Docker, reinstalling it, and verifying DNS settings (/etc/resolv.conf), the issue persisted. The community suggested checking Docker’s DNS configuration and noted that Ubuntu 20.04 on DigitalOcean may not pass the host’s DNS servers to Docker by default. A solution was proposed: creating /etc/docker/daemon.json with the host’s DNS servers (from /run/systemd/resolve/resolv.conf) and Google DNS (8.8.8.8), then restarting Docker and rebuilding the Discourse app. The user eventually resolved the issue by reinstalling and restoring from a backup, though others confirmed the DNS fix worked for them as well. agf1997, pfaffman, supermathie, yanokwa, rcauvin, niels-anders
I woke up this morning to reports that my discourse server was “down”.
I was able to ssh into the server without a problem, noticed some security updates needed to be applied, applied them with apt full-upgrade, and restarted the server with reboot.
Took a little while to come back up but when it did I checked that the discourse docker container was running. Oddly, no problem.
Tried stopping and restarting with ./launcher app stop and ./launcher app start. Didn’t help.
Tried doing a ./launcher app rebuild and got the following error.
fatal: unable to access 'https://github.com/discourse/pups.git/': Could not resolve host: github.com
d1412324832190f43a2d51b5f10c53d6fa671056f91d0be2178d17a5ba1ab692
I know this isn’t exactly a lot to go on but I’m stumped. Any suggestions would be greatly appreciated.
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
# 127.0.0.53 is the systemd-resolved stub resolver.
# run "systemd-resolve --status" to see details about the actual nameservers.
nameserver 8.8.8.8
nameserver 8.8.4.4
@supermathie Not sure if this has anything to do with it but in ./launcher logs app I see
[Tue 01 Dec 2020 07:07:13 PM UTC] Please refer to https://curl.haxx.se/libcurl/c/libcurl-errors.html for error code: 6
[Tue 01 Dec 2020 07:07:13 PM UTC] Can not init api.
[Tue 01 Dec 2020 07:07:13 PM UTC] Domains not changed.
[Tue 01 Dec 2020 07:07:13 PM UTC] Skip, Next renewal time is: Sat Dec 26 00:31:17 UTC 2020
[Tue 01 Dec 2020 07:07:13 PM UTC] Add '--force' to force to renew.
[Tue 01 Dec 2020 07:07:13 PM UTC] Installing key to:/shared/ssl/community.acescentral.com.key
[Tue 01 Dec 2020 07:07:13 PM UTC] Installing full chain to:/shared/ssl/community.acescentral.com.cer
[Tue 01 Dec 2020 07:07:13 PM UTC] Run reload cmd: sv reload nginx
fail: nginx: runsv not running
[Tue 01 Dec 2020 07:07:13 PM UTC] Reload error for :
[Tue 01 Dec 2020 07:07:33 PM UTC] Please refer to https://curl.haxx.se/libcurl/c/libcurl-errors.html for error code: 6
[Tue 01 Dec 2020 07:07:34 PM UTC] Can not init api.
[Tue 01 Dec 2020 07:07:34 PM UTC] Domains not changed.
[Tue 01 Dec 2020 07:07:34 PM UTC] Skip, Next renewal time is: Sat Dec 26 00:31:20 UTC 2020
[Tue 01 Dec 2020 07:07:34 PM UTC] Add '--force' to force to renew.
[Tue 01 Dec 2020 07:07:34 PM UTC] Installing key to:/shared/ssl/community.acescentral.com_ecc.key
[Tue 01 Dec 2020 07:07:34 PM UTC] Installing full chain to:/shared/ssl/community.acescentral.com_ecc.cer
[Tue 01 Dec 2020 07:07:34 PM UTC] Run reload cmd: sv reload nginx
fail: nginx: runsv not running
[Tue 01 Dec 2020 07:07:34 PM UTC] Reload error for :
Started runsvdir, PID is 458
chgrp: invalid group: ‘syslog’
ok: run: redis: (pid 473) 0s
ok: run: postgres: (pid 470) 0s
rsyslogd: imklog: cannot open kernel log (/proc/kmsg): Operation not permitted.
rsyslogd: activation of module imklog failed [v8.1901.0 try https://www.rsyslog.com/e/2145 ]
supervisor pid: 471 unicorn pid: 497
I might try purging and reinstalling docker; it’s hard to guess exactly what’s going on here and why docker broke.
Before you do anything though, ensure you have a backup of your data (either a Discourse backup, or at least postgres & uploads). Image the VM if you can.