I did a system upgrade from 14.04 to 16.04 and it seems that after that my system is broken. Trying to start discourse with sudo ./launcher start app gives the following:
Device “docker0” does not exist.
Cannot connect to the docker daemon - verify it is running and you have access
Reinstall Docker using apt-get
root@DOMAIN:/var/discourse# sudo apt-get install docker
Reading package lists... Done
Building dependency tree
Reading state information... Done
docker is already the newest version (1.5-1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.Is this a problem with Upstard vs. systemd?
With start docker I’m getting simply:
start: Unable to connect to Upstart: Failed to connect to socket /com/ubuntu/upstart: Connection refused
How the **** did I manage to get my system this broken… Ngh.
Did apt-get remove, apt-get install for docker and docker-engine. Getting some weird message about error initializing graphdriver: driver not supported.
Should I just rm -rf /var/lib/docker?
That is a wicked out-of-date version of Docker. We recommend always keeping Docker up-to-date, and they’re up to 1.12.5 now.
Yeeeeeah… newer versions of Docker suck about maintaining compatibility. You were probably using aufs before, so try adding -s aufs to your DOCKER_OPTS (in /etc/default/docker), or wherever they’ve moved that to in the Brave New World of systemd. Making sure you’ve installed the extras kernel package for the aufs.ko kernel module will help, too.
Alternately, yeah, you could nuke /var/lib/docker (with docker-engine uninstalled) and then install docker-engine again, that’d work. You’ll need to do a full rebuild afterwards to get Discourse going again, but as long as you don’t touch /var/discourse, your database, etc should remain intact.
Thanks a lot for the help! So how do I get a newer docker if the apt-get doesn’t give it to me?
Edit: Found it. Just a sec.
Edit2: Updated docker. Nuked /var/lib/docker. New error in service docker status:
level=error msg="[graphdriver] prior storage driver \"devicemapper\" failed: devicemapper: Error running deviceCreate (CreatePool) dm_task_run failed"
level=fatal msg="Error starting daemon: error initializin graphdriver: devicemapper: Error running deviceCreate (CreatePool) dm_task_run failed"
Ngh.
Oh man, devicemapper… why is it trying to use that as default? Definitely time to add -s aufs to the docker daemon options.
I’m quite lost. How do I do that?
Am I supposed to create a file into /etc/systemd/system/docker.service.d/docker.conf that has the line:
ExecStart=/usr/bin/dockerd -H fd:// -s aufs
Or how?
Well, I edit /etc/default/docker and add it to the DOCKER_OPTS line, but I don’t know if systemd respects that file, or if there’s some other magical way of making it happen in systemd fairyland. I’d be surprised if a quick Google search couldn’t help you out, though.
Found this:
For a given service foo, the package would provide /lib/systemd/system/foo.service. You can check its status using systemctl status foo, or view its logs using journalctl -u foo. To override something in the definition of foo, do:
sudo systemctl edit foo
This creates a directory in /etc/systemd/system named after the unit, and an override.conf file in that directory (/etc/systemd/system/foo.service.d/override.conf). You can add or override settings using this file (or other .conf files in /etc/systemd/system/foo.service.d/).
Did this, but I can’t find any DOCKER_OPTS line here…
cat /lib/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network.target docker.socket
Requires=docker.socket
[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd -H fd://
ExecReload=/bin/kill -s HUP $MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
[Install]
WantedBy=multi-user.target
This is then replaced by the file in /etc/… but I have no idea how to add DOCKER_OPTS…
So, I added
Service]
ExecStart=
ExecStart=/usr/bin/dockerd -s aufs
but this didn’t fix it. Now the error message is:
level=fatal msg="Error starting daemon: error initializing graphdriver: driver not supported"
Add an EnvironmentFile=/etc/default/docker directive in the Service section and put the line in that file.
Sounds like the aufs driver isn’t installed, do you have linux-image-extra-(version) installed?
You mean this:
install linux-image-extra-$(uname -r)
This results in:
E: Unable to locate package linux-image-extra-3.13.0-24-generic
From somewhere I read that since 15.04 or something like that the extra is no longer needed, or something like that…
Uhhh how long have you been holding back kernel upgrades?
$ uname -r
4.4.0-53-generic
Well, I’m running a DigitalOcean droplet and I just did a dist-upgrade…
I’m at lost here. I was supposed to get to sleep 4 hrs ago and by trying to do a “quick update” to the forum I’ve managed to get 4 hrs of downtime and I’m really, really tired.
I’ll just leave the forum down and try to fix this tomorrow… 
At this point, it may be best to take a backup and just restore it on a brand new droplet.
Well, how do I get a backup from discourse, since I can’t run it?
Some older DigitalOcean droplets don’t do kernels the normal way. You need to play silly-buggers to upgrade your kernel. Get yourself running the correct kernel, install the extras kernel package, and then see where that gets you.
Thank you. Will have a look after waking up. Good night!
This usually isn’t the best idea, lowering DNS TTL, getting a new server and moving stuff is way easier.