Cleanup failing due to version mismatch

docker

(Steven Greco) #1

I have been trying to run the ./launcher cleanup command and have been receiving this error.

Starting Cleanup (bytes free 23089320)
time="2015-11-25T20:21:03-05:00" level=fatal msg="Error response from daemon: client and server don't have same version (client : 1.18, server: 1.16)"
Finished Cleanup (bytes free 23089304)

When I do a git pull it is saying that I am already up to date. Here is what my version info shows.

xxx@GrimReaperGamers:/var/discourse# git version
git version 1.9.1
xxx@GrimReaperGamers:/var/discourse# docker version
Client version: 1.4.1
Client API version: 1.16
Go version (client): go1.3.3
Git commit (client): 5bc2ff8
OS/Arch (client): linux/amd64
Server version: 1.4.1
Server API version: 1.16
Go version (server): go1.3.3
Git commit (server): 5bc2ff8

(Matt Palmer) #2

I would hazard a guess that the docker binary that is run on the command line is not the docker binary that is run from within launcher. That is possible, if you have a binary named docker.io on your system:

docker_path=`which docker.io || which docker`

What does which docker.io print on your system?


(Sam Saffron) #3

Looks like you are running an old version of docker all our images are now built with 1.6 support and up, we should probably add to pre-reqs.


(Steven Greco) #4

yup that was the issue. When i ran an apt-get update i noticed that there was a message that said docker package has been left back. I had to specifically tell it to upgrade that package.

EDIT: It also looks like I needed to update my repo for docker as I was still on lxc-docker and now moved to docker engine. Install procedures if anyone needs them for ubuntu. Redirecting…

Thanks


(Christopher Heald) #5

I am experiencing this too. We had locked down our container version due to an ImageMagick bug, but we are now no longer specifying a base_image. ./launcher cleanup throws

time="2015-12-01T19:01:15-05:00"
level=fatal msg="Error response from daemon: client and server don't have same version (client : 1.18, server: 1.17)" 

I tried to update docker using @Grex315’s link, and that left me with a non-running Discourse instance, and the following error from ./launcher:

Your Docker installation is not using a supported filesystem if we were to proceed you may have a broken install. aufs is the recommended filesystem you should be using (zfs/btrfs and overlay may work as well) You can tell what filesystem you are using by running “docker info” and looking at the driver

If you wish to continue anyway using your existing unsupported filesystem read the source code of launcher and figure out how to bypass this.

I had to install docker 1.5.0 to be able to access the Discourse container again. Fortunately, this is on our dev server, but we will run out of disk space as we do frequent rebuilds to test plugins.

I don’t know what to do to address this.


(Sam Saffron) #6

You no longer need to have your version locked down, we fixed the bug a while back.


(Christopher Heald) #7

Thanks Sam. The version is no longer locked down, and Discourse rebuild didn’t help, I’m still getting the same error message on ./launcher cleanup.


(Sam Saffron) #8

You need to get on latest docker … can you upgrade it?


(Matt Palmer) #9

That can happen if, when you upgrade Docker, it gets the sense that it should use another filesystem backend other than aufs. I’ve found that adding -s aufs to the docker daemon command line (editing /etc/default/docker and setting DOCKER_OPTS works on Debian/Ubuntu hosts) and then restarting Docker (service docker restart) does the trick.


(Christopher Heald) #10

I can, but then I get back down the rabbit-hole:

# apt-get upgrade docker
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Calculating upgrade... Done
The following packages were automatically installed and are no longer required:
  aufs-tools cgroup-lite git git-man liberror-perl
Use 'apt-get autoremove' to remove them.
The following packages will be REMOVED:
  lxc-docker-1.5.0
The following NEW packages will be installed:
  docker
The following packages have been kept back:
  linux-generic linux-headers-generic linux-image-generic
0 upgraded, 1 newly installed, 1 to remove and 3 not upgraded.
Need to get 0 B/12.2 kB of archives.
After this operation, 15.1 MB disk space will be freed.
Do you want to continue? [Y/n] 
(Reading database ... 89139 files and directories currently installed.)
Removing lxc-docker-1.5.0 (1.5.0) ...
docker stop/waiting
Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
Selecting previously unselected package docker.
(Reading database ... 89086 files and directories currently installed.)
Preparing to unpack .../docker_1.5-1_amd64.deb ...
Unpacking docker (1.5-1) ...
Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
Setting up docker (1.5-1) ...

Which leads me back to:

/var/discourse# ./launcher cleanup
Your Docker installation is not using a supported filesystem if we were to proceed you may have a broken install.
aufs is the recommended filesystem you should be using (zfs/btrfs and overlay may work as well)
You can tell what filesystem you are using by running "docker info" and looking at the driver

If you wish to continue anyway using your existing unsupported filesystem
read the source code of launcher and figure out how to bypass this.

And docker info gives me:

/usr/bin/docker: No such file or directory

And the fun keeps on going:

# service docker status
docker stop/waiting
# service docker start
start: Job failed to start

To get Discourse up and running again, I have to apt-get install docker-1.5.0

# apt-get install docker-1.5.0
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Note, selecting 'lxc-docker-1.5.0' for regex 'docker-1.5.0'
The following packages will be REMOVED:
  docker
The following NEW packages will be installed:
  lxc-docker-1.5.0
0 upgraded, 1 newly installed, 1 to remove and 3 not upgraded.
Need to get 0 B/4,632 kB of archives.
After this operation, 15.1 MB of additional disk space will be used.
Do you want to continue? [Y/n] 
(Reading database ... 89093 files and directories currently installed.)
Removing docker (1.5-1) ...
Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
Selecting previously unselected package lxc-docker-1.5.0.
(Reading database ... 89086 files and directories currently installed.)
Preparing to unpack .../lxc-docker-1.5.0_1.5.0_amd64.deb ...
Unpacking lxc-docker-1.5.0 (1.5.0) ...
Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
Processing triggers for ureadahead (0.100.0-16) ...
Setting up lxc-docker-1.5.0 (1.5.0) ...
docker start/running, process 22311

(Christopher Heald) #11

I gave that a try, no luck. DOCKER_OPTS="-s aufs" is present in my /etc/default/docker, but the service won’t (re)start:

# service docker restart
stop: Unknown instance: 
start: Job failed to start

(Matt Palmer) #12

Your system is, in a word, boned. You definitely want to be running Docker 1.6 or later (preferably the latest), for both Discourse and other reasons, so you should spend the time to debug what’s going wrong with your Docker installation and fix it up.

For the record, this is how all the Discourse machines are setup to install docker, and it all works rather nicely:

$ cat /etc/apt/sources.list.d/docker.list 
deb https://apt.dockerproject.org/repo ubuntu-trusty main
$ dpkg -l docker*
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                        Version                    Architecture               Description
+++-===========================================-==========================-==========================-    ============================================================================================
un  docker                                      <none>                     <none>                     (no description available)
ii  docker-engine                               1.9.1-0~trusty             amd64                      Docker: the open-source     application container engine
un  docker.io                                   <none>                     <none>                     (no description available)

(Christopher Heald) #13

Tell me what you really think :wink:

Our install is a clone of the Civilized Discourse Construction Kit one-time install, performed in February 2015. So when did the contents of /etc/apt/sources.list.d/docker.list change from the following?

# cat /etc/apt/sources.list.d/docker.list
deb https://get.docker.com/ubuntu docker main

# dpkg -l docker*
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                   Version                  Architecture             Description
+++-======================================-========================-========================-=================================================================================
rc  docker                                 1.5-1                    amd64                    System tray for KDE3/GNOME2 docklet applications
un  docker.io                              <none>                   <none>                   (no description available)

(Christopher Heald) #14

At this stage, I am concerned that our production server, which is a fairly vanilla CDCK install, is also - ahem - boned. I can’t imagine us being the only instance that is going to face this issue.


(Matt Palmer) #15

According to a bit of git blame archaeology on https://github.com/docker/docker/blob/master/hack/install.sh, the change was made on July 19, 2015. Any new Docker installs done with the curl | sh method after that date would have gotten the new apt repo. The old repo that you’re using does have 1.9.1, it’s just called lxc-docker, not docker-engine. What that old repo doesn’t appear to have, however, is a docker package, so you may want to work out where that’s coming from, with apt-cache policy docker. I suspect that may unlock the key to the mystery.


(Christopher Heald) #16

This is what I get on our dev server:

# apt-cache policy docker
docker:
  Installed: (none)
  Candidate: 1.5-1
  Version table:
     1.5-1 0
        500 http://mirrors.digitalocean.com/ubuntu/ trusty-updates/universe amd64 Packages
        100 /var/lib/dpkg/status
     1.4-5ubuntu1 0
        500 http://mirrors.digitalocean.com/ubuntu/ trusty/universe amd64 Packages

And this is what I get on our production server:

$ apt-cache policy docker
docker:
  Installed: (none)
  Candidate: 1.5-1
  Version table:
     1.5-1 0
        500 http://mirrors.digitalocean.com/ubuntu/ trusty-updates/universe amd64 Packages
     1.4-5ubuntu1 0
        500 http://mirrors.digitalocean.com/ubuntu/ trusty/universe amd64 Packages

(Matt Palmer) #17

Ooooooh, Ubuntu are shipping Docker as docker, are they? Never noticed.

It kinda looks like Ubuntu’s docker package wasn’t even the one that was initially installed:

# apt-get upgrade docker
[...]
The following packages will be REMOVED:
  lxc-docker-1.5.0
The following NEW packages will be installed:
  docker

If you want to stick with the apt repo you already had, you can apt-get install lxc-docker, while if you want to switch to the new apt repo (apt.dockerproject.org) then it’ll be apt-get install docker-engine. Either way, make sure you apt-get remove --purge docker, because it is a pox on everyone’s house.


(Christopher Heald) #18

And that did the trick!

I went with the new apt repo, I will stick with the Discourse team’s preferred setup.

So here are the steps to correct this:

Edit /etc/apt/sources.list.d/docker.list and replace contents (which, prior to mid-July 2015, were deb https://get.docker.com/ubuntu docker main) with:

deb https://apt.dockerproject.org/repo ubuntu-trusty main

Then run the following:

apt-get remove --purge docker

apt-get update

apt-get install docker-engine

Results:

# docker info
Containers: 1
Images: 41
Server Version: 1.9.1
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 43
 Dirperm1 Supported: false
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.13.0-43-generic
Operating System: Ubuntu 14.04.3 LTS [...]

and

# dpkg -l docker*
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                   Version                  Architecture             Description
+++-======================================-========================-========================-=================================================================================
un  docker                                 <none>                   <none>                   (no description available)
ii  docker-engine                          1.9.1-0~trusty           amd64                    Docker: the open-source application container engine
un  docker.io                              <none>                   <none>                   (no description available)

./launcher cleanup is now working as expected once again.

Huge thanks @mpalmer and @sam. Many rounds of :beers: for you.


(Sam) #19

Incidentally - what is considered best practice here? Is it recommended to switch over to the new repo and the latest docker version?

Or instead keep working with what’s useable and only upgrade when necessary? My live server has lxc-docker at 1.7.1, which appears to be working fine at the moment.


(Jeff Atwood) #20

lxc-docker updates to 1.9.1 for me, so it seems relatively current so far?