When to upgrade docker (and how)?

[tophee (Christoph)]

Continuing the discussion from Docker using a lot of CPU:

@mpalmer

> Looks like one should be rather conservative in upgrading docker?

Being conservative in upgrading docker isn’t even enough; we follow that policy (still on 17.06), and we still got hit by a ripper of a bug that exists in all versions from 17.06 to 17.11 (and, by the glacial pace at which the PR is moving, will probably be in 18.01 or whatever the next release will be).

Basically, Docker’s gonna Docker, and all we can do is hang on for the ride and try and dodge the worst of the low-hanging tree branches.

I had made a mental not that I’ll eventuall need to allow my instance to upgrade docker again, after downgrading because of that bug, but I’m not sure when I should do that nor how I do it without breaking stuff… Or do I need to do anything?

[Stephen]

Is 17.06 still the recommended? We’re on 17.10.0-ce ourselves.

If you’ve installed via apt it’s relatively straightforward, uninstall, reboot:

sudo apt-get autoremove -y docker-ce \ 
& sudo apt-get purge docker-ce -y \ 
& sudo rm -rf /etc/docker/ \ 
& sudo rm -f /etc/systemd/system/multi-user.target.wants/docker.service \ 
& sudo rm -rf /var/lib/docker \ 
& sudo systemctl daemon-reload 

then use:

sudo apt-cache policy docker-ce

to see which versions are available to apt, and then:

sudo apt-get install -y docker-ce=[build string]

to install a specific version.

Checking our install we can see builds ranging from 17.04.0-ce all the way up to 18.04.0-ce:

 docker-ce:
  Installed: 17.10.0~ce-0~ubuntu
  Candidate: 18.04.0~ce~3-0~ubuntu
  Version table:
     18.04.0~ce~3-0~ubuntu 500
        500 https://download.docker.com/linux/ubuntu xenial/edge amd64 Packages
     18.03.1~ce-0~ubuntu 500
        500 https://download.docker.com/linux/ubuntu xenial/edge amd64 Packages
     18.03.0~ce-0~ubuntu 500
        500 https://download.docker.com/linux/ubuntu xenial/edge amd64 Packages
     18.02.0~ce-0~ubuntu 500
        500 https://download.docker.com/linux/ubuntu xenial/edge amd64 Packages
     18.01.0~ce-0~ubuntu 500
        500 https://download.docker.com/linux/ubuntu xenial/edge amd64 Packages
     17.12.1~ce-0~ubuntu 500
        500 https://download.docker.com/linux/ubuntu xenial/edge amd64 Packages
     17.12.0~ce-0~ubuntu 500
        500 https://download.docker.com/linux/ubuntu xenial/edge amd64 Packages
     17.11.0~ce-0~ubuntu 500
        500 https://download.docker.com/linux/ubuntu xenial/edge amd64 Packages
 *** 17.10.0~ce-0~ubuntu 500
        500 https://download.docker.com/linux/ubuntu xenial/edge amd64 Packages
        100 /var/lib/dpkg/status
     17.09.1~ce-0~ubuntu 500
        500 https://download.docker.com/linux/ubuntu xenial/edge amd64 Packages
     17.09.0~ce-0~ubuntu 500
        500 https://download.docker.com/linux/ubuntu xenial/edge amd64 Packages
     17.07.0~ce-0~ubuntu 500
        500 https://download.docker.com/linux/ubuntu xenial/edge amd64 Packages
     17.06.2~ce-0~ubuntu 500
        500 https://download.docker.com/linux/ubuntu xenial/edge amd64 Packages
     17.06.1~ce-0~ubuntu 500
        500 https://download.docker.com/linux/ubuntu xenial/edge amd64 Packages
     17.06.0~ce-0~ubuntu 500
        500 https://download.docker.com/linux/ubuntu xenial/edge amd64 Packages
     17.05.0~ce-0~ubuntu-xenial 500
        500 https://download.docker.com/linux/ubuntu xenial/edge amd64 Packages
     17.04.0~ce-0~ubuntu-xenial 500
        500 https://download.docker.com/linux/ubuntu xenial/edge amd64 Packages

[itsbhanusharma (Bhanu Sharma)]

We’re on version 18.04.0-ce, build 3d479c0 across at least 5 known good instances. Haven’t had any issues due to docker yet!

[Stephen]

Haven’t had any issues due to docker yet!

Pretty much a guaranteed way to jinx your installs!

Seriously though, over the past few years we’ve had a number of run-ins with docker versions. Even if instances appear to be building and running ok there have been plenty of performance issues between versions.

Running doesn’t necessarily mean running well.

[itsbhanusharma (Bhanu Sharma)]

I’d totally agree to that, There have been multiple edge cases and other complaints due to people being on a specific version of docker which contains some bug. About the performance part, I haven’t really invested myself into creating stats for known good and the delta between those and the latest so I can’t say much but I never had it trigger any alerts for me.

I’m however curious to know what strategy you use to observe the performance metrics for docker in relation to discourse?

[Stephen]

Usually the difference is observable from whatever server management you use, we’ve got everything in either AWS or DO and most of it is hooked up into New Relic. If there’s a performance difference there’s also usually an observable step up or down in CPU usage.

We’ve not gone so far as to look at transaction times, but as these machines only run Discourse it’s very easy to see if a docker version has an impact on load.

[codinghorror (Jeff Atwood)]

I would use latest Docker unless you have specific reasons not to.

[tophee (Christoph)]

Well, @mpalmer seemed to think differently, and he seemed to suggest that that is not what you’re doing:

Docker using a lot of CPU

we follow that policy (still on 17.06)

But okay. So my question then is: is there still a reason not to use the latest version?

[mpalmer (Matt Palmer)]

I provide insights into our environment, processes, and decision-making for informational purposes only, unless clearly stated otherwise. What we at CDCK do, given our scale, priorities, and the skills and experience of our team, is not necessarily a good idea for everyone else, whose circumstances are likely very different. That’s the “unless you have specific reasons not to” in Jeff’s post.

[tophee (Christoph)]

Yes, I understand. I was just wondering whether such reasons currently exist.

[riking (Kane York)]

I’d interpret that as “specific reasons that your installation of Discourse is different from ‘Most’ installations of Discourse”. Or “you should already know if this applies to you” or “your sysadmin said you need to hold back the upgrade” or …

[tophee (Christoph)]

your sysadmin said you need to hold back the upgrade

Well, something like that happened back then. But I guess the reasons for being more than conservative with docker upgrades that existed some months ago don’t exist anymore. Otherwise we’d have heard them by now.

[codinghorror (Jeff Atwood)]

For example. All our internal stuff is wedded deeply to ipv6. It turns out the rest of the world barely uses ipv6, apparently, because Docker had a release where it blew away all ipv6 routing (which caused a massive internal hosting outage for us), and we were basically the only people to report it as a problem. It still isn’t fixed as far as we know.

There’s a concrete, very specific example of how “the world” and “discourse internal” can deviate.

[tophee (Christoph)]

@tophee

I guess the reasons for being more than conservative with docker upgrades that existed some months ago don’t exist anymore.

Having some doubts now that I found the time to take a closer look. The reason why Matt recommended to downgrade and to lock your docker Version back in December was this bug in docker:

https://github.com/moby/moby/issues/35865

And from what I can tell, it has still not been entirely fixed. So I suppose it’s saver too stick to version 17. 10 for now…

[codinghorror (Jeff Atwood)]

You’d only run into that with very badly behaved inputs though, we were triggering it with some weirdness on our end, which we resolved.

[mpalmer (Matt Palmer)]

It still isn’t fixed as far as we know.

The IPv6 one got fixed, it just missed the next stable release which was frustrating, since we needed to roll custom packages with the bugfix. It’s all the other ones that are still open. The “logging long lines” bug hasn’t even been substantively looked at, as far as I can tell, however we deployed a workaround that shouldn’t cause it to trip when running Discourse. I wouldn’t call wanting to write your logs to Docker “badly behaved”, though, since that’s what Docker’s supposed to be for (“ill-advised”, yes, but not badly behaved).

[peternlewis]

What is not covered here is the “how” part.

Currently, according to app-cache, I have docker-engine installed:

docker-engine/ubuntu-trusty,now 17.05.0~ce-0~ubuntu-trusty amd64 [installed]

and I get the warning

WARNING: Docker version 17.05.0-ce deprecated, recommend upgrade to 17.06.2 or newer.

However the Get Docker CE for Ubuntu instructions start with “Uninstall old versions docker or docker-engine” before installing docker-ce.

Needless to say, anything that starts with “uninstall stuff” makes me more than a little nervous as to how much of my site it plans on deleting, and how safe it is to go ahead and uninstall docker-engine, and then install docker-ce, and expect my site to be there and working when I finish. How safe is this process? If I have a site working with docker-engine, and I follow the Get Docker CE for Ubuntu instructions, uninstall docker-engine, install docker-ce via the “Install using the repository” instructions, am I likely to actually have a working site at the end of it?

[codinghorror (Jeff Atwood)]

It is quite safe since the container file is not touched.

[peternlewis]

OK, followed the instructions and it all went smoothly. The most time consuming part by far was taking a snapshot beforehand as a safety precaution (on top of normal backups and such).

docker info now reports:

Server Version: 18.06.0-ce
Storage Driver: aufs

I include the aufs since the update Get Docker page says:

For Ubuntu 16.04 and higher, the Linux kernel includes support for OverlayFS, and Docker CE uses the overlay2 storage driver by default. If you need to use aufs instead, you need to configure it manually. See aufs

But it seems that the update did not transition to using overlay instead of aufs.

Do I need to do something to make that happen as well, or just ignore it and continue with aufs?

Thanks!

[dandv (Dan Dascalescu)]

@Stephen

then use:

sudo apt-cache policy docker-ce

to see which versions are available to apt, and then:

I get an empty version table with this command. Has something changed about the package in the meantime?

f# apt-cache policy docker-ce
docker-ce:
  Installed: (none)
  Candidate: (none)
  Version table: