When to upgrade docker (and how)?


(Christoph) #1

Continuing the discussion from Docker using a lot of CPU:

I had made a mental not that I’ll eventuall need to allow my instance to upgrade docker again, after downgrading because of that bug, but I’m not sure when I should do that nor how I do it without breaking stuff… Or do I need to do anything?


(Stephen) #2

Is 17.06 still the recommended? We’re on 17.10.0-ce ourselves.

If you’ve installed via apt it’s relatively straightforward, uninstall, reboot:

sudo apt-get autoremove -y docker-ce \ 
& sudo apt-get purge docker-ce -y \ 
& sudo rm -rf /etc/docker/ \ 
& sudo rm -f /etc/systemd/system/multi-user.target.wants/docker.service \ 
& sudo rm -rf /var/lib/docker \ 
& sudo systemctl daemon-reload 

then use:

sudo apt-cache policy docker-ce

to see which versions are available to apt, and then:

sudo apt-get install -y docker-ce=[build string]

to install a specific version.

Checking our install we can see builds ranging from 17.04.0-ce all the way up to 18.04.0-ce:

 docker-ce:
  Installed: 17.10.0~ce-0~ubuntu
  Candidate: 18.04.0~ce~3-0~ubuntu
  Version table:
     18.04.0~ce~3-0~ubuntu 500
        500 https://download.docker.com/linux/ubuntu xenial/edge amd64 Packages
     18.03.1~ce-0~ubuntu 500
        500 https://download.docker.com/linux/ubuntu xenial/edge amd64 Packages
     18.03.0~ce-0~ubuntu 500
        500 https://download.docker.com/linux/ubuntu xenial/edge amd64 Packages
     18.02.0~ce-0~ubuntu 500
        500 https://download.docker.com/linux/ubuntu xenial/edge amd64 Packages
     18.01.0~ce-0~ubuntu 500
        500 https://download.docker.com/linux/ubuntu xenial/edge amd64 Packages
     17.12.1~ce-0~ubuntu 500
        500 https://download.docker.com/linux/ubuntu xenial/edge amd64 Packages
     17.12.0~ce-0~ubuntu 500
        500 https://download.docker.com/linux/ubuntu xenial/edge amd64 Packages
     17.11.0~ce-0~ubuntu 500
        500 https://download.docker.com/linux/ubuntu xenial/edge amd64 Packages
 *** 17.10.0~ce-0~ubuntu 500
        500 https://download.docker.com/linux/ubuntu xenial/edge amd64 Packages
        100 /var/lib/dpkg/status
     17.09.1~ce-0~ubuntu 500
        500 https://download.docker.com/linux/ubuntu xenial/edge amd64 Packages
     17.09.0~ce-0~ubuntu 500
        500 https://download.docker.com/linux/ubuntu xenial/edge amd64 Packages
     17.07.0~ce-0~ubuntu 500
        500 https://download.docker.com/linux/ubuntu xenial/edge amd64 Packages
     17.06.2~ce-0~ubuntu 500
        500 https://download.docker.com/linux/ubuntu xenial/edge amd64 Packages
     17.06.1~ce-0~ubuntu 500
        500 https://download.docker.com/linux/ubuntu xenial/edge amd64 Packages
     17.06.0~ce-0~ubuntu 500
        500 https://download.docker.com/linux/ubuntu xenial/edge amd64 Packages
     17.05.0~ce-0~ubuntu-xenial 500
        500 https://download.docker.com/linux/ubuntu xenial/edge amd64 Packages
     17.04.0~ce-0~ubuntu-xenial 500
        500 https://download.docker.com/linux/ubuntu xenial/edge amd64 Packages

(Bhanu Sharma) #3

We’re on version 18.04.0-ce, build 3d479c0 across at least 5 known good instances. Haven’t had any issues due to docker yet!


(Stephen) #4

Pretty much a guaranteed way to jinx your installs!

Seriously though, over the past few years we’ve had a number of run-ins with docker versions. Even if instances appear to be building and running ok there have been plenty of performance issues between versions.

Running doesn’t necessarily mean running well.


(Bhanu Sharma) #5

I’d totally agree to that, There have been multiple edge cases and other complaints due to people being on a specific version of docker which contains some bug. About the performance part, I haven’t really invested myself into creating stats for known good and the delta between those and the latest so I can’t say much but I never had it trigger any alerts for me.

I’m however curious to know what strategy you use to observe the performance metrics for docker in relation to discourse?


(Stephen) #6

Usually the difference is observable from whatever server management you use, we’ve got everything in either AWS or DO and most of it is hooked up into New Relic. If there’s a performance difference there’s also usually an observable step up or down in CPU usage.

We’ve not gone so far as to look at transaction times, but as these machines only run Discourse it’s very easy to see if a docker version has an impact on load.


(Jeff Atwood) #7

I would use latest Docker unless you have specific reasons not to.


(Christoph) #8

Well, @mpalmer seemed to think differently, and he seemed to suggest that that is not what you’re doing:

But okay. So my question then is: is there still a reason not to use the latest version?


(Matt Palmer) #9

I provide insights into our environment, processes, and decision-making for informational purposes only, unless clearly stated otherwise. What we at CDCK do, given our scale, priorities, and the skills and experience of our team, is not necessarily a good idea for everyone else, whose circumstances are likely very different. That’s the “unless you have specific reasons not to” in Jeff’s post.


(Christoph) #10

Yes, I understand. I was just wondering whether such reasons currently exist.


(Kane York) #11

I’d interpret that as “specific reasons that your installation of Discourse is different from ‘Most’ installations of Discourse”. Or “you should already know if this applies to you” or “your sysadmin said you need to hold back the upgrade” or …


(Christoph) #12

Well, something like that happened back then. But I guess the reasons for being more than conservative with docker upgrades that existed some months ago don’t exist anymore. Otherwise we’d have heard them by now.


(Jeff Atwood) #13

For example. All our internal stuff is wedded deeply to ipv6. It turns out the rest of the world barely uses ipv6, apparently, because Docker had a release where it blew away all ipv6 routing (which caused a massive internal hosting outage for us), and we were basically the only people to report it as a problem. It still isn’t fixed as far as we know.

There’s a concrete, very specific example of how “the world” and “discourse internal” can deviate.


(Christoph) #14

Having some doubts now that I found the time to take a closer look. The reason why Matt recommended to downgrade and to lock your docker Version back in December was this bug in docker:

And from what I can tell, it has still not been entirely fixed. So I suppose it’s saver too stick to version 17. 10 for now…


(Jeff Atwood) #15

You’d only run into that with very badly behaved inputs though, we were triggering it with some weirdness on our end, which we resolved.


(Matt Palmer) #16

The IPv6 one got fixed, it just missed the next stable release which was frustrating, since we needed to roll custom packages with the bugfix. It’s all the other ones that are still open. The “logging long lines” bug hasn’t even been substantively looked at, as far as I can tell, however we deployed a workaround that shouldn’t cause it to trip when running Discourse. I wouldn’t call wanting to write your logs to Docker “badly behaved”, though, since that’s what Docker’s supposed to be for (“ill-advised”, yes, but not badly behaved).