# Docker using a lot of CPU

**URL:** https://meta.discourse.org/t/docker-using-a-lot-of-cpu/76788
**Category:** Self-hosting
**Tags:** server-resources
**Created:** [December 26, 2017, 10:37pm UTC](https://meta.discourse.org/t/docker-using-a-lot-of-cpu/76788 "2017-12-26T22:37:01Z")
**Posts on this page:** 9
**Page:** 1

<div class="post-metadata">

### Author: ![tophee](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/tophee/32/73406_2.png) [@tophee](https://meta.discourse.org/u/tophee)
#### Post date: [December 26, 2017, 10:37pm UTC](https://meta.discourse.org/t/docker-using-a-lot-of-cpu/76788/1 "2017-12-26T22:37:01Z")

</div>

For no apparent reason, my server (Hetzner VPS) became virtually unreachable today, i.e. it was so busy that it took multiple attempts to login via ssh and even when it didn’t fail, it literally took minutes until I got a first response. Turns out, docker was running wild:

 ![image](https://global.discourse-cdn.com/meta/original/3X/c/1/c15c2d6de062d7039822907926eea7e45eaf6a8a.png)

I ended up restarting the server via the Hetzner dashboard after I lost connection again. Luckily, I did not have to do a hard reset but `shutdown` took ages (10 minutes?) until the server was actually down. When I restarted it, discourse was still not reachable (5xx error) so I rebuilt the app. After that, I first thought it was still not reachable but then the 5xx error went away and it’s working now. So I guess the container just needs some time to start up too and it might have worked without the rebuild.

In any case: what could be the reason for docker using so much CPU?

In case it matters, `docker info` gives me this:

```plaintext
Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 12
Server Version: 17.11.0-ce
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 54
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 992280e8e265f491f7a624ab82f3e238be086e49
runc version: 0351df1c5a66838d0c392b4ac4cf9450de844e2d
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-104-generic
Operating System: Ubuntu 16.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.953GiB
Name: Ubuntu-1604-xenial-64-minimal
ID: 3DEN:KXQG:B3G3:DYG6:DVJU:CI6W:YUD4:OIBV:LJBW:ZG5Q:D2ZH:PDYN
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

```

And is there anything that can be done to prevent docker shutting down the entire server? I saw that [docker can be limited in how much CPU it may use](https://docs.docker.com/engine/admin/resource_constraints/#cpu), but I’m not sure how to implement this with discourse and whether it is advisable at all, since it would artificially restrain the resources that discourse has access to…

---

<div class="post-metadata">

### Author: ![codinghorror](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/codinghorror/32/110067_2.png) [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [December 26, 2017, 10:38pm UTC](https://meta.discourse.org/t/docker-using-a-lot-of-cpu/76788/2 "2017-12-26T22:38:31Z")

</div>

What version of docker? @mpalmer found a nasty bug in the current version that will cause problems during backups.

---

<div class="post-metadata">

### Author: ![tophee](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/tophee/32/73406_2.png) [@tophee](https://meta.discourse.org/u/tophee)
#### Post date: [December 26, 2017, 10:40pm UTC](https://meta.discourse.org/t/docker-using-a-lot-of-cpu/76788/3 "2017-12-26T22:40:19Z")

</div>

```plaintext
Client:
 Version: 17.11.0-ce
 API version: 1.34
 Go version: go1.8.3
 Git commit: 1caf76c
 Built: Mon Nov 20 18:37:39 2017
 OS/Arch: linux/amd64

Server:
 Version: 17.11.0-ce
 API version: 1.34 (minimum version 1.12)
 Go version: go1.8.3
 Git commit: 1caf76c
 Built: Mon Nov 20 18:36:09 2017
 OS/Arch: linux/amd64
 Experimental: false

```

---

<div class="post-metadata">

### Author: ![mpalmer](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/mpalmer/32/45740_2.png) [@mpalmer](https://meta.discourse.org/u/mpalmer)
#### Post date: [December 26, 2017, 10:42pm UTC](https://meta.discourse.org/t/docker-using-a-lot-of-cpu/76788/4 "2017-12-26T22:42:45Z")

</div>

The bug that I found would produce the opposite of CPU consumption; it causes the container shim to _stop_ doing anything useful.

The `docker info` shows 17.11.0-ce, which is the buggy-af version, so you’ll want to [downgrade](https://meta.discourse.org/t/discourse-web-interface-becomes-unresponsive-a-few-minutes-after-starting/76643/14), but not to fix this bug – this one’s a new one, and very definitely not Discourse-specific. If the problem persists, you’ll want to raise this on [the Docker forums](https://forums.docker.com/) (it should feel familiar…), or if you can get a reliable reproduction of the problem, [file a bug report](https://github.com/moby/moby/issues/new).

---

<div class="post-metadata">

### Author: ![tophee](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/tophee/32/73406_2.png) [@tophee](https://meta.discourse.org/u/tophee)
#### Post date: [December 26, 2017, 10:47pm UTC](https://meta.discourse.org/t/docker-using-a-lot-of-cpu/76788/5 "2017-12-26T22:47:46Z")

</div>

> [@mpalmer](#):
>
> which is the buggy-af version

How did I get that one? I don’t intentionally install experimental versions. All I do is `apt-get upgrade` from time to time… Looks like one should be rather conservative in upgrading docker?

---

<div class="post-metadata">

### Author: ![mpalmer](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/mpalmer/32/45740_2.png) [@mpalmer](https://meta.discourse.org/u/mpalmer)
#### Post date: [December 26, 2017, 10:57pm UTC](https://meta.discourse.org/t/docker-using-a-lot-of-cpu/76788/6 "2017-12-26T22:57:15Z")

</div>

> [@tophee](#):
>
> How did I get that one? I don’t intentionally install experimental versions. All I do is apt-get upgrade from time to time…

You probably got the “experimental” version because that’s what everyone gets if they initially installed via `curl https://get.docker.com | sh`. You have to change `/etc/apt/sources.list.d/docker.list` to track the `stable` repo rather than `edge` or whatever the default is.

> [@tophee](#):
>
> Looks like one should be rather conservative in upgrading docker?

Being conservative in upgrading docker isn’t even enough; we follow that policy (still on 17.06), and we _still_ got hit by [a ripper of a bug](https://github.com/moby/moby/issues/35757) that exists in all versions from 17.06 to 17.11 (and, by the glacial pace at which the PR is moving, will probably be in 18.01 or whatever the next release will be).

Basically, Docker’s gonna Docker, and all we can do is hang on for the ride and try and dodge the worst of the low-hanging tree branches.

---

<div class="post-metadata">

### Author: ![tophee](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/tophee/32/73406_2.png) [@tophee](https://meta.discourse.org/u/tophee)
#### Post date: [December 26, 2017, 11:05pm UTC](https://meta.discourse.org/t/docker-using-a-lot-of-cpu/76788/7 "2017-12-26T23:05:45Z")

</div>

Thanks Matt! Downgrading was super easy, app is currently rebuilding and I assume that was the end of that trouble.

👍 Really fantastic work that probably saved me countless hours and unbearable headaches, because a bug in docker is about the last thing I would assume, so I would just go crazy trying to understand what I’m doing wrong (and that’s particularly hard when you don’t even know what you’re doing right!).

---

<div class="post-metadata">

### Author: ![mpalmer](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/mpalmer/32/45740_2.png) [@mpalmer](https://meta.discourse.org/u/mpalmer)
#### Post date: [December 26, 2017, 11:27pm UTC](https://meta.discourse.org/t/docker-using-a-lot-of-cpu/76788/8 "2017-12-26T23:27:32Z")

</div>

> [@tophee](#):
>
> a bug in docker is about the last thing I would assume, so I would just go crazy trying to understand what I’m doing wrong

I feel you – we spent the better part of a week blaming backups, sidekiq, and whatever else looked good before I managed to get a shell on a site that the owner was happy to leave down for hours whilst I rummaged around and got to the bottom of the problem.

---

<div class="post-metadata">

### Author: ![codinghorror](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/codinghorror/32/110067_2.png) [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [December 27, 2017, 2:14am UTC](https://meta.discourse.org/t/docker-using-a-lot-of-cpu/76788/9 "2017-12-27T02:14:50Z")

</div>


