For no apparent reason, my server (Hetzner VPS) became virtually unreachable today, i.e. it was so busy that it took multiple attempts to login via ssh and even when it didn’t fail, it literally took minutes until I got a first response. Turns out, docker was running wild:
I ended up restarting the server via the Hetzner dashboard after I lost connection again. Luckily, I did not have to do a hard reset but
shutdown took ages (10 minutes?) until the server was actually down. When I restarted it, discourse was still not reachable (5xx error) so I rebuilt the app. After that, I first thought it was still not reachable but then the 5xx error went away and it’s working now. So I guess the container just needs some time to start up too and it might have worked without the rebuild.
In any case: what could be the reason for docker using so much CPU?
In case it matters,
docker info gives me this:
Containers: 1 Running: 1 Paused: 0 Stopped: 0 Images: 12 Server Version: 17.11.0-ce Storage Driver: aufs Root Dir: /var/lib/docker/aufs Backing Filesystem: extfs Dirs: 54 Dirperm1 Supported: true Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog Swarm: inactive Runtimes: runc Default Runtime: runc Init Binary: docker-init containerd version: 992280e8e265f491f7a624ab82f3e238be086e49 runc version: 0351df1c5a66838d0c392b4ac4cf9450de844e2d init version: 949e6fa Security Options: apparmor seccomp Profile: default Kernel Version: 4.4.0-104-generic Operating System: Ubuntu 16.04.3 LTS OSType: linux Architecture: x86_64 CPUs: 2 Total Memory: 1.953GiB Name: Ubuntu-1604-xenial-64-minimal ID: 3DEN:KXQG:B3G3:DYG6:DVJU:CI6W:YUD4:OIBV:LJBW:ZG5Q:D2ZH:PDYN Docker Root Dir: /var/lib/docker Debug Mode (client): false Debug Mode (server): false Registry: https://index.docker.io/v1/ Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false WARNING: No swap limit support
And is there anything that can be done to prevent docker shutting down the entire server? I saw that docker can be limited in how much CPU it may use, but I’m not sure how to implement this with discourse and whether it is advisable at all, since it would artificially restrain the resources that discourse has access to…