When to upgrade docker (and how)?

tophee · May 9, 2018, 10:57pm

Having some doubts now that I found the time to take a closer look. The reason why Matt recommended to downgrade and to lock your docker Version back in December was this bug in docker:

github.com/moby/moby

Logging long lines breaks container

opened 02:15AM - 24 Dec 17 UTC

closed 08:36PM - 03 Aug 18 UTC

mpalmer

area/logging status/needs-attention version/17.11

**Description** In a container which has a TTY allocated (`docker run -t`), i…f a process whose stdout/stderr is being collected by `docker-containerd-shim` sends a line several thousand characters long (observed to be anywhere in the range of about 5k-9k), `docker-containerd-shim` stops reading from the PTY master. This causes processes writing to the terminal to hang in the `write`(2) call, which quickly renders the container unserviceable. **Steps to reproduce the issue:** Here's how I can get a reliable reproduction: 1. Install `docker-ce` version `17.11.0~ce-0~ubuntu` on an Ubuntu 16.04 x64 system. 2. Start a container like this: docker run -t busybox /bin/sh -c 's="a"; while true; do echo $s | wc -c; echo $s; s="a$s"; done' 3. The screen will quickly start scrolling successively longer lines of `a`s, however within a few seconds, the terminal will stop scrolling, having printed a number in the mid-to-high thousands and a long line of `a` characters. While the number (indicating the length of the string printed immediately afterwards) will vary on each run, I've never seen it reach 10k. **Describe the results you received:** The container's logs stop displaying lines. **Describe the results you expected:** The container's logs continue to display lines until the container is terminated. **Additional information you deem important (e.g. issue happens only occasionally):** The exact line length that the demo program stops at varies between runs by a considerable margin (5k to 9k in my observations), which suggests this isn't a buffer size problem, but is instead a race condition somewhere. The problem only occurs when a TTY is allocated, suggesting this is not a general problem with log I/O. If you re-run the same `docker run` command but *without* the `-t` option (which, I've noticed, feeds the container output via a pipe, rather than a pty), you can let this run as long as your patience will last. I've let it go to line lengths of 42k before getting bored and hitting ^C. Downgrading to `17.10.0~ce-0~ubuntu` (commit ID `f4ffd25`) and re-running the problematic command (*with* `-t`) also does not exhibit the unhelpful behaviour, suggesting this problem is a recent addition to the pantheon, and (if it is a problem in, say, the kernel) it's being triggered by recent changes in Docker. **Output of `docker version`:** ``` Client: Version: 17.11.0-ce API version: 1.34 Go version: go1.8.3 Git commit: 1caf76c Built: Mon Nov 20 18:36:37 2017 OS/Arch: linux/amd64 Server: Version: 17.11.0-ce API version: 1.34 (minimum version 1.12) Go version: go1.8.3 Git commit: 1caf76c Built: Mon Nov 20 18:35:10 2017 OS/Arch: linux/amd64 Experimental: false ``` **Output of `docker info`:** ``` Containers: 6 Running: 4 Paused: 0 Stopped: 2 Images: 1 Server Version: 17.11.0-ce Storage Driver: aufs Root Dir: /var/lib/docker/aufs Backing Filesystem: extfs Dirs: 11 Dirperm1 Supported: false Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog Swarm: inactive Runtimes: runc Default Runtime: runc Init Binary: docker-init containerd version: 992280e8e265f491f7a624ab82f3e238be086e49 runc version: 0351df1c5a66838d0c392b4ac4cf9450de844e2d init version: 949e6fa Security Options: apparmor Kernel Version: 3.13.0-135-generic Operating System: Ubuntu 14.04.5 LTS OSType: linux Architecture: x86_64 CPUs: 2 Total Memory: 1.955GiB Name: docker-test ID: MF2K:XQZL:SO7O:ZU6O:JIAR:AJZC:3EE2:YILN:PKN2:IXST:7KMB:J7K5 Docker Root Dir: /var/lib/docker Debug Mode (client): false Debug Mode (server): false Registry: https://index.docker.io/v1/ Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: true WARNING: No swap limit support ```

And from what I can tell, it has still not been entirely fixed. So I suppose it’s saver too stick to version 17. 10 for now…

Topic		Replies	Views
Upgrade to 3.3 is failing for me Installation	22	153	October 22, 2024
Docker using a lot of CPU Installation server-resources	8	5299	December 27, 2017
Docker shim error on CentOS 7 Support	26	9344	June 8, 2024
Can't run ./launcher rebuild app - Docker storage driver is unsupported Support	72	12414	September 25, 2019
Ubuntu 16.04 LTS - upgrade or migrate? Installation	14	2559	March 28, 2023

When to upgrade docker (and how)?

Related topics