Container fails startup after yum update


(Mark Moorcroft) #1
[root@xxx /var/discourse]# ./launcher start app

starting up existing container
+ /usr/bin/docker.io start app
Error response from daemon: shim error: docker-runc not installed on system
Error: failed to start containers: app

Linux xxx.gov 3.10.0-514.6.1.el7.x86_64 #1 SMP Wed Jan 18 13:06:36 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

I can however start a new container “docker run -it --rm busybox sh” with no errors.

Ideas?


(Rafael dos Santos Silva) #2

What does docker info prints?


Docker shim error on CentOS 7
(Mark Moorcroft) #3
[root@xxx ~]# docker info
Containers: 1
 Running: 0
 Paused: 0
 Stopped: 1
Images: 38
Server Version: 1.12.5
Storage Driver: devicemapper
 Pool Name: vg_alder_main-docker--pool
 Pool Blocksize: 524.3 kB
 Base Device Size: 107.4 GB
 Backing Filesystem: xfs
 Data file:
 Metadata file:
 Data Space Used: 15.88 GB
 Data Space Total: 28.5 GB
 Data Space Available: 12.63 GB
 Metadata Space Used: 3.076 MB
 Metadata Space Total: 67.11 MB
 Metadata Space Available: 64.03 MB
 Thin Pool Minimum Free Space: 2.85 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: true
 Deferred Deletion Enabled: true
 Deferred Deleted Device Count: 0
 Library Version: 1.02.135-RHEL7 (2016-11-16)
Logging Driver: json-file
Cgroup Driver: systemd
Plugins:
 Volume: local
 Network: host bridge null overlay
Swarm: inactive
Runtimes: docker-runc runc
Default Runtime: docker-runc
Security Options: seccomp
Kernel Version: 3.10.0-514.6.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 2
CPUs: 2
Total Memory: 3.86 GiB
Name: alder.ndc.nasa.gov
ID: 4HCD:CXPT:UOD3:TH6S:6CZ3:XOEV:FYIF:GHBZ:OCIR:JBTX:UVG6:G37X
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
 127.0.0.0/8
Registries: docker.io (secure)

(Matt Palmer) #4

You’re not running the same Docker binary as ./launcher is:

What happens if you try to run /usr/bin/docker.io run -it --rm busybox sh?


(Mark Moorcroft) #5

[root@xxx ~]# /usr/bin/docker.io run -it --rm busybox sh
/ #

No errors


(Matt Palmer) #6

“Curiouser and curiouser!” cried Alice (she was so much surprised, that for the moment she quite forgot how to speak good English).

What about /usr/bin/docker.io start app? Also, does docker.io info give different results to docker info? Are they symlinked, or at least provided by the same RPM? If they’re in different RPMs, try removing the older one (it should be the one providing docker.io, because that’s a legacy name), and see if ./launcher becomes infinitely happier.


(Mark Moorcroft) #7

They are symlinked.

/usr/bin/docker.io start app gives the same error as ./launcher start app.


(Matt Palmer) #8

Oooooh… have you done a ./launcher rebuild app? Or at least a docker rm app? Try the latter first, and if that still doesn’t work, give the former a shot.


(Mark Moorcroft) #9

I had to rebuild, but now it works.

Thanks


(Matt Palmer) #10

Isn’t it great how Docker makes everything so much easier…


(Mark Moorcroft) #11

Right… sooo much

:mdr:


(Rafael dos Santos Silva) #12

You are using devicemapper, and this isn’t what we recommend as a storage drive, so keep that in mind.


(Mark Moorcroft) #13

Thanks, note to self.


(Jeff Atwood) #14

You will experience drive corruption with devicemapper, it is just a matter of time. We block installs if devicemapper is detected.


(Mark Moorcroft) #15

Is this something you see as a permanent problem, or will later OS versions possibly allow reliable devmapper? Is it open bugs?


(Jeff Atwood) #16

Devicemapper is not stable in our experience, so we don’t support it.


(Mark Moorcroft) #17

I’m wondering if that is an over simplified policy? Today turned ugly when I tried to update Discourse. It said I needed to update Docker. I was reminded that I never switched devmapper from loop mode to direct-lvm. According to the Docker site devmapper direct-lvm is the recommended method on Redhat and CentOS. Because AUFS is not supported. Now I have to use --skip-prereqs using the Docker recommended storage driver. Is the default loop mode actually the problem because people never do the xtra steps to direct-lvm (block level)?

> [root@xxxx ~]# docker info
> Containers: 2
>  Running: 0
>  Paused: 0
>  Stopped: 2
> Images: 39
> Server Version: 1.12.6
> Storage Driver: devicemapper
>  Pool Name: docker-thinpool
>  Pool Blocksize: 524.3 kB
>  Base Device Size: 10.74 GB
>  Backing Filesystem: xfs
>  Data file:
>  Metadata file:
>  Data Space Used: 490.7 MB
>  Data Space Total: 490.7 MB
>  Data Space Available: 0 B
>  Metadata Space Used: 122.9 kB
>  Metadata Space Total: 4.194 MB
>  Metadata Space Available: 4.071 MB
>  Thin Pool Minimum Free Space: 48.76 MB
>  Udev Sync Supported: true
>  Deferred Removal Enabled: true
>  Deferred Deletion Enabled: true
>  Deferred Deleted Device Count: 0
>  Library Version: 1.02.135-RHEL7 (2016-11-16)
> Logging Driver: json-file
> Cgroup Driver: systemd
> Plugins:
>  Volume: local
>  Network: host bridge null overlay
> Swarm: inactive
> Runtimes: docker-runc runc
> Default Runtime: docker-runc
> Security Options: seccomp
> Kernel Version: 3.10.0-514.10.2.el7.x86_64
> Operating System: CentOS Linux 7 (Core)
> OSType: linux
> Architecture: x86_64
> Number of Docker Hooks: 2
> CPUs: 2
> Total Memory: 3.86 GiB
> Name: xxxx
> ID: 4HCD:CXPT:UOD3:TH6S:6CZ3:XOEV:FYIF:GHBZ:OCIR:JBTX:UVG6:G37X
> Docker Root Dir: /var/lib/docker
> Debug Mode (client): false
> Debug Mode (server): false
> Registry: https://index.docker.io/v1/
> Insecure Registries:
>  127.0.0.0/8
> Registries: docker.io (secure)

(Matt Palmer) #18

Devicemapper in general was a dumpster fire for a long, long time. It may be that these days dm+loop is still bad but dm+lvm is OK, but… once bitten and all that.


(Mark Moorcroft) #19

Yesterday I had to drop to command line to update Discourse. This was the first time seeing the need to start using --ignore-prereqs. Will the web based updater fail from now on?


(Rafael dos Santos Silva) #20

The web based updated will ask for a command line rebuild every time we make a change that depends on new OS level software, like updated nginx, ruby, postgresql, etc.

We upgrade database every two years, and ruby almost every year, for reference.