Docker shim error on CentOS 7


(James Mc Mahon) #1

At the moment I’m getting message
"no response from daemon: shim error; docker.runc not installed on system".
I tried a possible solution for that to no avail

Earlier today I installed telnet (which I guess updated yum?) to troubleshoot an email issue. Right now, I’m kinda hosed. I’m on Centos 7 I believe. Initially docker would not start, stated a problem with the network bridge. Removed and installed docker from yum to get that working again.

Here’s my docker.info at present:

Containers = 0 kinda concerns me, although I did make a backup of my docker directory before messing around. I did run docker rm app at one point (After the error about docker-runc occured)… maybe not such a good idea.

Attempt to rebuild results in same error message regarding docker-runc


(Rafael dos Santos Silva) #2

To install Docker CE, you need a maintained version of CentOS 7. Archived versions aren’t supported or tested.


(James Mc Mahon) #3

Thanks Rafael, I don’t fully understand but will follow the instructions for installing Docker CE


(James Mc Mahon) #4

Ok I’ve installed Docker-CE from Yun (After fully removing docker components per the guide you linked).
Docker fails to start

Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)

Doing some digging to see what might cause this.


(James Mc Mahon) #5

Ugh, I can’t figure out why docker-CE won’t start on this server. It installed without issue.
I even deleted /var/lib/docker reinstalled docker, no change.


(Jay Pfaffman) #6

For some reason, Docker doesn’t seem to play well with CentOS. Most of us here use Ubuntu, so don’t have solutions off the top of our heads. If you’re stuck and just want someone to fix it, I’ll figure out something. You can place an order here: Rebuild Your Droplet — Literate Computing . I’ll just fix what you’ve got rather than re-installing the OS as described there, though if you’d be willing to switch to Ubuntu, I could switch you over for the same price if Discourse is all that’s running there.


(James Mc Mahon) #7

That’s what I’m finding. I think my host (Interserver) might have only offered Centos at the time so I went with it. They have ubuntu now, spinning an ubuntu box here as I type.


(Jay Pfaffman) #8

As long as you’ve got a recent backup, you should be good to go. Actually, you could even just rsync /var/discourse to the new server and rebuild.


(James Mc Mahon) #9

I have back up from yesterday on S3. But rsync would probably be better so I have all the content until things went south.


(Michael Friedrich) #10

If systemd says “disabled”, you have multiple options

  • systemctl enable docker
  • SELinux is causing fun
  • this is a meta unit which is disabled. The real service is named differently. You can see that for example with Elasticsearch, where the service unit files are named differently for each cluster instance.

Let’s do this in a CentOS Vagrant VM (actually, it is icinga-vagrant/icinga2x with a stopped httpd)

yum install -y yum-utils \
  device-mapper-persistent-data \
  lvm2

yum-config-manager \
    --add-repo \
    https://download.docker.com/linux/centos/docker-ce.repo

yum -y install docker-ce

systemctl enable docker.service
systemctl start docker.service

git clone https://github.com/discourse/discourse_docker /var/discourse
cd /var/discourse

./discourse-setup
<enter all the time, let it fail at the end with the defaults, I have special requirements here>

Bootstrap afterwards works then.

[root@icinga2 discourse]# ./launcher bootstrap app
which: no docker.io in (/usr/local/sbin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin:/root/bin)

WARNING: We are about to start downloading the Discourse base image
This process may take anywhere between a few minutes to an hour, depending on your network speed

Please be patient

This also works for me with setenforce 1 as the Docker package installs an SELinux policy.

So this somehow seems to be related to the missing enable/start step on CentOS. No need for a migration run to Ubuntu imho.


(James Mc Mahon) #11

Thanks. So removed an installed docker-ce again per the guide above, ran “systemctl enable docker.service” but it still fails to start when I try to start it from systemctrl

I’m getting a little more information this time under journalctrl… "Failed at step EXEC spawning usr/bin/docker-current: no such file or directory.


(Michael Friedrich) #12

I’d appreciate copy pasted logs, so everyone can copy them into Google too. Screenshots are really hard to re-type.

Please run docker -v to show the current version. To me it seems you still have the old system’s docker installed side-by-side (the linked setup docs tell you how to remove that beforehand).


(James Mc Mahon) #13

I understand. The (web) VNC client I’m using doesnt seem to support copy and paste and my local VNC client doesnt seem to connect… Ill try again though

Docker version displays as 17.12.0-ce, build c97c6d6


(Michael Friedrich) #14

Hmmm. It seems that docker is actually a script which performs a lookup for possible paths. But only on RHEL/CentOS 7 where this comes from the RHEL repositories.

I don’t have /usr/bin/docker-current here.

[root@icinga2 ~]# docker -v
Docker version 17.12.0-ce, build c97c6d6
[root@icinga2 ~]# ps aux | grep docker
root     20688  0.9  1.9 542268 35964 ?        Ssl  20:56   0:37 /usr/bin/dockerd
root     20693  0.7  1.2 366356 22872 ?        Ssl  20:56   0:29 docker-containerd --config /var/run/docker/containerd/containerd.toml
root     29081  0.0  0.0 112668   944 pts/0    S+   22:05   0:00 grep --color=auto docker

Systemd starts dockerd here, you can see that from the service file in /lib/systemd/system/docker.service:

[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target

[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd
ExecReload=/bin/kill -s HUP $MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
#TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
# restart the docker process if it exits prematurely
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s

[Install]
WantedBy=multi-user.target

Do you have any other occurrences of a Systemd service file for docker, e.g. in /etc/systemd?

Anyhow, I’d suggest to start “fresh”. Purge anything which has docker in its name from the system

yum remove docker \
                  docker-common \
                  docker-selinux \
                  docker-engine

find /etc/systemd -name '*docker*' -exec rm -f {} \;
find /lib/systemd -name '*docker*' -exec rm -f {} \;

and go for installing it using this guide again.


(James Mc Mahon) #15

This appears to be the key. I did go though and remove as many docker-xxx packages as I could think of earlier, and it made no difference.

This time, docker installed and started after I ran the find lines.
Discourse is updating it’s base now… We’ll she

I was (still am) pondering moving to ubuntu purely because my exisitng setup on centos seems to be very fragile. Anytime I try to update discourse, I end up with a broken container, or a docker that won’t start.


(Jay Pfaffman) #16

If I’m reading this thread correctly, I’m pretty sure you’d have spent much less time if you’d moved to Ubuntu.


(James Mc Mahon) #17

Probably :smile: Although I would had to move all my data across and I’m not sure how involved that would be.

I was doing both really. Alternating between working on ubuntu and this centos install. Hit a snag with docker under ubuntu on the same hosting. Went through signup process with another host, docker and discourse was simple, but email with sendgrid is not co-operating for some reason

Thanks everyone for your help.


(Jay Pfaffman) #18

Some mail service I dealt with lately restricted access by IP so I had to provide the discourse server address to the mail service.


(Michael Friedrich) #19

I’m a friend of solving immediate issues, they give you some experience with debugging things, and also the obligatory “yes, I solved it” feeling.

I see that in the monitoring community a lot. Problem X is there, doesn’t work. Another user jumps in and suggests to replace it with project Y. Keeping a question with a discussion and a possible solution helps many others who find the topic years after with Google. I do see my knowledge input as investment into the future, not to answer that again and again.

Still, if many community members choose Ubuntu LTS, go for it. You are now in the first steps where such a decision can be made easily. At NETWAYS, we’re also going the Ubuntu route these days, and my Discourse instance runs on 14.04 at the moment. dist-upgrade to 16.04 is planned, 18.04 will be out this year too.

The good thing about the container image is that you don’t have to care about Ruby, gem, etc. version dependencies. In terms of security aspects in such an AIO package, I do have the feeling that the Discourse team acts responsible with announcing and forcing users to upgrade on possible CVEs and security bugs.

Cheers,
Michael


(James Mc Mahon) #20

True, and sendgrid is one that offers this. I wasn’t seeing any requests hit sendgrid (It will show attempts from IPs not on the approved list in the dashboard), yet was able to telnet to the smtp server, so it wasn’t a firewall issue either.

In the end, started over with a separate account sendgrid account, a fresh API key and it’s happy now.