Problems with docker and kernel on digital ocean


(Allen - Watchman Monitoring) #1

On the forum I use as a test bed, I just did some normal maintenance routines, and now I get this error during a rebuild

Removing old container
+ /usr/bin/docker rm app
Error response from daemon: Driver aufs failed to remove root filesystem b610111d39e8634398cd28b510bf362b09640e9111438503c12d22b3cd046950: rename /var/lib/docker/aufs/mnt/2b8f3dac5f82f25dfbc6a17d9c0fd70aca12cace46fc72be6090eb3554c99886 /var/lib/docker/aufs/mnt/2b8f3dac5f82f25dfbc6a17d9c0fd70aca12cace46fc72be6090eb3554c99886-removing: device or resource busy

starting up existing container
+ /usr/bin/docker start app
Error response from daemon: Container is marked for removal and cannot be started.
Error: failed to start containers: app

Prior to this, I ran apt-get upgrade, which netted Docker version 17.04.0-ce, build 4845c56

This is a digitalocean ubuntu 14.0.4 LTS droplet using the standard docker install.

History tells me there’s a problem with my kernel & image extra versions.

uname -r tells me 3.13.0-71-generic

I also see

:/var/discourse# dpkg --get-selections | grep linux-image
<snip>
linux-image-3.13.0-105-generic			deinstall
linux-image-3.13.0-106-generic			deinstall
linux-image-3.13.0-107-generic			deinstall
linux-image-3.13.0-108-generic			deinstall
linux-image-3.13.0-57-generic			deinstall
linux-image-3.13.0-61-generic			deinstall
linux-image-3.13.0-68-generic			deinstall
linux-image-3.13.0-71-generic			install
linux-image-3.13.0-74-generic			deinstall
linux-image-3.13.0-76-generic			deinstall
linux-image-3.13.0-77-generic			deinstall
<snip>

So, I guess I’m missing the one image that matches my kernel?

I guess my question here is… how best to avoid issues like this, where I’m dealing kernel mis-matches?

(Unless, of course, there’s some other issue like a known problem in docker, etc)

TIA


(Robby O'Connor) #2

Why not use a 16.04 image? Also you could give the output of uname -r – which would be immensely helpful here but I’d recommend just updating to 16.04 unless you have a reason to be on 14.04


(Allen - Watchman Monitoring) #3

This has been 14.04 since I installed it a while back… I could upgrade to 16, but will that stop the mysterious aufs issues which have come up from time to time?


(Robby O'Connor) #4

I ran discourse just fine on 14.04…i’m confused

Have you done: sudo apt-get install \ linux-image-extra-$(uname -r) \ linux-image-extra-virtual from the installation guide?


(Robby O'Connor) #5

Yup, it certainly will! I’ve been sysadmin for two discourse installs, so can vouch for it not sucking. Just ensure you follow the docker install instructions.


(Allen - Watchman Monitoring) #6

Well then… that’s reason enough to upgrade to ubuntu 16 :heart:

After the latest apt-get upgrade, the right 3.13.0-71 image-extra package was missing, hence the aufs error.

The command you linked to solved it for now, I’ll be making backups of all my discourse hosts & upgrading.


(Robby O'Connor) #7

You don’t need to upgrade now…you won’t have an issue…but you REALLY should at some point :slight_smile:


(Allen - Watchman Monitoring) #8

Upgrade now, when I’m ready for issues? Or when some random apt-get upgrade will stop the show?

Hahaha, thanks, this is plenty reason to start the process of upgrading.


(Robby O'Connor) #9

You didn’t install a package dependency, which is in the docker installation guide… that’s what caused your problem.


(Jeff Atwood) #10

This has always been an odd problem to me since the official install guide says to use the wget command from get.docker.com to install Docker. And I have never observed this problem on any install we did, which obviously follows the official guide slavishly.

wget -qO- https://get.docker.com/ | sh

So this implies people who do have this problem installed Docker in some oddball, non-install-guide way… somehow.


(Allen - Watchman Monitoring) #11

I dunno about the docker aspect… this instance has been running fine for over a year, through other docker upgrades even.

Today I did apt-get upgrade and I got fussed at about a lack of aufs. I don’t even think it was a docker issue, but a mismatch of kernel & image-extra. If ubuntu 16 resolves that issue, I’m going to put image-extra out of my life with an OS upgrade.


(Allen - Watchman Monitoring) #12

This is the start of this host’s .bash_history

sudo install -o root -g root -m 0600 /dev/null /swapfile
dd if=/dev/zero of=/swapfile bs=1k count=1024k
mkswap /swapfile
swapon /swapfile
echo "/swapfile       swap    swap    auto      0       0" | sudo tee -a /etc/fstab
sudo sysctl -w vm.swappiness=10
echo vm.swappiness = 10 | sudo tee -a /etc/sysctl.conf
wget -qO- https://get.docker.com/ | sh
mkdir /var/discourse
cd /var/discourse
git clone https://github.com/discourse/discourse_docker.git /var/discourse
ls -al
vi containers/app.yml
apt-get update

(Robby O'Connor) #13

Didn’t docker silently deprecate this method?


(Jeff Atwood) #14

Nope try it yourself and see.


(Allen - Watchman Monitoring) #15

Too bad it didn’t include a trigger to always add the kernel’s related image-extras package.


(Robby O'Connor) #16

It works – but it could stop working at any time is what I mean.


(Allen - Watchman Monitoring) #17

Soooo what’d they replace it with?


(Robby O'Connor) #18

this – I actually use ansible to install docker in almost all cases.


(Allen - Watchman Monitoring) #19

Does it help prevent aufs related failures after apt-get upgrades?


(Jeff Atwood) #20

Still something super hinky with this as we literally never saw this problem with hundreds of official guide based installs across a period of literally years. Did you add any “extra” stuff or steps?

(I agree it has been reported on meta a few times but I strongly suspect “extra” things were done…)