Problems with docker and kernel on digital ocean

installation
1

On the forum I use as a test bed, I just did some normal maintenance routines, and now I get this error during a rebuild

Removing old container
+ /usr/bin/docker rm app
Error response from daemon: Driver aufs failed to remove root filesystem b610111d39e8634398cd28b510bf362b09640e9111438503c12d22b3cd046950: rename /var/lib/docker/aufs/mnt/2b8f3dac5f82f25dfbc6a17d9c0fd70aca12cace46fc72be6090eb3554c99886 /var/lib/docker/aufs/mnt/2b8f3dac5f82f25dfbc6a17d9c0fd70aca12cace46fc72be6090eb3554c99886-removing: device or resource busy

starting up existing container
+ /usr/bin/docker start app
Error response from daemon: Container is marked for removal and cannot be started.
Error: failed to start containers: app

Prior to this, I ran apt-get upgrade, which netted Docker version 17.04.0-ce, build 4845c56

This is a digitalocean ubuntu 14.0.4 LTS droplet using the standard docker install.

History tells me there’s a problem with my kernel & image extra versions.

uname -r tells me 3.13.0-71-generic

I also see

:/var/discourse# dpkg --get-selections | grep linux-image
<snip>
linux-image-3.13.0-105-generic			deinstall
linux-image-3.13.0-106-generic			deinstall
linux-image-3.13.0-107-generic			deinstall
linux-image-3.13.0-108-generic			deinstall
linux-image-3.13.0-57-generic			deinstall
linux-image-3.13.0-61-generic			deinstall
linux-image-3.13.0-68-generic			deinstall
linux-image-3.13.0-71-generic			install
linux-image-3.13.0-74-generic			deinstall
linux-image-3.13.0-76-generic			deinstall
linux-image-3.13.0-77-generic			deinstall
<snip>

So, I guess I’m missing the one image that matches my kernel?

I guess my question here is… how best to avoid issues like this, where I’m dealing kernel mis-matches?

(Unless, of course, there’s some other issue like a known problem in docker, etc)

TIA

2

Why not use a 16.04 image? Also you could give the output of uname -r – which would be immensely helpful here but I’d recommend just updating to 16.04 unless you have a reason to be on 14.04

2 Likes
3

This has been 14.04 since I installed it a while back… I could upgrade to 16, but will that stop the mysterious aufs issues which have come up from time to time?

4

I ran discourse just fine on 14.04…i’m confused

Have you done: sudo apt-get install \ linux-image-extra-$(uname -r) \ linux-image-extra-virtual from the installation guide?

5

Yup, it certainly will! I’ve been sysadmin for two discourse installs, so can vouch for it not sucking. Just ensure you follow the docker install instructions.

1 Like
6

Well then… that’s reason enough to upgrade to ubuntu 16 :heart:

After the latest apt-get upgrade, the right 3.13.0-71 image-extra package was missing, hence the aufs error.

The command you linked to solved it for now, I’ll be making backups of all my discourse hosts & upgrading.

7

You don’t need to upgrade now…you won’t have an issue…but you REALLY should at some point :slight_smile:

3 Likes
8

Upgrade now, when I’m ready for issues? Or when some random apt-get upgrade will stop the show?

Hahaha, thanks, this is plenty reason to start the process of upgrading.

2 Likes
9

You didn’t install a package dependency, which is in the docker installation guide… that’s what caused your problem.

10

This has always been an odd problem to me since the official install guide says to use the wget command from get.docker.com to install Docker. And I have never observed this problem on any install we did, which obviously follows the official guide slavishly.

wget -qO- https://get.docker.com/ | sh

So this implies people who do have this problem installed Docker in some oddball, non-install-guide way… somehow.

11

I dunno about the docker aspect… this instance has been running fine for over a year, through other docker upgrades even.

Today I did apt-get upgrade and I got fussed at about a lack of aufs. I don’t even think it was a docker issue, but a mismatch of kernel & image-extra. If ubuntu 16 resolves that issue, I’m going to put image-extra out of my life with an OS upgrade.

12

This is the start of this host’s .bash_history

sudo install -o root -g root -m 0600 /dev/null /swapfile
dd if=/dev/zero of=/swapfile bs=1k count=1024k
mkswap /swapfile
swapon /swapfile
echo "/swapfile       swap    swap    auto      0       0" | sudo tee -a /etc/fstab
sudo sysctl -w vm.swappiness=10
echo vm.swappiness = 10 | sudo tee -a /etc/sysctl.conf
wget -qO- https://get.docker.com/ | sh
mkdir /var/discourse
cd /var/discourse
git clone https://github.com/discourse/discourse_docker.git /var/discourse
ls -al
vi containers/app.yml
apt-get update
2 Likes
13

Didn’t docker silently deprecate this method?

14

Nope try it yourself and see.

15

Too bad it didn’t include a trigger to always add the kernel’s related image-extras package.

16

It works – but it could stop working at any time is what I mean.

17

Soooo what’d they replace it with?

18

this – I actually use ansible to install docker in almost all cases.

19

Does it help prevent aufs related failures after apt-get upgrades?

20

Still something super hinky with this as we literally never saw this problem with hundreds of official guide based installs across a period of literally years. Did you add any “extra” stuff or steps?

(I agree it has been reported on meta a few times but I strongly suspect “extra” things were done…)