Ubuntu updates intefere with Docker and AUFS?

Support Self-hosting

AI-generated summary

Ubuntu Updates Interfere with Docker and AUFS

A discussion on the issues caused by Ubuntu updates interfering with Docker and AUFS. downey encountered an error when trying to destroy a container, and sam suggested that device mapper is not supported and AUFS needs to be used instead.

To recover from the issue, riking provided steps to manually remove leftover containers and images using docker rm/rmi. renoirb suggested ensuring that linux-image-extra-virtual and aufs-tools are installed, and adding DOCKER_OPTS="-s aufs" to /etc/default/docker.

codinghorror mentioned that updating Docker every few months might prevent the issue, and eschipul provided a solution to install linux-image-extra-$(uname -r) aufs-tools after a kernel upgrade.

The discussion concluded that installing linux-image-extra-virtual and ensuring AUFS is used can prevent the issue, and riking suggested adding this to the install docs.

1

This was going fine for me until the last moment, when I got the following super-useful error message:

Error response from daemon: Cannot destroy container 1cd705ffc49f5ce0622614316e3a742b8e6c8a77362528338db30a6f352477d2: Driver devicemapper failed to remove root filesystem 1cd705ffc49f5ce0622614316e3a742b8e6c8a77362528338db30a6f352477d2: Device is Busy
FATA[0014] Error: failed to remove one or more containers 
Invalid cid file, deleting, please re-run
  1. How to recover?
  2. How to make this more reliable and easier for users/admins?
2

Device mapper is not supported, please get aufs working.

You could probably recover by rebooting, but you are on your own here.

3

I’m using the standard “by the book” installation using the docs provided here and a Digital Ocean server, as recommended.

4

Nope, device mapper is not supported, never was. You MUST get aufs working.

5

I have no idea what either of those are. I only followed the official directions for the docker installation, and the only changes we’ve made were to add SSL & adding a few plugins.

6

Looks like I have the same problem reported here:

https://meta.discourse.org/t/discourse-did-not-restart-after-system-crash/24494/5?u=downey

So be careful installing normal updates to your OS, apparently it breaks Discourse. :stuck_out_tongue:

7

Supposedly docker 1.5 will make device mapper more reliable, on the fence on this cause I heard this many time before.

8

I am curious, how were you updating Ubuntu?

I tend to update Docker on mine every few months so perhaps that’s why I didn’t see any regression on kernel / aufs:

apt-get update
apt-get dist-upgrade lxc-docker
9

I think to recover, you’ll have to manually docker rm/rmi the leftovers, then rebuild.

Quick overview of that:

docker images -a
# identify useless images (old discourse base, untagged), then
docker rmi d25ec275fb1d 818beea55263 cdb5237bc8a7

docker ps -a
# identify dead/defunct containers, then
docker rm 7f32e8715c89
10

Just using apt-get upgrade and upgrading all packages every few weeks. As mentioned in the other thread, it’s hard to say when aufs went away, it was surely in a kernel upgrade at some point, and then this is likely the first docker image upgrade since then.

Yes, I had to follow the same steps as @danwad. I imagine we won’t be the only ones so I’m glad this is documented. :slight_smile:

11

Hi folks. I’ve hit that snag too.

Basically,

  1. ensure you have linux-image-extra-virtual and aufs-tools installed

  2. add to /etc/default/docker something like this

     DOCKER_OPTS="-s aufs"

I wrote a blog post about it here but the steps above is all you need. really.

12

Ok, I just did

apt-get update
apt-get upgrade
reboot

And nothing broke on my Digital Ocean instance. Site is still up: http://discourse.codinghorror.com

13

I haven’t noticed this problem lately but not sure when the last updates were run on the server. Will keep an eye on it and remember to come back here if we see any more problems.

14

Disappearing Dockers on Ubuntu after updates only happens when the kernel is upgraded. While your solution worked this time, it might not the next time given you had just done an update previously. This is the line you want to run specifically after a kernel upgrade.

sudo apt-get -y install linux-image-extra-$(uname -r) aufs-tools

or bundle them with your update with something similar to this:

sudo apt-get update && sudo apt-get upgrade -yy && \
sudo apt-get -y install linux-image-extra-$(uname -r) aufs-tools

Then you do have to reboot unfortunately. It happens on our Ubuntu servers all of the time. Worst case scenario it doesn’t find an update and just jumps past the line. Perhaps not perfect but effective and fast. I hope this helps!

15

What about installing the linux-image-extra-virtual metapackage so it stays updated, too?

Actually @codinghorror that should go in the install docs.

16

I just upgraded the lxc-docker package from 1.4.1 to 1.7.0 and was greeted with this afterwards:

Cannot connect to the Docker daemon. Is 'docker -d' running on this host?

Rebooting did not help. Trying to run docker manually I find out why:

$ sudo /usr/bin/docker -d
FATA[0000] Error starting daemon: error initializing graphdriver: "/var/lib/docker" contains other graphdrivers: devicemapper; Please cleanup or explicitly choose storage driver (-s <DRIVER>)

Added the line as suggested by @renoirb to /etc/default/docker and issued a sudo service docker restart and all was fine.

DOCKER_OPTS="-s aufs"

Not sure exactly why it was necessary but happy nonetheless.

17

Odd aufs is still priority 1

18

This just happened on my Linode (+ AUFS support) too. I had updated with apt-get dist-upgrade -y and rebooted.

Running docker daemon showed

[graphdriver] prior storage driver “aufs” failed: driver not supported

What fixed it was $ sudo apt-get install linux-image-extra-`uname -r` && sudo modprobe aufs as suggested by @danwad .