Disk space usage of /lib/docker is high

Installation
1

Ran into a disk space issue on one of my droplets this morning. Discourse has almost filled the 30GB

Using ncdu, I’m getting

/discourse      6.4GB
/lib/docker    18.2GB

There are a fair number of posts & uploads, so I think the /discourse number is as expected. Drilling down into /lib/docker:

/lib/docker/overlay       9.7GB
/lib/docker/devicemapper  8.1GB

So, two questions:

  • Is 9.7GB an “normal” level of usage for the overlay folder?
  • Now that I moved from devicemapper to overlay, does anyone know if it’s safe to delete the devicemapper folder? Or do I need to run a docker command to clean it up somehow?
2

If you are on Docker 17.x a docker system prune should be safe. (Take a backup before, just in case)

1 „Gefällt mir“
3 
sudo docker system prune

returns

Total reclaimed space: 0B

sudo docker system df

gives me

TYPE                TOTAL               ACTIVE              SIZE                RECLAIMABLE
Images              4                   2                   2.181GB             1.718GB (78%)
Containers          2                   2                   3.933GB             0B (0%)
Local Volumes       0                   0                   0B                  0B

So looks like it’s lying about “reclaimable”… But even if it could cleanup 1.7GB, that’s not the whole of devicemapper.

4

I did an rm -rf on /var/lib/docker/devicemapper and nothing exploded, even with a reboot. So hopefully that’ll be ok.

Still interested to know whether 9.7GB is a ‘normal’ level of overlay usage?

2 „Gefällt mir“
5

Are you certain that is actual usage and not counting layers multiple time despite small delta?

1 „Gefällt mir“
6

Ooh I hadn’t thought of that. I just used du - what’s the best way to determine whether it’s counting things twice or not?

1 „Gefällt mir“
7

No idea, this is just me guessing :vampire:t4:

1 „Gefällt mir“
8

Looking at the man page, straight du is clever about hard links

$ du -sch overlay
9.7G	overlay
9.7G	total

With -l it counts every occurrence of the hard link as a unique file, so that gives an even higher reading:

$ du -schl overlay
15G	overlay
15G	total

So I don’t think the 9.7GB is caused by the layers - unless there’s some method other than hard links that could be tripping me up.

2 „Gefällt mir“
9

I encountered a similar issue; perhaps the same, it’s hard to tell. This is what fixed it (after a fashion) for me:

1 „Gefällt mir“
10

I was low on disk space and could not upgrade discourse nor free up space. Turns out /var/lib/docker/aufs/diff was the culprit. This is what worked for me:

  • upgrade system software, autoremove, autoclean and reboot to new kernel;
  • reinstall docker via get.docker.io
  • /var/discourse/launcher cleanup (recover 4GB space)
  • docker system prune (recover 1.4 GB)

I was on an old 17.x docker version and after several ./launcher rebuild app I had less than 5GB free on a 20 GB disk. Same issue as discussed here (thread closed):

2 „Gefällt mir“
11

Sometimes what I do is just junk /var/lib/docker altogether and start from scratch (stop docker and all containers first, we store nothing important there), perhaps try that.

4 „Gefällt mir“
12

Is there a plugin that will check the disk size and, if less than the specified value, notify me by email?