Disk space usage of /lib/docker is high

AI-generated summary

User david reports high disk usage on a 30GB droplet, with /lib/docker consuming 18.2GB (9.7GB in overlay, 8.1GB in devicemapper). He questions whether 9.7GB for overlay is normal and whether it’s safe to delete the obsolete devicemapper folder after switching storage drivers. Falco suggests docker system prune, but david finds it ineffective, reclaiming 0B despite showing 1.7GB reclaimable images. He manually deletes /var/lib/docker/devicemapper without issues [4]. sam questions if du is double-counting layered data; david confirms du handles hard links correctly, ruling out duplication as the cause. Others share similar experiences: angus links to a fix involving Docker cleanup, and blau details a multi-step cleanup (reinstalling Docker, running launcher cleanup, and docker system prune) that recovered 5.4GB. sam suggests nuking /var/lib/docker entirely as a last resort. Dev_Work asks for a plugin to monitor disk space and email alerts — no solution provided. The thread highlights Docker storage bloat as a common issue, especially after rebuilds or driver changes, and offers manual cleanup as the most reliable fix.

Ran into a disk space issue on one of my droplets this morning. Discourse has almost filled the 30GB

Using ncdu, I’m getting

/discourse      6.4GB
/lib/docker    18.2GB

There are a fair number of posts & uploads, so I think the /discourse number is as expected. Drilling down into /lib/docker:

/lib/docker/overlay       9.7GB
/lib/docker/devicemapper  8.1GB

So, two questions:

  • Is 9.7GB an “normal” level of usage for the overlay folder?
  • Now that I moved from devicemapper to overlay, does anyone know if it’s safe to delete the devicemapper folder? Or do I need to run a docker command to clean it up somehow?

If you are on Docker 17.x a docker system prune should be safe. (Take a backup before, just in case)

sudo docker system prune

returns

Total reclaimed space: 0B

sudo docker system df

gives me

TYPE                TOTAL               ACTIVE              SIZE                RECLAIMABLE
Images              4                   2                   2.181GB             1.718GB (78%)
Containers          2                   2                   3.933GB             0B (0%)
Local Volumes       0                   0                   0B                  0B

So looks like it’s lying about “reclaimable”… But even if it could cleanup 1.7GB, that’s not the whole of devicemapper.

I did an rm -rf on /var/lib/docker/devicemapper and nothing exploded, even with a reboot. So hopefully that’ll be ok.

Still interested to know whether 9.7GB is a ‘normal’ level of overlay usage?

Are you certain that is actual usage and not counting layers multiple time despite small delta?

Ooh I hadn’t thought of that. I just used du - what’s the best way to determine whether it’s counting things twice or not?

No idea, this is just me guessing :vampire:t4:

Looking at the man page, straight du is clever about hard links

$ du -sch overlay
9.7G	overlay
9.7G	total

With -l it counts every occurrence of the hard link as a unique file, so that gives an even higher reading:

$ du -schl overlay
15G	overlay
15G	total

So I don’t think the 9.7GB is caused by the layers - unless there’s some method other than hard links that could be tripping me up.

I encountered a similar issue; perhaps the same, it’s hard to tell. This is what fixed it (after a fashion) for me:

I was low on disk space and could not upgrade discourse nor free up space. Turns out /var/lib/docker/aufs/diff was the culprit. This is what worked for me:

  • upgrade system software, autoremove, autoclean and reboot to new kernel;
  • reinstall docker via get.docker.io
  • /var/discourse/launcher cleanup (recover 4GB space)
  • docker system prune (recover 1.4 GB)

I was on an old 17.x docker version and after several ./launcher rebuild app I had less than 5GB free on a 20 GB disk. Same issue as discussed here (thread closed):

Sometimes what I do is just junk /var/lib/docker altogether and start from scratch (stop docker and all containers first, we store nothing important there), perhaps try that.

Is there a plugin that will check the disk size and, if less than the specified value, notify me by email?