Docker disk usage issue c. Sept 1 2017

Installation
1

If you do a fresh install, as I did recently for my sandbox, you may, despite all the improvements in the cleanup scripts and docker itself, still encounter the dreaded disk usage creep issue; i.e. after rebuilding a number of times you see your disk usage go steadily upwards until the instance is no longer usable.

This issue has had a number of iterations (hence ‘circa Sept 1 2017’). The current iteration seems to be un-removed aufs/diff files.

Specifically, after a fresh Discourse install on a DO 1G Memory / 30G HD droplet, and running ./launcher rebuild app a few times my /var/lib/docker/aufs/diff folder looked like this

root@discourse:/var/discourse# sudo du -hd 1 /var/lib/docker/aufs/diff | sort -hrk 1 | head -20

19G	/var/lib/docker/aufs/diff
1.8G	/var/lib/docker/aufs/diff/4164965e95f8bbe50b181d96e92ca48658ce57edfeefa0143d484bb1dd894653
1.1G	/var/lib/docker/aufs/diff/d40d45c7274e8dea5a14f0705bdcf92cdf966b85d654e18f32e97c7bec0b20f3-removing
1.1G	/var/lib/docker/aufs/diff/ba1aae9ec74952b960f9286a7d76f6792054f055aaafb2c1855660be92b15c6d-removing
1.1G	/var/lib/docker/aufs/diff/b1c46d5904681264046779dfb1bef34659f592859481f679d33bf64a48f85363-removing
1.1G	/var/lib/docker/aufs/diff/a42be4d77fddbfd1e0b3fc4c8580db017413bffcc258dc9696186e38ceb6a9b9-removing
1.1G	/var/lib/docker/aufs/diff/7f4fffafb2d96f038e3b40739d602ab4c038de21effa163900a1a8987b301770-removing
1.1G	/var/lib/docker/aufs/diff/4e6c6a7c74c7a6df64f5852f44ee4fb01998d0271ed9b9f30fe1176269cc6f58-removing
1.1G	/var/lib/docker/aufs/diff/3dd6e740aee8af437d70a49fe5cfc73fc091dc976bb170629bd3a4a953a05786-removing
1.1G	/var/lib/docker/aufs/diff/1e93113fcc299cccd8d90c22e46b6c5a6f5b3fda14c046d0403cc604731abd6d
1020M	/var/lib/docker/aufs/diff/f7397e9843aa175361686a5edeb7c77ac5c6754bf5e4662d7f507ccb1f012a71-removing
1020M	/var/lib/docker/aufs/diff/c3647165501e41129687594b06b73acd8fd96f7959014f5c6ae3fa71caa13824-removing
1020M	/var/lib/docker/aufs/diff/9393f77e06db23cb200b465cf0baa38fb403ac9ae3d993ceca69a57cdc389bfd-removing
1020M	/var/lib/docker/aufs/diff/862f2e0fb2caf2317cc3146461d48d23f193e8c0f22a58eb2c1078ea7e903ffc-removing
1020M	/var/lib/docker/aufs/diff/0bd33c4272acaa9766982fd0f2251e0f9912b1d1a7cbc3b019362e0ac3a812b9-removing
1017M	/var/lib/docker/aufs/diff/18b727db320d769b3ec4be54552872fea051c14d40cb62bcaa74a71287f011b3-removing
1016M	/var/lib/docker/aufs/diff/8700c369592b80d5c1302a27fdb198581c218b2000d17e7c7a6b60137bd6f56f-removing
1014M	/var/lib/docker/aufs/diff/329ae8ff791d6b41409b2ba06fef593bdf92d2361ac0629e07478ea88aa2f00c-removing
182M	/var/lib/docker/aufs/diff/30b1731d18550c37e596b08a04a6896d0a777c0dfb40ff03c4008d1e7c7d71bc-removing
139M	/var/lib/docker/aufs/diff/cdbd9d560aa8ab7eb8d5c6a05414645c4c58d8805113550653a957a199ce2a45-removing

None of the following touched it:

  • ./launcher cleanup
  • Using Spotify’s docker-gc directly
  • Docker’s own docker system prune

After reading the relevant moby/moby issue for this and trying all the suggested solutions, the only one that seems to work reliably at this stage is the proverbial ‘nuclear’ option:

sudo service docker stop
rm -rf /var/lib/docker
sudo service docker start

This ‘fixes’ the issue insofar as you are able to use your instance again - i.e. rebuild the app - and your data remains intact (as far as I can tell so far).

cc @Falco @sam

7 Likes
4

Neither one of the people you pinged are available to look at this in the next 2 weeks.

5

That’s fine. I fixed it for my own purposes for now. I’m guessing others may encounter the issue and find this useful.

It’s not so much a Discourse issue as a Docker issue anyway.

6

I’m having the same issue. I’ll be watching this space for a better clean-up method in case one is posted.

I’m not sure why this issue arises now on multiple installations where it hasn’t been an issue I’ve seen previously.

7

It looks like this issue is much improved on the latest version of Docker. I haven’t had to rm -rf /var/lib/docker for a while.

If you encounter this issue make sure your docker version is 17.06.2-ce or later.

See further: https://github.com/moby/moby/issues/22207

8 Likes
8

I just did some tests and found that to get an accurate count of disk usage for /var/lib/docker, you need to pass the -x option to du so it doesn’t count files in mounted overlay file systems.

Stopping docker, which unmounts the overlay filesystems will also work but of course is less convenient then just using the -x option for du.

At least this is the case when using the overlay2 docker backend, I would assume it’s the same for most of the docker backends.

Edit: Strangely the sizes reported by running ‘docker images’ shows image ‘local_discourse/app’ with a size of 2.82GB and image ‘discourse/base’ with a size of 1.73GB.
Which would mean the images take 4.55 GB but running ‘du -sh /var/lib/docker’ shows 3.1GB total.
I’m not sure why there is a difference in size or why there are two images for discourse.
Maybe the images share files and the ‘docker images’ command doesn’t account for that.

2 Likes