Why is docker devicemapper taking so much hard drive space?


(Mykhailo Poliarush) #1

Why is devicemapper taking 14Gb? and How to reduce it?

root@hosting:/var/lib/docker/devicemapper/devicemapper# ls -lah
total 14G
drwx------ 2 root root 4.0K Oct 17  2014 .
drwx------ 5 root root 4.0K Oct 17  2014 ..
-rw------- 1 root root 100G Oct 11 20:30 data
-rw------- 1 root root 2.0G Oct 11 20:30 metadata

I’ve done

cd /var/discourse/
git status
git pull
./launcher rebuild webprod

and i see that seems old docker images was removed

25e8d2ab4faa405038d121f53b46c581fb04078b5610e697bcffcdcf3321203f
e6b437d1ff501bb2559657d4a093bd9741488ab751ad503720089129f3c6417f
Removing old container
+ /usr/bin/docker rm webprod
webprod

Also i run

root@hosting:/var/lib/docker/devicemapper/devicemapper# docker info
Containers: 23
Images: 55
Storage Driver: devicemapper
 Pool Name: docker-202:0-760977-pool
 Pool Blocksize: 64 Kb
 Data file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
 Data Space Used: 13172.7 Mb
 Data Space Total: 102400.0 Mb
 Metadata Space Used: 10.6 Mb
 Metadata Space Total: 2048.0 Mb
Execution Driver: native-0.2
Kernel Version: 4.1.5-x86_64-linode61
Operating System: Ubuntu precise (12.04.2 LTS)
WARNING: No swap limit support

(Jeff Atwood) #2

Try running the cleanup command in launcher. Likely old images are still present.


(Mykhailo Poliarush) #4
root@hosting:/var/discourse# ./launcher cleanup

The following command will
- Delete all docker images for old containers
- Delete all stopped and orphan containers

Are you sure (Y/n): Y
Starting Cleanup (bytes free 69548108)
date: invalid date `"2014-10-20 09:18:3'
scripts/docker-gc: line 69: 1444598129 - : syntax error: operand expected (error token is "- ")
Finished Cleanup (bytes free 69548108)

any ideas where to fix this issue?


(Sander Datema) #5

Got the same kind of error:

Starting Cleanup (bytes free Beschikbaar
121149080)
date: ongeldige datum: ‘"2015-07-23 20:30:3’
scripts/docker-gc: regel 69: 1444651185 - : syntaxfout: operator verwacht (het onjuiste symbool is "- ")
Finished Cleanup (bytes free Beschikbaar
121149064)

(Mykhailo Poliarush) #6

there is a defect in parsing date in docker-gc script, it need to be fixed


(Mykhailo Poliarush) #7

I would like to manually delete docker images
how to find out what images can i remove? or can I remove all except of latest one on top?

root@hosting:/var/discourse# docker images -a
REPOSITORY                TAG                 IMAGE ID            CREATED              VIRTUAL SIZE
local_discourse/webprod   latest              25e8d2ab4faa        2 days ago           1.514 GB
<none>                    <none>              7ea991d02f7e        3 weeks ago          1.238 GB
samsaffron/discourse      1.0.13              27f52292c186        3 weeks ago          1.238 GB
<none>                    <none>              157f6a775410        3 weeks ago          1.238 GB
<none>                    <none>              65746d67224e        3 weeks ago          1.238 GB
<none>                    <none>              dd46ba35af06        3 weeks ago          821.1 MB
<none>                    <none>              e04f0be00b42        3 weeks ago          1.422 GB
<none>                    <none>              589a964a9895        5 weeks ago          1.418 GB
<none>                    <none>              d3a1f33e8a5a        7 weeks ago          188.2 MB
<none>                    <none>              4eb64bb76bf0        10 weeks ago         1.395 GB
<none>                    <none>              efa1f522a423        3 months ago         1.398 GB
<none>                    <none>              fc8cc7cde750        3 months ago         1.396 GB
<none>                    <none>              ed4ba02a6ede        3 months ago         1.396 GB
<none>                    <none>              63751254f0c9        3 months ago         1.396 GB
samsaffron/discourse      1.0.12              2285ff53ab18        4 months ago         1.217 GB
<none>                    <none>              4380a528f57f        4 months ago         1.217 GB
<none>                    <none>              69113ce854c6        4 months ago         1.217 GB
<none>                    <none>              01ec0f640267        4 months ago         1.217 GB
<none>                    <none>              56fe93484da0        4 months ago         821 MB
<none>                    <none>              689542ce25a4        5 months ago         1.403 GB
<none>                    <none>              2d8230e07a46        5 months ago         1.394 GB
<none>                    <none>              75f9a526253a        5 months ago         1.394 GB
<none>                    <none>              e9e06b06e14c        5 months ago         188.1 MB
<none>                    <none>              646645b49b57        6 months ago         1.333 GB
<none>                    <none>              032f4e3389f7        6 months ago         1.332 GB
local_discourse/app       latest              3e421b48558a        6 months ago         1.369 GB
<none>                    <none>              bcc32330fd43        8 months ago         1.273 GB
<none>                    <none>              1c91b4e84c80        9 months ago         1.204 GB
local_discourse/webstag   latest              8a6a78e362dc        10 months ago        1.215 GB
local_discourse/data      latest              fda96b10579c        11 months ago        1.072 GB
<none>                    <none>              814bb0e99fe9        11 months ago        1.16 GB
<none>                    <none>              0f90b76ce23b        12 months ago        1.035 GB
samsaffron/discourse      1.0.5               3a667f60316b        12 months ago        1.035 GB
<none>                    <none>              ff0911be7f52        12 months ago        1.035 GB
<none>                    <none>              bc4084da2f0f        12 months ago        1.035 GB
<none>                    <none>              75795f402bd5        12 months ago        690.6 MB
hello-world               latest              ef872312fe1b        12 months ago        910 B
<none>                    <none>              7fa0dcdc88de        12 months ago        910 B
samsaffron/nsenter        latest              8c777453fc88        15 months ago        3.384 MB
samsaffron/discourse      latest              2c6895c78d2a        16 months ago        1.11 GB
<none>                    <none>              51f67d00df48        16 months ago        1.071 GB
<none>                    <none>              ba805054bee0        16 months ago        787 MB
<none>                    <none>              7229a1cd6a96        16 months ago        787 MB
<none>                    <none>              943dda970ee3        16 months ago        787 MB
<none>                    <none>              fd22c18291de        16 months ago        787 MB
<none>                    <none>              0de77b78d083        16 months ago        787 MB
<none>                    <none>              4a010f0848e3        16 months ago        275.4 MB
<none>                    <none>              ad892dd21d60        16 months ago        275.4 MB
<none>                    <none>              9db365ecbcbb        16 months ago        192.7 MB
<none>                    <none>              23f361102fae        16 months ago        192.7 MB
<none>                    <none>              e465fff03bce        16 months ago        192.5 MB
<none>                    <none>              a9eb17255234        16 months ago        2.433 MB
<none>                    <none>              120e218dd395        16 months ago        2.433 MB
<none>                    <none>              42eed7f1bf2a        16 months ago        0 B
<none>                    <none>              511136ea3c5a        2.336223 years ago   0 B

(Kane York) #8

Everything except:

samsaffron/nsenter latest
samsaffron/discourse latest
samsaffron/discourse 1.0.13
local_discourse/webprod latest
local_discourse/webstag latest
local_discourse/data latest
local_discourse/app latest

Copy all the hex numbers and put them in a docker rmi command.


(Sam Saffron) #9

I updated the cleanup script a few days ago, is it still not working (pull latest first)


(Mykhailo Poliarush) #10

@sam defect is fixed (thanks) however problem is still there and cleanup doesn’t help to free up space

root@hosting:/var/discourse# ./launcher cleanup
The following command will
- Delete all docker images for old containers
- Delete all stopped and orphan containers

Are you sure (Y/n): Y
Starting Cleanup (bytes free 68698144)
Finished Cleanup (bytes free 68698148)

Still have a lot of images

root@hosting:/var/discourse# docker images -a
REPOSITORY                TAG                 IMAGE ID            CREATED              VIRTUAL SIZE
local_discourse/webprod   latest              25e8d2ab4faa        3 days ago           1.514 GB
<none>                    <none>              157f6a775410        3 weeks ago          1.238 GB
<none>                    <none>              27f52292c186        3 weeks ago          1.238 GB
<none>                    <none>              7ea991d02f7e        3 weeks ago          1.238 GB
<none>                    <none>              65746d67224e        3 weeks ago          1.238 GB
<none>                    <none>              dd46ba35af06        3 weeks ago          821.1 MB
<none>                    <none>              d3a1f33e8a5a        7 weeks ago          188.2 MB
local_discourse/webstag   latest              8a6a78e362dc        10 months ago        1.215 GB
local_discourse/data      latest              fda96b10579c        11 months ago        1.072 GB
<none>                    <none>              ff0911be7f52        12 months ago        1.035 GB
<none>                    <none>              0f90b76ce23b        12 months ago        1.035 GB
<none>                    <none>              3a667f60316b        12 months ago        1.035 GB
<none>                    <none>              bc4084da2f0f        12 months ago        1.035 GB
<none>                    <none>              75795f402bd5        12 months ago        690.6 MB
<none>                    <none>              511136ea3c5a        2.337060 years ago   0 B

and 14gb of space is allocated to devicemapper

root@hosting:/var/discourse# ls -la /var/lib/docker/devicemapper/devicemapper
total 13726844
drwx------ 2 root root         4096 Oct 17  2014 .
drwx------ 5 root root         4096 Oct 17  2014 ..
-rw------- 1 root root 107374182400 Oct 14 21:44 data
-rw------- 1 root root   2147483648 Oct 14 21:44 metadata

root@hosting:/var/discourse# docker info
Containers: 3
Images: 15
Storage Driver: devicemapper
 Pool Name: docker-202:0-760977-pool
 Pool Blocksize: 64 Kb
 Data file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
 Data Space Used: 4018.3 Mb
 Data Space Total: 102400.0 Mb
 Metadata Space Used: 3.5 Mb
 Metadata Space Total: 2048.0 Mb
Execution Driver: native-0.2
Kernel Version: 4.1.5-x86_64-linode61
Operating System: Ubuntu precise (12.04.2 LTS)
WARNING: No swap limit support

(Kane York) #11

Also, why are you using the devicemapper driver instead of aufs? devicemapper is known to have issues.

Ah, so the Linode kernels don’t have aufs support in by default, you’ll have to switch to a compiled kernel.

If you don’t, there will likely be pain later down the road.


Can't install Discourse with only 10 GB disk, run out of space
(Jeff Atwood) #12

Oh yeah, you will have guaranteed corruption with devicemapper and Docker, it is just a matter of time. We had a support customer blow up with this the other day. Sorry we did not catch that aspect of this question sooner.

We intentionally block install on devicemapper so I’m not sure how you even have Discourse running at this point.


(Allen - Watchman Monitoring) #13

I’m seeing this on a digitalocean box running ubuntu 16 and docker 17.05.0.

It was setup following the official instructions,

ncdu 1.11 ~ Use the arrow keys to navigate, press ? for help                                                                                                               
--- /var/lib/docker -------------------------------------------------------------------------------------------------------------------------------------------------------
                         /..
   17.6 GiB [##########] /devicemapper
   11.4 GiB [######    ] /aufs
   89.3 MiB [          ] /image

How can I confirm which storage driver I’m using?

What’s the best way to clean this all up?

(the launcher cleanup routine saved about 900MB when it ran)

root@forum:/var/discourse# ./launcher rebuild app
You have less than 5GB of free space on the disk where /var/lib/docker is located. You will need more space to continue
Filesystem                 Size  Used Avail Use% Mounted on
/dev/disk/by-label/DOROOT   40G   33G  4.8G  88% /

Would you like to attempt to recover space by cleaning docker images and containers in the system?(y/N)y
WARNING! This will remove:
	- all stopped containers
	- all volumes not used by at least one container
	- all networks not used by at least one container
	- all dangling images
Are you sure you want to continue? [y/N] y
Deleted Images:
deleted: sha256:3417b862eaf9fcc50680833dc22d6a8535ac95a63dce8654df74b25578e29ae5
deleted: sha256:eacfad7e966ab6d2de870d21d909032cb78ade5ca29a921779ff7c382cac2b96

Total reclaimed space: 888.5MB

(Allen - Watchman Monitoring) #14

Here’s where the space seems to be used, in files which weren’t touched since Febrary.

/# ls -al /var/lib/docker/devicemapper/devicemapper/
total 18465160
drwx------ 2 root root         4096 Feb  5  2016 .
drwx------ 5 root root         4096 Feb  5  2016 ..
-rw------- 1 root root 107374182400 Feb 12 23:44 data
-rw------- 1 root root   2147483648 Feb 12 22:43 metadata

Perhaps devicemanager used to be in use, but is no longer, if so, is it safe to

rm -rf /var/lib/docker/devicemapper

?


(David Taylor) #15

I had a similar problem, and deleting devicemapper didn’t cause me any issues. Would obviously recommend taking a full backup of everything first