/var/lib/docker/aufs/diff increasing in size


(Chris) #1

Hi,

My /var/lib/docker/aufs/diff directory is increasing in size. Now at 6.4G
The whole discourse site backup only weighs 66M

I wondering what I can do to keep that in check.

Thanks.
Chris.


(Sam Saffron) #2

You running latest discourse docker ?

cd /var/discourse
git pull
./launcher rebuild app

(Chris) #3

Dashboard had not tell me to update docker.

Ok. Done. (git pull did retrieve some changes.)
The site is online and /var/lib/docker/aufs/diff now weighs 6.8G
At this rate I’ll soon be out of disk space! :stuck_out_tongue:

Maybe this is related?

Cheers.


(Kane York) #4

Try ./launcher cleanup


(Chris) #5

That got /var/lib/docker/aufs/diff down to 4.4G
Does that seem reasonable?

Chris.


(Lee_Ars) #6

Yeah, even after running ./launcher cleanup, I’ve got damn near 20GB of stuff hanging out in /var/lib/docker/aufs, most of which is in diff. Can I nuke this, @sam? I need the extra space back in kind of a semi-emergency way.

[18:18:02] root@frylock:/var/lib/docker #  du -h -d 1 |sort -h -r
19G    ./aufs
19G    .
257M    ./init
1.8M    ./containers
1.2M    ./graph
116K    ./execdriver
84K    ./volumes
8.0K    ./apparmor
4.0K    ./trust
4.0K    ./tmp

[18:18:07] root@frylock:/var/lib/docker # cd aufs

[18:18:35] root@frylock:/var/lib/docker/aufs #  du -h -d 1 |sort -h -r
19G    .
17G    ./diff
1.5G    ./mnt
384K    ./layers

(Sam Saffron) #7

something is odd about your setup, seeing that with the logs as well, what does docker images return when you run it? What version of docker are you running?


(Lee_Ars) #8
[18:25:51] root@frylock:/var/cache # docker images
REPOSITORY             TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
local_discourse/app    latest              fc389284cd74        9 days ago          1.751 GB
samsaffron/discourse   1.0.12              2285ff53ab18        8 weeks ago         1.217 GB
samsaffron/discourse   1.0.11              ef8d4f405f84        3 months ago        1.167 GB
samsaffron/discourse   1.0.10              9d8d8cab6c87        3 months ago        1.166 GB
samsaffron/discourse   1.0.9               27769590d1d5        3 months ago        1.131 GB
samsaffron/discourse   1.0.8               2a29f41bee3a        4 months ago        1.143 GB
samsaffron/discourse   1.0.7               22d62951587e        7 months ago        1.121 GB
samsaffron/discourse   1.0.6               818beea55263        8 months ago        1.038 GB
samsaffron/discourse   1.0.4               2e3b522b59c5        10 months ago       1.034 GB
samsaffron/discourse   1.0.3               3166dccadd75        11 months ago       1.141 GB
samsaffron/discourse   0.2.5               0d5ccfb61810        13 months ago       1.164 GB
samsaffron/discourse   0.2.4               2c6895c78d2a        13 months ago       1.127 GB
samsaffron/discourse   0.2.1               b06b7d939b10        14 months ago       1.128 GB
samsaffron/discourse   0.2.0               610a44f81f75        15 months ago       1.124 GB
samsaffron/discourse   latest              c1299416c46a        19 months ago       779 MB
samsaffron/discourse   0.1.1               e1bbadbe2e16        20 months ago       785 MB

[18:25:53] root@frylock:/var/cache # docker -v
Docker version 1.5.0, build a8a31ef

Is there a newer version I should upgrade to?


(Sam Saffron) #9

well then you have much ancientness there.

did ./launcher cleanup not cleanup all those old images … you only need 1.0.12, you should be on latest docker but 1.5 is fine.


(Lee_Ars) #10

No, it doesn’t appear to do anything:

[18:29:35] root@frylock:/var/discourse # ./launcher cleanup

The following command will
- Delete all docker images for old containers
- Delete all stopped and orphan containers

Are you sure (Y/n): 

Starting Cleanup
31287048
31287012
./launcher cleanup cleared up 36 of disk space.

[18:29:42] root@frylock:/var/discourse # docker images
REPOSITORY             TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
local_discourse/app    latest              fc389284cd74        9 days ago          1.751 GB
samsaffron/discourse   1.0.12              2285ff53ab18        8 weeks ago         1.217 GB
samsaffron/discourse   1.0.11              ef8d4f405f84        3 months ago        1.167 GB
samsaffron/discourse   1.0.10              9d8d8cab6c87        3 months ago        1.166 GB
samsaffron/discourse   1.0.9               27769590d1d5        3 months ago        1.131 GB
samsaffron/discourse   1.0.8               2a29f41bee3a        4 months ago        1.143 GB
samsaffron/discourse   1.0.7               22d62951587e        7 months ago        1.121 GB
samsaffron/discourse   1.0.6               818beea55263        8 months ago        1.038 GB
samsaffron/discourse   1.0.4               2e3b522b59c5        10 months ago       1.034 GB
samsaffron/discourse   1.0.3               3166dccadd75        11 months ago       1.141 GB
samsaffron/discourse   0.2.5               0d5ccfb61810        13 months ago       1.164 GB
samsaffron/discourse   0.2.4               2c6895c78d2a        13 months ago       1.127 GB
samsaffron/discourse   0.2.1               b06b7d939b10        14 months ago       1.128 GB
samsaffron/discourse   0.2.0               610a44f81f75        15 months ago       1.124 GB
samsaffron/discourse   latest              c1299416c46a        19 months ago       779 MB
samsaffron/discourse   0.1.1               e1bbadbe2e16        20 months ago       785 MB

(Sam Saffron) #11

what does docker ps -a return … can you nuke all the stopped containers that are no longer relevant. (docker rm command)


(Lee_Ars) #12
[18:31:07] root@frylock:/var/discourse # docker ps -a
CONTAINER ID        IMAGE                        COMMAND             CREATED             STATUS              PORTS                                        NAMES
658a28784e29        local_discourse/app:latest   "/sbin/boot"        9 days ago          Up 40 minutes       0.0.0.0:2222->22/tcp, 0.0.0.0:8089->80/tcp   app  

Stopped Discourse/docker, rm -rf’d /var/lib/docker/aufs/diff/*, then rebuilt the container. All seems well.

Is there a PPA you recommend for the latest Docker? I’m just using the lxc-docker package from whatever repo version comes standard in Ubuntu 14.04.


(Sam Saffron) #13

honestly 1.5 is ok, I usually use the docker official repo linked from their doco.

I would not just nuke diff, you have leftover stuff (logs etc) … this may cause some unknown issues … if you insist on nuking I recommend.

  1. Stopping docker service
  2. Junking everything in /var/lib/docker (all dirs/data etc)
  3. Starting docker service
  4. ./launcher rebuild app

Discourse isn't starting
(Lee_Ars) #14

Gotcha. Nuked & paved, and rebuilding now.

I’m not seeing a ppa on the Docker ubuntu installation instructions page—they seem to recommend downloading the app without using packages, and down that path lies madness IME :smiley:

Maybe I’m looking in the wrong place.


(Sam Saffron) #15
$ wget -qO- https://get.docker.com/ | sh

That is a pretty fancy script that simply adds the apt repo :slight_smile:


(Lee_Ars) #16

Sweet. Docker upgraded to 1.7.1, container rebuilt, disk space is now plentiful. Problems solved! U da man, @sam :slight_smile:

Edit - Quick recap for anyone entering the thread late, I fixed the issue by doing a launcher destroy app, then service docker stop, then rm -rf /var/lib/docker/*, then service docker start and then launcher rebuild app. Also upgraded Docker to 1.7.1 by adding the Docker folks’ 3rd party PPA.


(Chris) #17

Hi,

root@agora:~# docker images
REPOSITORY             TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
local_discourse/app    latest              65d8cb5a5ce7        19 hours ago        1.788 GB
samsaffron/discourse   1.0.12              2285ff53ab18        8 weeks ago         1.217 GB
samsaffron/discourse   1.0.7               22d62951587e        7 months ago        1.121 GB
samsaffron/discourse   1.0.3               3166dccadd75        11 months ago       1.141 GB
hello-world            latest              565a9d68a73f        12 months ago       922 B
root@agora:~# docker -v
Docker version 1.2.0, build fa7b24f

I seems my docker version is outdated.

Is this the way to do it?

And then what?

Thanks.


(Kane York) #18

You can also do docker rmi 3166dccadd75 22d62951587e and see if that frees up some space, then use apt-get install lxc-docker to update Docker.


(Chris) #19

Yes, that freed up space.
/var/lib/docker/aufs/diff now at 1.6G

And docker version updated to 1.7.1, build 786b29d

Thank you @riking


(Kane York) #20

Okay then, @sam the issue is that it’s not deleting old base images