Low on disk space, cleaning up old Docker containers


(John Arce) #1

I’m now getting 70% disk space

Usage of /:   70.4% of 39.25GB  

I tried this command du | sort -rn | head

Let me know your thoughts and how can I cleanup my disk.

john@discourse:/# du | sort -rn | head
30332308	.
26911916	./var
25919260	./var/lib
25651820	./var/lib/docker
25581776	./var/lib/docker/aufs
24242456	./var/lib/docker/aufs/diff
1339072	./var/lib/docker/aufs/mnt
1338828 ./var/lib/docker/aufs/mnt/4bd7749b00a950fd265d031cfb48dc173aa526e8aba6035f845fdca975025c5a
947748	./usr
712200	./var/lib/docker/aufs/diff/bf95185ac9b9af59b5a63966ee56fa38ce9ac0dabebad0eb63e59c2de46d16d3

/var/lib/docker/aufs/diff increasing in size
Discourse Taking too Much Space on Every Upgrade
Please help: My Discourse got really slow due to Topic Preview plugin
Error while rebuilding app
Scheduled database backups cause periodic out of memory errors?
(Sam Saffron) #2

This will reliably clean up old Docker images, as well as any old Ubuntu updates:

apt-get autoclean
apt-get autoremove
cd /var/discourse
./launcher cleanup

I just fixed up cleanup so it is a lot less random, it uses docker-gc by spotify which has much less problems.


Many old Docker images to clean up / delete?
How to free up disk space on my Digital Ocean server?
Configure automatic backups for Discourse
Digital Ocean hosting: Do I need a system admin?
Disk full due to too many old daily backup files with S3 partially configured
Disk full due to too many old daily backup files with S3 partially configured
(Jeff Atwood) #33

Strongly related to this, I find that Ubuntu disk space is eaten up heavily by keeping old kernels around. Here’s a Ubuntu 14.04 LTS DO droplet from fall 2014:

Before:

Filesystem     1M-blocks  Used Available Use% Mounted on
/dev/vda           30110 16428     12131  58% /

Do this:

apt-get autoclean
apt-get autoremove

(this can take a while, maybe 5 minutes? it has to iterate through all the old kernels and remove all of them.)

After:

Filesystem     1M-blocks  Used Available Use% Mounted on
/dev/vda           30110 10530     18029  37% /

That removes 6GB of just old archived Linux kernels! At the end of this, it’s very clean, all the disk space is taken by

  1. /var/lib/docker/aufs 7GB
  2. swapfile (recommended 2GB on a 1GB RAM droplet)

(Jeff Atwood) #38

12 posts were split to a new topic: Disk full due to too many old daily backup files


(Jeff Atwood) #39

On a brand new clean install of Discourse on a 20GB Digital Ocean droplet, I see this disk usage:

Filesystem     1M-blocks  Used Available Use% Mounted on
/dev/vda1          30108  6079     22478  22% /
none                   1     0         1   0% /sys/fs/cgroup
udev                 487     1       487   1% /dev
tmpfs                100     1       100   1% /run
none                   5     0         5   0% /run/lock
none                 497     1       497   1% /run/shm
none                 100     0       100   0% /run/user

So consider that (4GB – includes 2GB of swapfile in this case) the absolute minimum amount of disk space Discourse will use


Rebuild fill up disk space
Disk full due to too many old daily backup files with S3 partially configured
(Tom Newsom) #44

./launcher cleanup

Is this safe to run when my discourse isn’t running?
I’ve run out of storage during a rebuild app and don’t want to accidentally hose my install.

Running on a DO droplet

FTR:

root@discourse:/var/discourse# du -Sx | sort -rn | head -n 10
519656  ./shared/standalone/uploads/default/original/1X
288832  ./shared/standalone/postgres_data/base/16384
74636   ./shared/standalone/uploads/default/optimized/1X
65540   ./shared/standalone/postgres_data/pg_xlog
27608   ./shared/standalone/redis_data
23128   ./shared/standalone/uploads/default/original/2X/6
20028   ./shared/standalone/log/var-log/nginx
19520   ./shared/standalone/uploads/default/original/2X/0
18984   ./shared/standalone/uploads/default/original/2X/8
17364   ./shared/standalone/uploads/default/original/2X/c

and

root@discourse:/# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        30G   28G  528M  99% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
udev            487M  4.0K  487M   1% /dev
tmpfs           100M  328K  100M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            498M     0  498M   0% /run/shm
none            100M     0  100M   0% /run/user

I’ve deleted all but one backup. I have quite a lot, but not TONS of uploads. What’s eating up that 28GB?


Site down due to lack of disk space
(Jeff Atwood) #45

yeah launcher cleanup is totally safe, go for it


(Tom Newsom) #46

22GB released O_O

now rebuilding :slight_smile:

and we’re back :slight_smile:

This sort of cleanup needs to be automated no? I wasn’t doing anything too crazy; half a dozen plugins, keeping up to date with latest beta, and accumulated cruft at a rate of 2GB per month…


Disk full due to too many old daily backup files with S3 partially configured
(Panteen Pro-V) #47

I think it should be automated. Can we implement something like this for /admin/upgrade page?


(Jeff Atwood) #48

There was recently a huge bug with tagging where the SQL queries would be 1mb of text, each, exploding the logs. Were you using tagging?


(Tom Newsom) #49

Not a massive amount, but yes. When I get to a computer, I’ll see what the disk space is like now…


(Vishalvsh1) #61

There are ways to clean up docker images and reclaim the disk space.

Above is the details on same.


(Equivalent) #62

As this is regular issue I’m solving I’ve wrote article with lot of nice copy-paste code examples :slight_smile: Spring Cleaning for WebDevelopers


(EW 👌) #63

Many thanks @sam. I just cleaned up over 22GB :confused::+1:


(Lars) #64

I just deleted 21GB worth of logs in /var/discourse/shared/standalone/log/var-log/nginx - maybe say something about that somewhere :smiley:


(Jeff Atwood) #65

Isn’t that an artifact of an old image @sam?


(Sam Saffron) #66

Most likely.

@daath when is the last time you did?

cd /var/discourse
git pull
./launcher rebuild app

(Lars) #67

I don’t think it was that long ago - Is there a way to tell how old it is?


(Sam Saffron) #68

just run it again, then watch for a few weeks for bloat around logs, we rotate daily per:

edit confirming this on our various instances now…

rails logs are definitely rotated, nginx may have some issues


(Sam Saffron) #69

OK one caveat … make sure these lines are in your template… otherwise there be dragons :dragon_face: