Abnormally high disk usage, 3 days after initial install

(Roy Petter Torgersen) #1

Hey, we seem to experience an extremely high storage usage:

System information as of Tue Jul 14 10:44:03 EDT 2015

System load: 0.0 Memory usage: 2% Processes: 60 
Usage of /: 71.1% of 39.25GB Swap usage: 0% Users logged in: 0

However, we have just recently (monday) installed a discourse app on the droplet, and it makes absolutely no sense that we should have spent 71% of the disk space already (I had to increase to capacity of the server (Digital Ocean) just to be able to run the site, and we have only got 5 users!
I posted a ticket to DO and run the following command as per their recommendation:

du / | sort -nr

That returned the following list:

30665576	/
29322808	/var
25564564	/var/discourse
25563676	/var/discourse/shared
25563672	/var/discourse/shared/standalone
25492036	/var/discourse/shared/standalone/log    
25490372        /var/discourse/shared/standalone/log/rails
3647708	/var/lib
3449284	/var/lib/docker

… and so on

I take it the culprit is the highlighted rails log folder. Before I go ahead and rm -r the folder contents, is there anything of importance in here?

(Jeff Atwood) #2

Quite odd, if this is a new install, you shouldn’t see that kind of disk usage.

Is this a new install or an old one that has been around for a while? I know in older Docker images there was an issue where logs weren’t being cleared properly.

Update: we measured actual new Discourse install disk usage in May and TL;DR it is about 3-4 GB.

(Sam Saffron) #3

Sounds to me like something is ill and your logs are bloating

I would look at the log file and checkout that error that is being repeated a million times before deleting it

(Roy Petter Torgersen) #4

@codinghorror, @sam, thanks for you feedback. I upgraded my DO to a 2GB droplet, which is what you guys recommend anyway. I deleted the log files under /var/discourse/shared/standalone/log/rails before I read you message Sam, so I’m afraid that train left already. However, my installation is still using 26GB, which makes very little sense to me; from what I have read on the forum, a 10GB install should be enough to run a forum.

System information as of Wed Jul 15 15:25:50 EDT 2015

  System load:  0.01               Processes:              111
  Usage of /:   63.7% of 39.25GB   Users logged in:        0
  Memory usage: 60%                IP address for eth0:
  Swap usage:   0%                 IP address for docker0:

Filesystem     1K-blocks     Used Available Use% Mounted on
/dev/vda1       41151808 26221116  12817272  68% /
none                   4        0         4   0% /sys/fs/cgroup
udev             1014128        4   1014124   1% /dev
tmpfs             205000      340    204660   1% /run
none                5120        0      5120   0% /run/lock
none             1024980      980   1024000   1% /run/shm
none              102400        0    102400   0% /run/user

Actually, the /var folder is only 4GB:

~# du --summ -k /var 
3919868	/var

I wonder if this is a problem with DO. I’ve found a couple of interesting posts in this forum after you pointed me in the right direction Jeff, I’ll work through those first and then get back to this thread with the results.

(Sam Saffron) #5

Run ncdu, find which directories are biggest

(Roy Petter Torgersen) #6

You guys are going to love this:

so I’ve been busy for the last few days and haven’t been looking at the
server. Tonight I decided to have a look at the system again to see how
much disk space has been used by now; whether it is still growing or if it
had stabilised. I got the following welcome screen when I logged on:

System information as of Fri Jul 17 18:02:47 EDT 2015

System load:  0.09               Processes:              115

Usage of /:   14.0% of 39.25GB   Users logged in:        0

Memory usage: 59%                IP address for eth0:    
Swap usage:   0%                 IP address for docker0: 

14% of 40 GB ties in with what you measured on a clean new install.
I swear I haven’t done a thing to the server. Seems like the docker elves
have been going at it on their own. Anyway, good news for yours truly.

By the way, thanks for making such a great piece of software. You guys

FYI, my setup is using a freshly spun digital ocean droplet located in FRA
and I followed your 30 minutes install instructions.

(Sam Saffron) #7

Getting these Docker :space_invader: to play nice with logrotated took me surprisingly long. I think it went through 4/5 releases where I was sure it worked perfect, yet another edge case.

As it stands we have custom logrotated retention rules and we use cron + anacron to orchestrated it.

I have huge amount of respect to communities who build Linux distros, this stuff is super fiddly.