Our disk space disappeared - how to find who/where?

[fearlessfrog]

Our free disk space went down by about a gig overnight, or rather the backup image size jumped up.

We’re trying to figure out where it went, as we think a user may be uploading files to themselves via PM or something?

Where are good places to start looking in Discourse for this type of info, i.e. where the upload allocation went per user?

[marcospreviato (Marcos P)]

try this commands in your host:

sudo apt-get autoclean

sudo apt-get autoremove

cd /var/discourse

and after

sudo ./launcher cleanup

and press y for cleanup files!

[tgxworld (Alan Tan)]

Usually what I’ll do is to run

df -h and see which file system is taking up most of the disk space and then try to narrow down the scope with du -h with an optional --max-depth=1 to find the offending file/folder. Another tool that you might want to checkout is ncdu

[fearlessfrog]

Thank you - yep, we do that and it certainly does stop the bumps. I’ll double check for sure, but also did want to just see if there was a quota/report per user as well - especially as it was within the Discourse backup.tar.gz.

[fearlessfrog]

So @sam just to be super brief perhaps the simplest thing that would work for us is that I just do a

./launcher enter app
rails c 

…and then just use models for Upload (:filesize) and User and go irb query that way? I think that would get what we want.

EDIT: Yep, I just did it that way, i.e. user.uploads.sum(:filesize) &map. We found someone with 9GB of uploaded content in Discourse.

[Mittineague]

@fearlessfrog

9GB of uploaded content

9GB Whew!
Are your settings very high?

max image size kb
The maximum image upload size in kB. This must be configured in nginx (client_max_body_size) / apache or proxy as well.
max attachment size kb
The maximum attachment files upload size in kB. This must be configured in nginx (client_max_body_size) / apache or proxy as well.

Or is this a member doing a mess of smaller uploads?

[fearlessfrog]

@Mittineague

Or is this a member doing a mess of smaller uploads?

^ That. Our max image size was left at 3072 kb. It looks like just over a year of constant screen-shot uploading, but we are still figuring that out. It doesn’t look malicious, just really keen/dedicated and we are a very screenshot-centric forum. One tutorial topic alone had over 100 images uploaded. An interesting thing is the 2nd place uploader has only 0.6 GB used, so we’ll talk to who is doing it and figure something better out.

I wonder if the trust_levels could have a file/image quota added? I haven’t searched through the feature wishlist to find out as yet. This user is a level_3.

EDIT: I added a feature idea here, if others might have the need too:

User Upload Reporting & Quota

We had a recent need to find where our uploads size was increasing quickly, so this feature request breaks down into two ideas: (1) A way to see per user the file upload capacity used, i.e. a per user total uploads in mb. (2) A trust-based or per-user upload quota limit that would either warn staff/admin that a quota was exceeded or prevent further uploading until the quota was increased. Our support experience I am basing this request for feature discussion is here:

[fearlessfrog]

Ok, some new information the more we dig.

This looks like a Discourse bug somehow. The user with the 9 GB of files has the same 13 files that were uploaded as a set repeated 1728 times, i.e. 22,464 duplicate file uploads.

I’ll raise a bug and hopefully we can remove these plus stop it happening again.

Bug is reported here. Will welcome any help/advice

Duplicate Uploaded Files

Hi! We’re on 1.6.0.beta8 and recently experience a quick disappearance of our local server disk space, as detailed here: We’ve found out that one user has taken 9GB of uploads disk space, despite not uploading that many files (file limits kept to defaults etc). Looking at the upload::original_filename values it seems like we have a repeating set of 14 files duplicated 1729 times, i.e. 24,206 new files. The files themselves in that 14 file set are roughly the right size that when repeated wo…

[Mittineague]

@fearlessfrog

the same 13 files that were uploaded as a set repeated 1728 times

Hmmm, AFAIK the only multiple images are optimized_images of uploaded avatars

I think there is something different going on here with your situation. I cant imagine avatars weighing all that much.

[cpradio]

Unless he enabled animated avatars.

[fearlessfrog]

I recognize the filenames repeated I think, and am pretty sure this was a bulk upload of screenshots either in a topic post or in a personal message. The size of the files (all .png’s) is like ‘595934 bytes’ or ‘322155’ as in, not files that look like they have been processed to thumbs or are avatars.

EDIT: We should move the discussion to the new bug topic I think, as I’m filling out the details as I find them there.

[tobiaseigen]

on my site it turned out to be backups that take up all the space. I’ve now changed it to only keep 3 on server, and the rest on s3. I also turned off “s3 disable cleanup” which means I can decide myself which backups there to keep.

[jord8on (Jordan)]

@tobiaseigen, Thank you for chiming back in with what you did here. This is super helpful insight!!