Completely Out of Disk Space

Installation
1

My forum is 1 month old, ~15000 posts… I tried checking it tonight and received a 502 error. I then reloaded my Droplet and tried a ./launcher rebuild and was met with a

"785ef3cb0f4bbe86b21afde2620a966a63e03427e89484b0946a94d1bb34-init: no space left on device.
See '/usr/bin/docker run --help'.
Your Docker installation is not working correctly"

So I did a df -h and saw that my disk was completely full:

Filesystem      Size  Used Avail Use% Mounted on
udev            981M     0  981M   0% /dev
tmpfs           201M  3.1M  197M   2% /run
/dev/vda1        39G   39G     0 100% /
tmpfs          1001M     0 1001M   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs          1001M     0 1001M   0% /sys/fs/cgroup
tmpfs           201M     0  201M   0% /run/user/0

How could this have happened?

I tried apt-get autoclean and apt-get autoremove which didn’t help. Also did ./launcher clean and that didn’t help. Also tried sudo du -xh -d 1

My disk space is still 39GB/39GB :frowning:

1 Like
2

Have you tried running

cd /var/discourse 
./launcher cleanup

That will probably solve your problem.

5 Likes
3

I did try that. I get the same out of space message when trying to cleanup:

./launcher: line 96: cannot create temp file for here-document: No space left on device
./launcher: line 97: cannot create temp file for here-document: No space left on device
./launcher: line 96: cannot create temp file for here-document: No space left on device
./launcher: line 97: cannot create temp file for here-document: No space left on device
./launcher: line 96: cannot create temp file for here-document: No space left on device
./launcher: line 97: cannot create temp file for here-document: No space left on device
./launcher: line 96: cannot create temp file for here-document: No space left on device
./launcher: line 97: cannot create temp file for here-document: No space left on device
/usr/bin/docker: Error response from daemon: mkdir /var/lib/docker/aufs/mnt/6dfedd25fd4d18e682780fc168b2f42ad42b1fee8a4f21b7728f8b4ecf36e51b-init: no space left on device.
See '/usr/bin/docker run --help'.
Your Docker installation is not working correctly

This doesn’t seem normal for a forum my size to take up 39GB?

5

Two things come to mind:

  1. If your instance was populated with a restore, or imported from another platform you may be carrying the restore schema around:

    https://meta.discourse.org/t/disk-space-usage-by-postgres/61137

  2. backups left on disk have filled your space:

    • Check /var/discourse/shared/standalone/backups/default
    • The backups are .tar.gz files
    • Look at the date AND size of the backups before you start deleting.
6 Likes
6

My instance was a fresh install. You read my mind on the automatic backups. I was trying to figure out what I could remove to clear enough space for the ./launcher cleanup to work. So I went into /var/discourse/shared/standalone/backups/default and deleted a few of the old backups which cleared about 150mb of space.

I then was able to run ./launcher cleanup and that netted me back 16GB of space. I still have 24GB used up which seems like quite a lot for a forum my size.

Now my other problem is, I did a reboot on my droplet and ever since I did that, the initial 502 Gateway issue has turned into a Failed to open page issue. I still can’t get access to my Discourse, even after do a ./launcher rebuild

7

@Ericsworld upload your backups to Amazon S3 – this way you save space on your discourse instance

8

What’s in the logs? Share those :wink:

9

Hmn…

Now that you’ve enough headroom, what about apt-get update and apt-get upgrade? Once your droplet went into tilt mode not only the updates to discourse will have failed, but also any to the underlying OS.

Can you see /logs and /sidekiq? What about /safemode?

Sorry if my answers are a bit generic, I’m still on my first coffee.

1 Like
10

Oops my mistake, I did the rebuild again and it went through correctly. Site is back up :slight_smile:

I just did another df -h and it is up to a 83% usage after rebuilding the app.

root@discourse-2gb-sfo1-01:/# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            981M     0  981M   0% /dev
tmpfs           201M  8.2M  192M   5% /run
/dev/vda1        39G   32G  6.9G  83% /
tmpfs          1001M  1.2M  999M   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs          1001M     0 1001M   0% /sys/fs/cgroup
tmpfs           201M     0  201M   0% /run/user/0
none             39G   32G  6.9G  83% /var/lib/docker/aufs/mnt/a6afa8c2711ae69fa62074c1a807463fae0a91b1b5c48cf34244df5a652cd90f
shm              64M  4.0K   64M   1% /var/lib/docker/containers/b75645ecd758ef2627d6d6d9e117757a97928851a2706a091492c3789ba65e0c/shm

So I fear this is not going to last… what is taking up all this space??

11

How large are your backups? What about uploads?

12

Also – I’d recommend getting more disk space.

13

A backup of my discourse is 5.9GB…

Where can I check the upload usage?

I don’t think it’s normal for a forum my size to be using up 32GB like this… or I guess it is normal. DigitalOcean’s Volume Block Storage isn’t available in my server’s region, I will need to look into that to add in more space.

14

Look at the bottom of /admin, there you can see total space occupied by both uploads and backups.

For reference, our backups are 6.8 GB, Uploads 2.96 GB, our instance had 1.8 Mio posts when the import was complete last Nov 13th.

If you’re getting 5.9 GB backups with “only” 150K posts then either your posts are all incredibly large, or you have a huge number/size of uploads.

1 Like
15

Is switching regions an option?

16

Install ncdu, run it on root, where is all the disk space?

6 Likes
17

Thanks, I installed ncdu and ran it on root. Turns out there was still a couple extra backups in the /shared/ folder that didn’t get removed. I think I need to disable Discourse backups due to their disk size and just rely on DigitalOcean Snapshots instead…?

My uploads folder is about 5GB.

1 Like
18

Yeah that would be an option, another option would be to put the backups on a block storage and get some more space.

5 Likes
19

Instead of fully disabling backups, I’d just exclude uploads from them. Snapshots will certainly have your uploads, but might have an inconsistent database.

5 Likes
20

Or toss the backups on Amazon S3…including uploads in backups is useful.

4 Likes
21

Indeed, offsite backups are your friend.

4 Likes