Site down due to lack of disk space


#1

My site ieadmin.info appears not to have made it into the new year…it just returns a 502 bad gateway error message. Have I forgotten to pay for something perhaps…?


#2

Is it a self hosted site or discourse-hosted site?

I meant to ask - did you take a plan from this page https://payments.discourse.org/pricing? Or did you host the site yourselves?


Onebox to payments.discourse.org has broken images
(Bhanu Sharma) #3

Seems like hosted on Digitalocean to me!

Do You have access to the server? have You tried to enter or otherwise reload the container?


(Michael Brown) #4

You’ll need to check the logs on your server for more information.


(Jay Pfaffman) #5

If it’s a self-hosted site on Digital Ocean or similar, though it could be a number of things, disk space is the most likely problem. You’ve likely filled up your disk with backups. You could do something like this:

rm /var/discourse/shared/standalone/backups/default/*tar

Often there’s an uncompressed .tar file if you run out of space and that’ll remove it.

You can check the free disk space before and after the above with

df -h

or

df -h /

to see just the partition that it most likely the one you care about.

You could just reboot and see if it comes back up, though. You might then be able to delete backups from the web interface and change your backup settings to keep a smaller number of backups.

Also, If you’re running Docker 17.11.0-ce (docker --version to check), you should upgrade (I don’t think you’d get that error, but it’s still a good idea).

If you’d rather not bother getting your hands dirty, I’ll figure out what’s wrong and upgrade everything if you place an order here: Discourse Upgrade — Literate Computing .


#6

Many thanks to all of you for your valuable input, which we’re now using in our troubleshootig efforts…


#7

Does this screenshot provide any clues as to what we need to do to get cleanup to run?


#8

Its crystal clear
Root partition is 100% full.
And that is why the site is down.


#9

Go to this location
/var/discourse/shared/standalone/backups/default

and then delete old backups to save space.

Once its done, reboot the server, and the site will be back online.


#10

Thanks, but apparently backups aren’t big enough to free up the needed space…that pic was taken after we’d deleted them. Is there some Docker file we can delete?


(Stephen Chung) #11

Did you do ./launcher cleanup app to clean up stopped docker containers?

Docker does not delete containers that have been stopped and no longer in use. They’ll pile up forever. Each time you do a rebuild, it adds one new container, and they stay in your hard disk forever.

EDIT: I can see that you’re attempting a cleanup. But don’t you have to add app to the command line?


(Michael Friedrich) #12

In order to analyze which directory uses lots of space, these commands will come in handy:

du -h --max-depth=1 /

ncdu

If Docker images are eating to much space, look into the images:

root@monitoring-portal:~# docker images
REPOSITORY                     TAG                 IMAGE ID            CREATED             SIZE
local_discourse/app            latest              fc09337e660c        6 days ago          2.59GB
discourse/base                 2.0.20171231        3925ef3919cc        10 days ago         1.73GB
discourse/base                 2.0.20171204        64d62a045a4e        5 weeks ago         1.81GB

You can either purge them with docker rmi imageid if they are not used, or use @sam 's docker_gc cleanup addition in launcher cleanup.

Note: This removes any unused Docker image, not exclusive to the Discourse app. Still, it works reliably here.


(Rafael dos Santos Silva) #13

Also, if you are on a supported docker version docker system prune is very good at making space.


#14

Profuse thanks to all (what a community!) and special thanks to Rafael for the command for pruning - that’s what we were trying to do, but kept getting an error message until we used that command, which did the trick.