Can't start/stop discourse or do anything due to lack of space

Support
1

Recently i’ve ran into a problem, I am running 2 discourse forums, and I want to edit one, because its in an infinite redirect loop that I can only fix via discourse config, but can’t because I can’t stop the container. I don’t want to force it using kill or something, as that would possibly cause issues.

I did df -h and it returns

Filesystem      Size  Used Avail Use% Mounted on
udev             16G  4.0K   16G   1% /dev
tmpfs           3.2G  792K  3.2G   1% /run
/dev/md2         20G   15G  4.0G  79% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
none            5.0M     0  5.0M   0% /run/lock
none             16G  2.2M   16G   1% /run/shm
none            100M     0  100M   0% /run/user
/dev/md3        1.8T   39G  1.7T   3% /home
none             20G   15G  4.0G  79% /var/lib/docker/aufs/mnt/c4a287fb570f9d346f768e40d819f78466922b241ab53c3b4d0e67d1093e948f
none             20G   15G  4.0G  79% /var/lib/docker/aufs/mnt/36c27719e26653039d1bac8fb1fc0337c45cd145eaad0b223cd5b5469b2ea295
shm              64M  4.0K   64M   1% /var/lib/docker/containers/5c7e37e4f1bb9339a900e19c6c74e918d0f0b0c76b4b3f23ed1cc342e1dbc610/shm
shm              64M  4.0K   64M   1% /var/lib/docker/containers/3e83e4a36695fd4272863b5b9f3e8a08c3377fbc246b813173202c472112867a/shm

I ran the command ./launcher cleanup however, it does not free up any space. Any command using start/stop will not work, it just says it doesn’t have enough so it can’t continue.

The server has 2x2TB harddrives in raid 1, so space isn’t a problem in general.

Help appreciated.

1 Like
2

When you try a rebuild, the launcher suggests a cleanup is that not working for you? What is the output?

Also, our cleanup scripts will only clean stopped images to try to get more space, they don’t work if you are actually using all this space.

3

When I try to rebuild it gives me this

You have less than 5GB of free space on the disk. You will need more space to continue

Would you like to attempt to recover space by cleaning docker images and containers in the system?(y/N)n
root@REDACTED:/home/discourse# apt-get clean
root@REDACTED:/home/discourse# ./launcher rebuild app
You have less than 5GB of free space on the disk. You will need more space to continue

Would you like to attempt to recover space by cleaning docker images and containers in the system?(y/N)

Then the next step it asks to confirm, and I input yes, then gives me this output

Would you like to attempt to recover space by cleaning docker images and containers in the system?(y/N)y
WARNING! This will remove:
        - all stopped containers
        - all volumes not used by at least one container
        - all networks not used by at least one container
        - all dangling images
Are you sure you want to continue? [y/N] y
Total reclaimed space: 0B
If the cleanup was successful, you may try again now

But trying again won’t do anything.

4

Looks like you have no old images, what’s the output of docker images and docker ps -a ?

5 
CONTAINER ID        IMAGE                         COMMAND             CREATED             STATUS              PORTS                                         NAMES
5c7e37e4f1bb        local_discourse/app1        "/sbin/boot"        3 months ago        Up 9 days           0.0.0.0:redacted->80/tcp, 0.0.0.0:redacted->443/tcp   app1
3e83e4a36695        local_discourse/app2   "/sbin/boot"        4 months ago        Up 9 days           0.0.0.0:redacted->80/tcp, 0.0.0.0:redacted->443/tcp   app2
6

annnnd docker images?

7

Sorry, forgot that one

REPOSITORY                    TAG                 IMAGE ID            CREATED             SIZE
discourse/base                2.0.20170531        4aa3fe49ae44        6 weeks ago         1.66GB
local_discourse/app1        latest              a2e3806eec2b        3 months ago        2.48GB
local_discourse/app2   latest              0fb69bf47bfb        4 months ago        2.42GB
discourse/discourse           1.3.10              1881a9bd3afd        6 months ago        1.54GB
samsaffron/docker-gc          latest              54ca424ca8d6        20 months ago       57.7MB
8

… which you allocated almost all of to /home. So, short of bind-mounting /var/lib/docker and /var/discourse into there, you’re always going to be very short of space. You might want to consider reinstalling the machine with a saner partitioning scheme.

4 Likes
9

Yeah, it was default layout for the install, I am not even sure if I can install it differently ,and its probably too late at this point, I have too much other applications setup on it. I moved discourse to /home anyway (or the shared data, and container config, etc) so it didn’t use up the root partition.

Is it possible to just move docker images to /home? I really can’t reinstall the server right now.

10

Docker images live under /var/lib/docker; there’s probably a way to change where docker stores all its stuff, otherwise the aforementioned bind mounting should probably work.

11

Well what should I do with the containers if I can’t stop them? what damage would killing them do? or perhaps is there another way I can stop it safely?

12

./launcher stop app --skip-prereqs should work.

3 Likes
14

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.