# Stuck in a loop of freeing up space and filling it up again when rebuilding

**URL:** https://meta.discourse.org/t/stuck-in-a-loop-of-freeing-up-space-and-filling-it-up-again-when-rebuilding/96044
**Category:** Self-hosting
**Tags:** server-resources
**Created:** [30.Август.2018 23:25:04 UTC](https://meta.discourse.org/t/stuck-in-a-loop-of-freeing-up-space-and-filling-it-up-again-when-rebuilding/96044 "2018-08-30T23:25:04Z")
**Posts on this page:** 20
**Page:** 1

<div class="post-metadata">

### Author: ![tophee](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/tophee/32/73406_2.png) [@tophee](https://meta.discourse.org/u/tophee)
#### Post date: [30.Август.2018 23:25:04 UTC](https://meta.discourse.org/t/stuck-in-a-loop-of-freeing-up-space-and-filling-it-up-again-when-rebuilding/96044/1 "2018-08-30T23:25:04Z")

</div>

My rebuild failed due to lack of disk space so I need to free up some space. But I’m stuck in a loop: `./launcher cleanup` frees up sufficient space to be above 5 GB. So I start a rebuild but that then fills up the recovered space again so that it fails to complete. See below.

How do I get things running again?

```plaintext
$ sudo ./launcher cleanup

The following command will
- Delete all docker images for old containers
- Delete all stopped and orphan containers

Are you sure (Y/n):

Starting Cleanup (bytes free 3931580)
Finished Cleanup (bytes free 5903356)
$ sudo ./launcher rebuild app
WARNING: Docker version 17.05.0-ce deprecated, recommend upgrade to 17.06.2 or newer.

WARNING: We are about to start downloading the Discourse base image
This process may take anywhere between a few minutes to an hour, depending on your network speed

Please be patient

Unable to find image 'discourse/base:2.0.20180802' locally
2.0.20180802: Pulling from discourse/base
8ee29e426c26: Pulling fs layer
6e83b260b73b: Pulling fs layer
e26b65fd1143: Pulling fs layer
40dca07f8222: Pulling fs layer
b420ae9e10b3: Pulling fs layer
b89ccfe9dadc: Pulling fs layer
40dca07f8222: Waiting
b420ae9e10b3: Waiting
b89ccfe9dadc: Waiting
e26b65fd1143: Verifying Checksum
e26b65fd1143: Download complete
6e83b260b73b: Verifying Checksum
6e83b260b73b: Download complete
b420ae9e10b3: Verifying Checksum
b420ae9e10b3: Download complete
40dca07f8222: Verifying Checksum
40dca07f8222: Download complete
8ee29e426c26: Verifying Checksum
8ee29e426c26: Download complete
8ee29e426c26: Pull complete
6e83b260b73b: Pull complete
e26b65fd1143: Pull complete
40dca07f8222: Pull complete
b420ae9e10b3: Pull complete
b89ccfe9dadc: Verifying Checksum
b89ccfe9dadc: Download complete
b89ccfe9dadc: Pull complete
Digest: sha256:be738714169c78e371f93bfa1079f750475b0910567d4f86fa50d6e66910b656
Status: Downloaded newer image for discourse/base:2.0.20180802
You have less than 5GB of free space on the disk where /var/lib/docker is located. You will need more space to continue
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg-lv_root 19G 14G 3.8G 79% /

Would you like to attempt to recover space by cleaning docker images and containers in the system?(y/N)y
WARNING! This will remove:
        - all stopped containers
        - all volumes not used by at least one container
        - all networks not used by at least one container
        - all dangling images
Are you sure you want to continue? [y/N] y
Total reclaimed space: 0B
If the cleanup was successful, you may try again now
$

```

---

<div class="post-metadata">

### Author: ![mpalmer](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/mpalmer/32/45740_2.png) [@mpalmer](https://meta.discourse.org/u/mpalmer)
#### Post date: [30.Август.2018 23:38:10 UTC](https://meta.discourse.org/t/stuck-in-a-loop-of-freeing-up-space-and-filling-it-up-again-when-rebuilding/96044/2 "2018-08-30T23:38:10Z")

</div>

Clean up some more space so that the build has enough breathing room to complete. I find that Docker’s cleanup system is not wonderful at purging old images sometimes, so I sometimes have to do a `docker images` followed by a long `docker rmi <ID> <ID> <ID> ...`.

---

<div class="post-metadata">

### Author: ![tophee](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/tophee/32/73406_2.png) [@tophee](https://meta.discourse.org/u/tophee)
#### Post date: [30.Август.2018 23:42:22 UTC](https://meta.discourse.org/t/stuck-in-a-loop-of-freeing-up-space-and-filling-it-up-again-when-rebuilding/96044/3 "2018-08-30T23:42:22Z")

</div>

How do I know which images I can delete?

```plaintext
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
discourse/base 2.0.20180802 d6f8b6029227 4 weeks ago 1.74GB
local_discourse/web_only latest 0301af96b2e8 5 months ago 2.95GB
local_discourse/data latest a8b8c3da644a 8 months ago 1.85GB
local_discourse/mail-receiver latest a322c9207234 8 months ago 142MB
discourse/base 2.0.20171204 64d62a045a4e 8 months ago 1.81GB
discourse/mail-receiver 1.1.2 44042627246b 14 months ago 142MB
samsaffron/docker-gc latest 54ca424ca8d6 2 years ago 57.7MB

```

---

<div class="post-metadata">

### Author: ![mpalmer](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/mpalmer/32/45740_2.png) [@mpalmer](https://meta.discourse.org/u/mpalmer)
#### Post date: [30.Август.2018 23:45:19 UTC](https://meta.discourse.org/t/stuck-in-a-loop-of-freeing-up-space-and-filling-it-up-again-when-rebuilding/96044/4 "2018-08-30T23:45:19Z")

</div>

Anything not in use by a running container is usually safe enough, as far as Discourse is concerned, because it’ll be re-downloaded and/or rebuilt when you do the needful. There’s not a huge pile of images there, though; it’s probably time for you to get a disk upgrade.

---

<div class="post-metadata">

### Author: ![tophee](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/tophee/32/73406_2.png) [@tophee](https://meta.discourse.org/u/tophee)
#### Post date: [30.Август.2018 23:47:54 UTC](https://meta.discourse.org/t/stuck-in-a-loop-of-freeing-up-space-and-filling-it-up-again-when-rebuilding/96044/5 "2018-08-30T23:47:54Z")

</div>

> [@mpalmer](#):
>
> because it’ll be re-downloaded and/or rebuilt when you do the needful.

So deleting discourse containers is always safe?

> [@mpalmer](#):
>
> it’s probably time for you to get a disk upgrade.

Not really. This is a tiny and very low activity forum that has hardly grown in the past months…

* * *

`docker rmi 64d62a045a4e` gives me

```plaintext
Error response from daemon: conflict: unable to delete 64d62a045a4e (cannot be forced) - image has dependent child images

```

---

<div class="post-metadata">

### Author: ![tophee](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/tophee/32/73406_2.png) [@tophee](https://meta.discourse.org/u/tophee)
#### Post date: [31.Август.2018 00:15:53 UTC](https://meta.discourse.org/t/stuck-in-a-loop-of-freeing-up-space-and-filling-it-up-again-when-rebuilding/96044/6 "2018-08-31T00:15:53Z")

</div>

Is there any way I can stop it from downloading the latest discourse base image every time I try to rebuild or start the app? I’d like it to just use the old one for now so that I can go to bed…

---

<div class="post-metadata">

### Author: ![sam](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/sam/32/102149_2.png) [@sam](https://meta.discourse.org/u/sam)
#### Post date: [31.Август.2018 00:20:15 UTC](https://meta.discourse.org/t/stuck-in-a-loop-of-freeing-up-space-and-filling-it-up-again-when-rebuilding/96044/7 "2018-08-31T00:20:15Z")

</div>

> [@tophee](#):
>
> Is there any way I can stop it from downloading the latest discourse base image every time I try to rebuild or start the app

It only downloads it if it is not on local, we really only download an image once. We only bump the required image once every few months in `launcher`. There are ways to specify a base image BUT you do not want to do that for a rainbow of reasons.

---

<div class="post-metadata">

### Author: ![tophee](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/tophee/32/73406_2.png) [@tophee](https://meta.discourse.org/u/tophee)
#### Post date: [31.Август.2018 00:21:56 UTC](https://meta.discourse.org/t/stuck-in-a-loop-of-freeing-up-space-and-filling-it-up-again-when-rebuilding/96044/8 "2018-08-31T00:21:56Z")

</div>

Okay, so is there any way I can get rid of one of [those](https://meta.discourse.org/t/stuck-in-a-loop-of-freeing-up-space-and-filling-it-up-again-when-rebuilding/96044/3) images? I surely don’t need two base images?

---

<div class="post-metadata">

### Author: ![pfaffman](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/pfaffman/32/120154_2.png) [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [31.Август.2018 00:27:08 UTC](https://meta.discourse.org/t/stuck-in-a-loop-of-freeing-up-space-and-filling-it-up-again-when-rebuilding/96044/9 "2018-08-31T00:27:08Z")

</div>

How much disk space do you have?

Can you delete some backups?

---

<div class="post-metadata">

### Author: ![sam](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/sam/32/102149_2.png) [@sam](https://meta.discourse.org/u/sam)
#### Post date: [31.Август.2018 00:27:49 UTC](https://meta.discourse.org/t/stuck-in-a-loop-of-freeing-up-space-and-filling-it-up-again-when-rebuilding/96044/10 "2018-08-31T00:27:49Z")

</div>

Sure…

Have you ran `./launcher cleanup` it will purge all images not in use.

---

<div class="post-metadata">

### Author: ![tophee](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/tophee/32/73406_2.png) [@tophee](https://meta.discourse.org/u/tophee)
#### Post date: [31.Август.2018 00:31:01 UTC](https://meta.discourse.org/t/stuck-in-a-loop-of-freeing-up-space-and-filling-it-up-again-when-rebuilding/96044/11 "2018-08-31T00:31:01Z")

</div>

> [@pfaffman](#):
>
> How much disk space do you have?

```plaintext
$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 476M 0 476M 0% /dev
tmpfs 100M 8.9M 91M 9% /run
/dev/mapper/vg-lv_root 19G 14G 4.1G 77% /
tmpfs 497M 1.1M 496M 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 497M 0 497M 0% /sys/fs/cgroup
/dev/sda1 461M 160M 278M 37% /boot
tmpfs 100M 0 100M 0% /run/user/1000

```

> [@pfaffman](#):
>
> Can you delete some backups?

Deleted them already.

> [@sam](#):
>
> Have you ran `./launcher cleanup` it will purge all images not in use.

Yes, I tried that. and I get

> [@tophee](#):
>
> Starting Cleanup (bytes free 3931580) Finished Cleanup (bytes free 5903356)

But when I try to rebuild or start the app, the base images that was presumably deleted by cleanup gets downloaded again and I’m back to where I started.

---

<div class="post-metadata">

### Author: ![sam](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/sam/32/102149_2.png) [@sam](https://meta.discourse.org/u/sam)
#### Post date: [31.Август.2018 00:35:06 UTC](https://meta.discourse.org/t/stuck-in-a-loop-of-freeing-up-space-and-filling-it-up-again-when-rebuilding/96044/12 "2018-08-31T00:35:06Z")

</div>

My suggestion here then if you are ULTRA tight on space is just to go nuclear and and start from scratch.

What is the output of `docker ps`, you basically need to kill your app container so you can free up the old base image.

---

<div class="post-metadata">

### Author: ![tophee](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/tophee/32/73406_2.png) [@tophee](https://meta.discourse.org/u/tophee)
#### Post date: [31.Август.2018 00:37:52 UTC](https://meta.discourse.org/t/stuck-in-a-loop-of-freeing-up-space-and-filling-it-up-again-when-rebuilding/96044/13 "2018-08-31T00:37:52Z")

</div>

```plaintext
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1fba0860cbc3 local_discourse/web_only "/sbin/boot" 5 months ago Up 29 minutes 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp web_only
aa6b422d88ca local_discourse/data "/sbin/boot" 8 months ago Up 29 minutes data
2940a1603151 local_discourse/mail-receiver "/sbin/boot" 8 months ago Up 29 minutes 0.0.0.0:25->25/tcp mail-receiver

```

> [@sam](#):
>
> go nuclear and and start from scratch

What do you mean by that? Will I need a discourse backup? Cause I don’t have one…

---

<div class="post-metadata">

### Author: ![sam](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/sam/32/102149_2.png) [@sam](https://meta.discourse.org/u/sam)
#### Post date: [31.Август.2018 00:40:29 UTC](https://meta.discourse.org/t/stuck-in-a-loop-of-freeing-up-space-and-filling-it-up-again-when-rebuilding/96044/14 "2018-08-31T00:40:29Z")

</div>

```plaintext
docker rm -f aa6b422d88ca
docker rm -f 1fba0860cbc3
./launcher cleanup 
./launcher rebuild data
./launcher rebuild web_only

```

But really the root cause here is that you are just way too tight on disk space. I would recommend doubling the size `/dev/mappper/vg-lv_root`

---

<div class="post-metadata">

### Author: ![tophee](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/tophee/32/73406_2.png) [@tophee](https://meta.discourse.org/u/tophee)
#### Post date: [31.Август.2018 00:54:36 UTC](https://meta.discourse.org/t/stuck-in-a-loop-of-freeing-up-space-and-filling-it-up-again-when-rebuilding/96044/15 "2018-08-31T00:54:36Z")

</div>

The last step (rebuild web\_only) fails:

```plaintext
Bundle complete! 110 Gemfile dependencies, 201 gems now installed.
Gems in the group development were not installed.
Bundled gems are installed into `./vendor/bundle`

I, [2018-08-31T00:50:57.518677 #13] INFO -- : > cd /var/www/discourse && su discourse -c 'bundle exec rake db:migrate'
rake aborted!
Gem::LoadError: can't activate public_suffix-2.0.5, already activated public_suffix-3.0.2
/var/www/discourse/lib/plugin_gem.rb:18:in `load'
/var/www/discourse/lib/plugin/instance.rb:501:in `gem'
/var/www/discourse/plugins/discourse-sync-to-dropbox/plugin.rb:7:in `activate!'
/var/www/discourse/lib/plugin/instance.rb:431:in `instance_eval'
/var/www/discourse/lib/plugin/instance.rb:431:in `activate!'
lib/discourse.rb:164:in `block in activate_plugins!'
lib/discourse.rb:161:in `each'
lib/discourse.rb:161:in `activate_plugins!'
/var/www/discourse/config/application.rb:223:in `<class:Application>'
/var/www/discourse/config/application.rb:39:in `<module:Discourse>'
/var/www/discourse/config/application.rb:38:in `<top (required)>'
/var/www/discourse/Rakefile:5:in `require'
/var/www/discourse/Rakefile:5:in `<top (required)>'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rake-12.3.1/exe/rake:27:in `<top (required)>'
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'
(See full trace by running task with --trace)
I, [2018-08-31T00:51:18.343389 #13] INFO -- : gem install geocoder -v 1.4.4 -i /var/www/discourse/plugins/discourse-locations/gems/2.5.1 --no-document --ignore-dependencies
Successfully installed geocoder-1.4.4
1 gem installed
gem install public_suffix -v 2.0.5 -i /var/www/discourse/plugins/discourse-sync-to-dropbox/gems/2.5.1 --no-document --ignore-dependencies
Successfully installed public_suffix-2.0.5
1 gem installed

FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && su discourse -c 'bundle exec rake db:migrate' failed with return #<Process::Status: pid 507 exit 1>
Location of failure: /pups/lib/pups/exec_command.rb:112:in `spawn'
exec failed with the params {"cd"=>"$home", "hook"=>"bundle_exec", "cmd"=>["su discourse -c 'bundle install --deployment --verbose --without test --without development --retry 3 --jobs 4'", "su discourse -c 'bundle exec rake db:migrate'", "su discourse -c 'bundle exec rake assets:precompile'"]}
36b568dc13935b0491b99afd4763776899dad8d57a1df149633b9c7c02fd3c0b
**FAILED TO BOOTSTRAP** please scroll up and look for earlier error messages, there may be more than one

```

Trying without the dropbox plugin now…

---

<div class="post-metadata">

### Author: ![sam](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/sam/32/102149_2.png) [@sam](https://meta.discourse.org/u/sam)
#### Post date: [31.Август.2018 01:00:10 UTC](https://meta.discourse.org/t/stuck-in-a-loop-of-freeing-up-space-and-filling-it-up-again-when-rebuilding/96044/16 "2018-08-31T01:00:10Z")

</div>

> [@tophee](#):
>
> Gem::LoadError: can’t activate public\_suffix-2.0.5, already activated public\_suffix-3.0.2 /var/www/discourse/lib/plugin\_gem.rb:18:in `load

Hmmmm … are you on tests passed… stable or beta?

Also, remove `discourse-sync-to-dropbox` asap. It looks like it is not working at the moment.

---

<div class="post-metadata">

### Author: ![tophee](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/tophee/32/73406_2.png) [@tophee](https://meta.discourse.org/u/tophee)
#### Post date: [31.Август.2018 01:09:57 UTC](https://meta.discourse.org/t/stuck-in-a-loop-of-freeing-up-space-and-filling-it-up-again-when-rebuilding/96044/17 "2018-08-31T01:09:57Z")

</div>

Phew! After removing the dropbox plugin my site is finally back up and running. Thanks a lot to everyone for your support!

> [@sam](#):
>
> are you on tests passed… stable or beta?

Since I never changed that, I guess I’m on tests-passed.

> [@sam](#):
>
> I would recommend doubling the size `/dev/mappper/vg-lv_root`

I guess you’re right, but how does that relate to discourse’s minimum requirements of 10 GB HDD?

---

<div class="post-metadata">

### Author: ![sam](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/sam/32/102149_2.png) [@sam](https://meta.discourse.org/u/sam)
#### Post date: [31.Август.2018 01:10:55 UTC](https://meta.discourse.org/t/stuck-in-a-loop-of-freeing-up-space-and-filling-it-up-again-when-rebuilding/96044/18 "2018-08-31T01:10:55Z")

</div>

10GB is ambitious and absolutely not the case if you are running seperate data and web containers.

---

<div class="post-metadata">

### Author: ![tophee](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/tophee/32/73406_2.png) [@tophee](https://meta.discourse.org/u/tophee)
#### Post date: [31.Август.2018 01:14:11 UTC](https://meta.discourse.org/t/stuck-in-a-loop-of-freeing-up-space-and-filling-it-up-again-when-rebuilding/96044/19 "2018-08-31T01:14:11Z")

</div>

> [@sam](#):
>
> absolutely not the case if you are running seperate data and web containers

Ah, I see. I don’t think that was mentioned anywhere. I’ll check tomorrow where that information might be usefully added.

However, the situation seems to look much brighter now:

```plaintext
$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 476M 0 476M 0% /dev
tmpfs 100M 12M 88M 12% /run
/dev/mapper/vg-lv_root 19G 8.4G 9.0G 49% /
tmpfs 497M 1.4M 495M 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 497M 0 497M 0% /sys/fs/cgroup
/dev/sda1 461M 160M 278M 37% /boot
tmpfs 100M 0 100M 0% /run/user/1000

```

Not sure where all the space came from but it looks good, doesn’t it? Do you still think I need to upgrade the server?

---

<div class="post-metadata">

### Author: ![sam](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/sam/32/102149_2.png) [@sam](https://meta.discourse.org/u/sam)
#### Post date: [31.Август.2018 01:15:48 UTC](https://meta.discourse.org/t/stuck-in-a-loop-of-freeing-up-space-and-filling-it-up-again-when-rebuilding/96044/20 "2018-08-31T01:15:48Z")

</div>

The data container was using the old image, so you had a situation where:

“data container” was saying HEY I need the old Discourse base image  
“web container” was saying I need the new image

Not sure where all the space came from but this is a chunk of it.

[Следующая страница](https://meta.discourse.org/t/stuck-in-a-loop-of-freeing-up-space-and-filling-it-up-again-when-rebuilding/96044.md?page=2)
