# Stuck in a loop of freeing up space and filling it up again when rebuilding
**URL:** https://meta.discourse.org/t/stuck-in-a-loop-of-freeing-up-space-and-filling-it-up-again-when-rebuilding/96044
**Category:** Self-hosting
**Tags:** server-resources
**Created:** [August 30, 2018, 11:25pm UTC](https://meta.discourse.org/t/stuck-in-a-loop-of-freeing-up-space-and-filling-it-up-again-when-rebuilding/96044 "2018-08-30T23:25:04Z")
**Posts on this page:** 14
**Page:** 2
### Author:  [@tophee](https://meta.discourse.org/u/tophee)
#### Post date: [August 31, 2018, 1:19am UTC](https://meta.discourse.org/t/stuck-in-a-loop-of-freeing-up-space-and-filling-it-up-again-when-rebuilding/96044/21 "2018-08-31T01:19:25Z")
> [@sam](#):
>
> “data container” was saying HEY I need the old Discourse base image
> “web container” was saying I need the new image
How did that happen?
From my side, I can say that I started an update via `/admin/upgrade` which failed, presumably due to a lack of space. Since the website was down, I tried to rebuild via ssh and that’s when the whole misery started…
Regardless of this specific situation, I wonder if the web-updater could not handle lack of disk space better. Rather than crashing the site, could it not just exit before it’s too late?
---
### Author:  [@mpalmer](https://meta.discourse.org/u/mpalmer)
#### Post date: [August 31, 2018, 1:24am UTC](https://meta.discourse.org/t/stuck-in-a-loop-of-freeing-up-space-and-filling-it-up-again-when-rebuilding/96044/22 "2018-08-31T01:24:38Z")
> [@tophee](#):
>
> How did that happen?
Because you hadn’t upgraded the data container.
> [@tophee](#):
>
> I wonder if the web-updater could not handle lack of disk space better.
It could probably do a disk space check before it started, and refuse to proceed if there wasn’t enough space.
---
### Author:  [@tophee](https://meta.discourse.org/u/tophee)
#### Post date: [August 31, 2018, 1:27am UTC](https://meta.discourse.org/t/stuck-in-a-loop-of-freeing-up-space-and-filling-it-up-again-when-rebuilding/96044/23 "2018-08-31T01:27:34Z")
Another question:
Where did you get these IDs from:
> [@sam](#):
>
> aa6b422d88ca […] 1fba0860cbc3
---
### Author:  [@mpalmer](https://meta.discourse.org/u/mpalmer)
#### Post date: [August 31, 2018, 1:28am UTC](https://meta.discourse.org/t/stuck-in-a-loop-of-freeing-up-space-and-filling-it-up-again-when-rebuilding/96044/24 "2018-08-31T01:28:30Z")
Those are the container IDs (first column in the `docker ps` output).
---
### Author:  [@system](https://meta.discourse.org/u/system)
#### Post date: [September 30, 2018, 1:28am UTC](https://meta.discourse.org/t/stuck-in-a-loop-of-freeing-up-space-and-filling-it-up-again-when-rebuilding/96044/25 "2018-09-30T01:28:42Z")
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.
---
### Author:  [@simon](https://meta.discourse.org/u/simon)
#### Post date: [March 18, 2019, 10:14pm UTC](https://meta.discourse.org/t/stuck-in-a-loop-of-freeing-up-space-and-filling-it-up-again-when-rebuilding/96044/26 "2019-03-18T22:14:10Z")
---
### Author:  [@tophee](https://meta.discourse.org/u/tophee)
#### Post date: [March 18, 2019, 10:38pm UTC](https://meta.discourse.org/t/stuck-in-a-loop-of-freeing-up-space-and-filling-it-up-again-when-rebuilding/96044/27 "2019-03-18T22:38:46Z")
> [@mpalmer](#):
>
> Because you hadn’t upgraded the data container.
I still don’t quite understand this. If I’m supposed to upgrade the data conatainer too, why doesn’t it appear in these instructions?
> [@Faster rebuilds?](https://meta.discourse.org/t/faster-rebuilds/40341/13):
>
> ```plaintext
> ./launcher bootstrap web_only
>
> ```
>
> Start outage
>
> ```plaintext
> ./launcher destroy web_only
> ./launcher start web_only
>
> ```
>
> End outage
And isn’t the whole point of the two container setup that I don’t have to rebuild the data container? Well, I guess, not, but if someone could help me understand what the proper procedure for a two container setup is when `/admin/upgrade/` tells me to do
```
cd /var/discourse/
git pull
./launcher rebuild app
```
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [March 19, 2019, 12:02am UTC](https://meta.discourse.org/t/stuck-in-a-loop-of-freeing-up-space-and-filling-it-up-again-when-rebuilding/96044/28 "2019-03-19T00:02:19Z")
> [@tophee](#):
>
> And isn’t the whole point of the two container setup that I don’t have to rebuild the data container?
Hence the command you quoted is talking about the **web\_only** container… not the data container.
> [@tophee](#):
>
> container setup is when `/admin/upgrade/` tells me to do
2 container setup is not something we detect in the web upgrader. The commands you want to run are exactly the ones you quoted…
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [March 19, 2019, 12:27am UTC](https://meta.discourse.org/t/stuck-in-a-loop-of-freeing-up-space-and-filling-it-up-again-when-rebuilding/96044/29 "2019-03-19T00:27:02Z")
It’s assumed that if you can set up a two container install that you can figure out the nuances. Usually you just do the bootstrap/destroy/start sequence.
Database upgrades don’t happen often and they are typically not required when they are first available. They are sometimes required. When they are you can just do a rebuild on the data container. (twice for the last upgrade)
---
### Author:  [@tophee](https://meta.discourse.org/u/tophee)
#### Post date: [March 19, 2019, 6:23am UTC](https://meta.discourse.org/t/stuck-in-a-loop-of-freeing-up-space-and-filling-it-up-again-when-rebuilding/96044/30 "2019-03-19T06:23:45Z")
> [@sam](#):
>
> The commands you want to run are exactly the ones you quoted…
Yet, things went wrong because I did exactly that:
> [@mpalmer](#):
>
> you hadn’t upgraded the data container.
So I guess this was a situation where a database upgrade was required.
> [@pfaffman](#):
>
> Database upgrades … are sometimes required. When they are you can just do a rebuild on the data container.
Bur how do I know whether a database upgrade is required?
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [March 19, 2019, 1:35pm UTC](https://meta.discourse.org/t/stuck-in-a-loop-of-freeing-up-space-and-filling-it-up-again-when-rebuilding/96044/31 "2019-03-19T13:35:41Z")
You see what version of the database you’re running and look at the database templates to see what version they are upgrading to.
One way to see what version you’re running is
```
./launcher enter data
psql --version
```
There are a bunch of topics about the last database upgrade, about 6 months ago (but it might have been 3 or 9). As much as I see you here in surprised you missed it!
And if disk space is a problem then the database upgrade will be challenging because you need at least 2x free space since you’ll be making another copy of the database to upgrade it.
---
### Author:  [@tophee](https://meta.discourse.org/u/tophee)
#### Post date: [March 20, 2019, 12:25am UTC](https://meta.discourse.org/t/stuck-in-a-loop-of-freeing-up-space-and-filling-it-up-again-when-rebuilding/96044/32 "2019-03-20T00:25:24Z")
> [@pfaffman](#):
>
> As much as I see you here in surprised you missed it!
Haha, well, I may have missed it or I may have seen it and scrolled past it because I had no reason to be interested in this kind of low level stuff. Database? It’s like the road when driving a car: you don’t really think about it until it gets bad.
It even took me a while to find what you (probably) where referring to now:
> [@Docker base image with PostgreSQL 10 released](https://meta.discourse.org/t/docker-base-image-with-postgresql-10-released/84536):
>
> launcher has been updated to use the new [Docker base image](https://hub.docker.com/r/discourse/base/tags/) and [postgres.template.yml](https://github.com/discourse/discourse_docker/blob/44f132443bdf72dccf07111832de4c15b5e8abf8/templates/postgres.template.yml) which would use PostgreSQL 10 by default instead of Postgres 9.5. As long as your container’s config is using the default postgres.template.yml, the upgrade of the PostgreSQL data directory would happen automatically the next time you do a ./launcher rebuild. In the event that the PostgreSQL data directory fails to upgrade, please follow the instructions printed by ./launcher rebuild or you may refer to the ins…
But on the way, I found this useful specification of what “not often” means:
> [@Container fails startup after yum update](https://meta.discourse.org/t/container-fails-startup-after-yum-update/56755/20):
>
> We upgrade database every two years, and ruby almost every year, for reference.
To summary my lesson from this topic:
The confusion around the original issue was due to a combination of rather rare circumstances which meant that all standard instructions failed for various reasons:
1. two-container setup
2. need for (rare) database upgrade (which requires more disc space than other upgrades)
3. only 20 GB SSD
Points 1 and 3 are still true but probably not nr. 2. Yet, I still had problems doing the web\_only upgrade. So the db upgrade probably worsened things but it seems it was a mistake to run the two-container setup on 20GB.
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [March 20, 2019, 12:39am UTC](https://meta.discourse.org/t/stuck-in-a-loop-of-freeing-up-space-and-filling-it-up-again-when-rebuilding/96044/33 "2019-03-20T00:39:21Z")
> [@tophee](#):
>
> it seems it was a mistake to run the two-container setup on 20GB.
That seems true. I won’t do it with less than a 2GB-50GB-$10/month Digital Ocean droplet. But it’s hard to get by on 20GB SSD at all. And unless this is your Discourse for your family who you can’t convince to use Discourse to use Discourse at all (maybe I’m projecting), even 25GB is tough.
Glad you got it sorted (you did, right?)
---
### Author:  [@JammyDodger](https://meta.discourse.org/u/JammyDodger)
#### Post date: [June 8, 2024, 12:37pm UTC](https://meta.discourse.org/t/stuck-in-a-loop-of-freeing-up-space-and-filling-it-up-again-when-rebuilding/96044/34 "2024-06-08T12:37:13Z")
This topic was automatically closed after 2108 days. New replies are no longer allowed.
[Previous page](https://meta.discourse.org/t/stuck-in-a-loop-of-freeing-up-space-and-filling-it-up-again-when-rebuilding/96044.md?page=1)