In a discussion about upgrading Discourse when disk space is low, jbrains describes a frustrating loop where ./launcher rebuild app fails due to insufficient space, even after ./launcher cleanup frees up space. They eventually succeeded by stopping the app, cleaning up, pulling updates, and rebuilding — gaining 2 GB extra space. pfaffman explains that Docker leaves old containers/images behind, and cleanup is safer while Discourse runs since it won’t delete active containers. He advises against stopping the app first unless absolutely necessary, and notes that git pull is redundant since ./launcher handles it. Firepup650 questions whether skipping the stop step would just recreate the original failure loop, linking to a related thread about misleading “less than 5GB free” messages. pfaffman clarifies that if space is too tight to hold two images, you risk being unable to rebuild — suggesting moving backups to S3. jbrains expresses surprise that 7.9 GB free wasn’t enough, later realizing the rebuild required downloading a new base image — a hidden space requirement. They suggest improving the upgrade UI with advance warnings or clearer error messages, and propose documenting this issue centrally, noting that the --skip-prereqs flag is hard to find. The core problem: infrequent but disruptive base image updates can silently demand extra space, trapping users in upgrade loops without clear guidance.
Hi, folks. This feels like a dumb question, but I’ll ask it anyway.
I was stuck in this familiar loop:
Upgrade failed.
Try ./launcher rebuild app, but not enough disk space. (Under 5 GB free.)
./launcher cleanup, which seemed to liberate some disk space. (Over 5 GB free.)
GO TO 2.
Since I don’t quite yet know which operations are safe (as in won’t destroy my site data), I stumbled around nervously, hoping not to need to restore from a backup. I ended up doing this and it seemed to work for me.
./launcher stop app
./launcher cleanup, which cleaned up significantly more space than it did previously, which made me immediately nervous.
git pull
./launcher rebuild app
This not only worked, but resulted in a running site with data intact and with more free space than I had previously by approximately 2 GB.
Now I’m not sure whether what I did was brilliant, obvious, or risky and it just happened to work. I’d like to understand a little better why I got the results I got and whether that was a sensible way to upgrade.
docker creates a bunch of files when it builds a container (include the containers that it builds) when you build a new container, you still have the old containers (and also disk images). That’s what gets removed when you do a cleanup.
It’s a little bit safer to run the cleanup command while discourse is running, because the cleanup command won’t delete a running container. If something goes wrong with the rebuild, you can still restart the old container if it exists.
So, I’d remove step (1) above and step (3) is unnecessary because ./launcher does a pull (back in the old days, it didn’t).
So yeah, if your disk space is so tight that you don’t have room for two images, then you need to destroy all of them before building a new one, which puts you in a bad way if for some reason you can’t build a new image. If that’s the case, you really need more disk space. The easiest thing to do is move backups to S3.
I’m surprised by “disk space is so tight” in my situation. I run cleanup, I have 7.9 GB free, then I run ./launcher rebuild app and it refuses to finish. Is 7.9 GB free when I want to rebuild the app really so little?!
(UPDATE: Now I understand how this can happen. I leave this here for Innocent Bystanders to find in a web search. Please read this entire thread, folks!)
For this particular rebuild you needed to download a new base image
It would be particularly helpful to include this information somehow in the upgrade process, either as an advance warning “We need to download a new base image, so you probably need 3 GB more space than normal to upgrade. If you run out of space, that’s probably why.” or as a more-detailed error when the upgrade/rebuild process runs out of space. Otherwise, we end up in an endless loop of “There! There’s all the space you (seem to) need!” (Picard with arm outstretched.)
It would also be particularly helpful not to have to dig quite so much to find the option to use ./launcher rebuild app --skip-prereqs when you mean “Trust me, bro. I have enough disk space.” Yes, put warnings in bold, red, 72-point font.
This “new base image makes upgrading funky” happens infrequently enough for us to forget it, but causes problems during upgrade. Where is the appropriate place for an article about this that centralizes what we know about how to work around the issues?