# Upgrade failed: low disk space -- excess overlay files?

**URL:** https://meta.discourse.org/t/upgrade-failed-low-disk-space-excess-overlay-files/287048
**Category:** Self-hosting
**Tags:** server-resources
**Created:** [November 30, 2023, 8:31am UTC](https://meta.discourse.org/t/upgrade-failed-low-disk-space-excess-overlay-files/287048 "2023-11-30T08:31:57Z")
**Posts on this page:** 7
**Page:** 1

<div class="post-metadata">

### Author: ![ToddZ](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/toddz/32/328350_2.png) [@ToddZ](https://meta.discourse.org/u/ToddZ)
#### Post date: [November 30, 2023, 8:31am UTC](https://meta.discourse.org/t/upgrade-failed-low-disk-space-excess-overlay-files/287048/1 "2023-11-30T08:31:57Z")

</div>

Well, sigh – I’m stuck on a failed upgrade.

I’m on a 25G VPS, using the supported Docker installation.

Upgrading docker\_manager via the admin panel went fine.

Upgrading Discourse from **v3.2.0.beta1 +125** to **v3.2.0.beta3 +325** via the admin panel failed, so I tried a command line install:

```plaintext
cd /var/discourse
git pull
./launcher rebuild app

```

…which also failed:

```plaintext
You have less than 5GB of free space on the disk where /var/lib/docker is located. You will need more space to continue
Filesystem Size Used Avail Use% Mounted on
/dev/vda2 23G 22G 640M 98% /

```

Apparently because of two 18G “overlay” files:

```plaintext
root@forum:/var/discourse# df -h
Filesystem Size Used Avail Use% Mounted on
tmpfs 95M 1.4M 94M 2% /run
/dev/vda2 23G 18G 4.1G 82% /
tmpfs 474M 0 474M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/vda1 511M 6.1M 505M 2% /boot/efi
tmpfs 95M 4.0K 95M 1% /run/user/0
overlay 23G 18G 4.1G 82% /var/lib/docker/overlay2/8a331589d7fa9046a6ef73489cc830c2583cb76c9174125c8bfe1064d58cd503/merged
overlay 23G 18G 4.1G 82% /var/lib/docker/overlay2/d56574358c8edbc9bc1fb50022585b854462a8ce56daa636b07f3a3771949251/merged

```

(_Three_ 18G files on a 25G server? That’s 54G…)

It seems _something_ is reclaimable:

```plaintext
root@forum:/var/discourse# docker system df
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 2 2 4.3GB 3.334GB (77%)
Containers 2 2 1.849GB 0B (0%)
Local Volumes 0 0 0B 0B
Build Cache 0 0 0B 0B

```

…but I’m not sure what or how.

Contents of `/var/discourse/shared/standalone/backups/default` only amount to 67Mb.

I stopped docker with `systemctl stop docker` and tried these to no effect:

```plaintext
docker system prune -a
docker buildx prune --all
docker builder prune --all

```

…all reported 0B freed.

I have two Docker images, one for Discourse and one for… “none?”

```plaintext
root@forum:/var/discourse/image# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
local_discourse/app latest 5ff1dcfe050c 2 months ago 4.09GB
<none> <none> bbaceb5f4a80 2 months ago 214MB

```

Apparently “none” indicates a dangling or intermediate image: [Why the “none” image appears in Docker and how can we avoid it - Stack Overflow](https://stackoverflow.com/questions/53221412/why-the-none-image-appears-in-docker-and-how-can-we-avoid-it) – but it’s so small I don’t think it’s my first priority.

When advice at [Is it safe to clean docker/overlay2/ - Stack Overflow](https://stackoverflow.com/questions/46672001/is-it-safe-to-clean-docker-overlay2/70527862#70527862) gets into grepping overlays against images, I lose steam. There are 60 hashed folders in my docker/overlay2… please don’t make me grep 120 times…

I imagine my options at this point are:  
A. Get some help figuring out if either of these overlays can be deleted.  
B. Restore from a snapshot, upgrade for more disk space and try again. Will I always have these huge overlays?

(And how do I even have 3 x 18G files on a 25G instance..?)

If anyone’s up at this hour and has any input, I’d appreciate it.

---

<div class="post-metadata">

### Author: ![JammyDodger](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/jammydodger/32/254611_2.png) [@JammyDodger](https://meta.discourse.org/u/JammyDodger)
#### Post date: [November 30, 2023, 9:29am UTC](https://meta.discourse.org/t/upgrade-failed-low-disk-space-excess-overlay-files/287048/2 "2023-11-30T09:29:22Z")

</div>

Just to tick off the basics, but you have tried `./launcher cleanup` and this is what’s left after?

---

<div class="post-metadata">

### Author: ![ToddZ](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/toddz/32/328350_2.png) [@ToddZ](https://meta.discourse.org/u/ToddZ)
#### Post date: [November 30, 2023, 9:52am UTC](https://meta.discourse.org/t/upgrade-failed-low-disk-space-excess-overlay-files/287048/3 "2023-11-30T09:52:08Z")

</div>

Yes - cleanup had no effect.

---

<div class="post-metadata">

### Author: ![Stephen](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/stephen/32/95011_2.png) [@Stephen](https://meta.discourse.org/u/Stephen)
#### Post date: [November 30, 2023, 10:09am UTC](https://meta.discourse.org/t/upgrade-failed-low-disk-space-excess-overlay-files/287048/4 "2023-11-30T10:09:34Z")

</div>

> [@ToddZ](#):
>
> Apparently because of two 18G “overlay” files:

You don’t have two 18GB overlay files, that’s a red herring. Docker uses overlayfs and those are just how your existing disk is presented to the container. `/dev/vda2` is your disk and is mounted at `/` - that’s where you should be directing your efforts.

If `./launcher cleanup` did nothing then I’m assuming `docker image prune` (removes dangling images) won’t either. If you’re only using this server for discourse you may need to just check around to make sure there are no large files/folders in your home directory.

---

<div class="post-metadata">

### Author: ![ToddZ](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/toddz/32/328350_2.png) [@ToddZ](https://meta.discourse.org/u/ToddZ)
#### Post date: [November 30, 2023, 10:39am UTC](https://meta.discourse.org/t/upgrade-failed-low-disk-space-excess-overlay-files/287048/5 "2023-11-30T10:39:49Z")

</div>

Ohh – well, that’s tricky of Docker…  
No, prune operations didn’t recover anything.  
I’m poking through /usr now with ncdu… nothing looks like it obviously doesn’t belong, though I’m not sure what to make of /usr/lib/modules:

```plaintext
  547.2 MiB [###########################] /6.2.0-37-generic
  547.2 MiB [##########################] /6.2.0-34-generic
    1.2 MiB [] /6.2.0-33-generic
    1.2 MiB [] /6.2.0-32-generic
    1.2 MiB [] /6.2.0-35-generic
    1.2 MiB [] /6.2.0-36-generic

```

By far the most use is reported as the overlays down in /var:

```plaintext
   16.0 GiB [###########################] /var
    4.3 GiB [#######] /usr
    2.3 GiB [###] swapfile
    1.7 GiB [##] /snap
  289.5 MiB [] /boot

```

There’s nothing in /snap but what it came with:

```plaintext
root@forum:/# snap list
Name Version Rev Tracking Publisher Notes
core22 20230801 864 latest/stable canonical✓ base
lxd 5.19-8635f82 26200 latest/stable/… canonical✓ -
snapd 2.60.4 20290 latest/stable canonical✓ snapd

```

Whoa – /var/log/journal got big!

```plaintext
    1.8 GiB [###########################] /7341e5ac94ae440bbd06f743e242da89
   16.0 MiB [] /7025a9ae870140c1bef8e55211d339dc

```

Looks like it’s been tons of bots trying to log in over just a couple of months.  
Seems prudent to retain logs for a while, but this forum is still beta.  
Maybe vacuuming that will be enough to get me going again.

---

<div class="post-metadata">

### Author: ![ToddZ](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/toddz/32/328350_2.png) [@ToddZ](https://meta.discourse.org/u/ToddZ)
#### Post date: [November 30, 2023, 12:38pm UTC](https://meta.discourse.org/t/upgrade-failed-low-disk-space-excess-overlay-files/287048/6 "2023-11-30T12:38:19Z")

</div>

Well, that didn’t quite do it, so I upgraded the server to 55G. If those big overlay files are inevitable, I guess there wasn’t really a choice.

A Discourse upgrade just completed, the site appears to be working fine on 3.2.0.beta4-dev. 😅

Thank you @JammyDodger and @Stephen for your attention and input!

---

<div class="post-metadata">

### Author: ![dandv](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/dandv/32/169779_2.png) [@dandv](https://meta.discourse.org/u/dandv)
#### Post date: [April 3, 2025, 5:22pm UTC](https://meta.discourse.org/t/upgrade-failed-low-disk-space-excess-overlay-files/287048/7 "2025-04-03T17:22:46Z")

</div>

I was running out of space on a 50GB Linode VPS.

Below are some space hogs that haven’t been mentioned yet. Some are Discourse-specific, some are general to Linux systems.

1. Run `ll /lib/modules`. I was surprised to see about 15GB of old kernels that `apt autoremove` didn’t care to remove. Claude thinks they were installed in a non-standard way, and generated a safe [removal script](https://claude.site/artifacts/662583a0-d2e1-468d-926b-0c0b50378a5f). It took about an hour but it worked (run at your own risk, of course) and I was able to run `./launcher rebuild` without the `no space left on device` error.

2. The script didn’t delete the corresponding headers in `/usr/src`. For that ChatGPT created [another script](https://chatgpt.com/share/67eec136-1688-8004-b195-e69de55bdb21).

3. About half a gigabyte of space was taken by [useless locales](https://meta.discourse.org/t/hundreds-of-megabytes-of-apparently-duplicate-locale-js-files/360254).

4. Another GB+ was taken by the `/var/lib/docker/overlay2/.../merged/home/discourse/.cache` directory.

Maybe a stupid question, but what exactly do the `merge` and `diff` directories contain? Can one of them be deleted safely at some point?

 ![Docker container with fatass /merged and /diff directories](https://global.discourse-cdn.com/meta/original/4X/8/3/8/838b119081805a61e6f8e544eace583ebbc80424.png)
