Well, sigh – I’m stuck on a failed upgrade. I’m on a 25G VPS, using the supported Docker installation. Upgrading docker_manager via the admin panel went fine. Upgrading Discourse from v3.2.0.beta1 +125 to v3.2.0.beta3 +325 via the admin panel failed, so I tried a command line install: cd /var/discourse git pull ./launcher rebuild app …which also failed: You have less than 5GB of free space on the disk where /var/lib/docker is located. You will need more space to continue Filesystem Size Used Avail Use% Mounted on /dev/vda2 23G 22G 640M 98% / Apparently because of two 18G “overlay” files: root@forum:/var/discourse# df -h Filesystem Size Used Avail Use% Mounted on tmpfs 95M 1.4M 94M 2% /run /dev/vda2 23G 18G 4.1G 82% / tmpfs 474M 0 474M 0% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock /dev/vda1 511M 6.1M 505M 2% /boot/efi tmpfs 95M 4.0K 95M 1% /run/user/0 overlay 23G 18G 4.1G 82% /var/lib/docker/overlay2/8a331589d7fa9046a6ef73489cc830c2583cb76c9174125c8bfe1064d58cd503/merged overlay 23G 18G 4.1G 82% /var/lib/docker/overlay2/d56574358c8edbc9bc1fb50022585b854462a8ce56daa636b07f3a3771949251/merged (Three 18G files on a 25G server? That’s 54G…) It seems something is reclaimable: root@forum:/var/discourse# docker system df TYPE TOTAL ACTIVE SIZE RECLAIMABLE Images 2 2 4.3GB 3.334GB (77%) Containers 2 2 1.849GB 0B (0%) Local Volumes 0 0 0B 0B Build Cache 0 0 0B 0B …but I’m not sure what or how. Contents of /var/discourse/shared/standalone/backups/default only amount to 67Mb. I stopped docker with systemctl stop docker and tried these to no effect: docker system prune -a docker buildx prune --all docker builder prune --all …all reported 0B freed. I have two Docker images, one for Discourse and one for… “none?” root@forum:/var/discourse/image# docker images REPOSITORY TAG IMAGE ID CREATED SIZE local_discourse/app latest 5ff1dcfe050c 2 months ago 4.09GB bbaceb5f4a80 2 months ago 214MB Apparently “none” indicates a dangling or intermediate image: Why the “none” image appears in Docker and how can we avoid it - Stack Overflow – but it’s so small I don’t think it’s my first priority. When advice at Is it safe to clean docker/overlay2/ - Stack Overflow gets into grepping overlays against images, I lose steam. There are 60 hashed folders in my docker/overlay2… please don’t make me grep 120 times… I imagine my options at this point are: A. Get some help figuring out if either of these overlays can be deleted. B. Restore from a snapshot, upgrade for more disk space and try again. Will I always have these huge overlays? (And how do I even have 3 x 18G files on a 25G instance..?) If anyone’s up at this hour and has any input, I’d appreciate it.

[image] ToddZ: Apparently because of two 18G “overlay” files: You don’t have two 18GB overlay files, that’s a red herring. Docker uses overlayfs and those are just how your existing disk is presented to the container. /dev/vda2 is your disk and is mounted at / - that’s where you should be directing your efforts. If ./launcher cleanup did nothing then I’m assuming docker image prune (removes dangling images) won’t either. If you’re only using this server for discourse you may need to just check around to make sure there are no large files/folders in your home directory.

Ohh – well, that’s tricky of Docker… No, prune operations didn’t recover anything. I’m poking through /usr now with ncdu… nothing looks like it obviously doesn’t belong, though I’m not sure what to make of /usr/lib/modules: 547.2 MiB [###########################] /6.2.0-37-generic 547.2 MiB [########################## ] /6.2.0-34-generic 1.2 MiB [ ] /6.2.0-33-generic 1.2 MiB [ ] /6.2.0-32-generic 1.2 MiB [ ] /6.2.0-35-generic 1.2 MiB [ ] /6.2.0-36-generic By far the most use is reported as the overlays down in /var: 16.0 GiB [###########################] /var 4.3 GiB [####### ] /usr 2.3 GiB [### ] swapfile 1.7 GiB [## ] /snap 289.5 MiB [ ] /boot There’s nothing in /snap but what it came with: root@forum:/# snap list Name Version Rev Tracking Publisher Notes core22 20230801 864 latest/stable canonical✓ base lxd 5.19-8635f82 26200 latest/stable/… canonical✓ - snapd 2.60.4 20290 latest/stable canonical✓ snapd Whoa – /var/log/journal got big! 1.8 GiB [###########################] /7341e5ac94ae440bbd06f743e242da89 16.0 MiB [ ] /7025a9ae870140c1bef8e55211d339dc Looks like it’s been tons of bots trying to log in over just a couple of months. Seems prudent to retain logs for a while, but this forum is still beta. Maybe vacuuming that will be enough to get me going again.

Well, that didn’t quite do it, so I upgraded the server to 55G. If those big overlay files are inevitable, I guess there wasn’t really a choice. A Discourse upgrade just completed, the site appears to be working fine on 3.2.0.beta4-dev. :sweat_smile: Thank you @JammyDodger and @Stephen for your attention and input!

I was running out of space on a 50GB Linode VPS. Below are some space hogs that haven’t been mentioned yet. Some are Discourse-specific, some are general to Linux systems. Run ll /lib/modules. I was surprised to see about 15GB of old kernels that apt autoremove didn’t care to remove. Claude thinks they were installed in a non-standard way, and generated a safe removal script. It took about an hour but it worked (run at your own risk, of course) and I was able to run ./launcher rebuild without the no space left on device error. The script didn’t delete the corresponding headers in /usr/src. For that ChatGPT created another script. About half a gigabyte of space was taken by useless locales. Another GB+ was taken by the /var/lib/docker/overlay2/.../merged/home/discourse/.cache directory. Maybe a stupid question, but what exactly do the merge and diff directories contain? Can one of them be deleted safely at some point? [Docker container with fatass /merged and /diff directories]

Upgrade failed: low disk space -- excess overlay files?

Product Support Self-hosting

JammyDodger November 30, 2023, 9:29am 2

Just to tick off the basics, but you have tried ./launcher cleanup and this is what’s left after?

Topic		Replies	Views
Disk space usage of /lib/docker is high Self-hosting server-resources	11	6022	October 14, 2020
Cannot upgrade to latest version, disk full Self-hosting server-resources	10	872	February 3, 2023
Less than 5GB of disk space, can't upgrade Discourse Self-hosting server-resources	12	2592	December 9, 2021
Discourse image and installation size. Clean /var/lib/docker/overlay2? Self-hosting server-resources	10	4683	July 28, 2021
/var/lib/docker/overlay2 disk usage Very High Self-hosting server-resources	6	1305	February 19, 2019

Upgrade failed: low disk space -- excess overlay files?

Related topics