Suddenly having backup failures

Product Support Self-hosting

AI-generated summary

carmiac reports sudden backup failures after a recent update, with logs pointing to “No space left on device” during gzip compression, despite df -h showing 7.5GB free. The backup is destined for AWS, and user uploads total 5.9GB. Further investigation reveals /var/lib/docker/aufs consuming 15GB and /var/discourse using 12GB, with no other containers or services running. carmiac has already run launcher cleanup but remains unsure if the issue is truly disk space or something else, and asks whether /var/lib/docker/aufs can be safely cleaned. pfaffman clarifies that sufficient space is needed to hold the uncompressed database and images during backup, while pfaffman questions whether the gzip issue might be transient or related to recent bugs. Genisis confirms the problem didn’t resolve itself, and resorted to manually gzipping the backup mid-process to complete the transfer — though now faces inconsistent video upload failures even for small files (e.g., 40MB).

1

I’ve started having backup failures after a recent update. Looking through the log, it seems that the issue is caused by a lack of disk space:

[2023-09-18 20:30:34] EXCEPTION: /var/www/discourse/lib/discourse.rb:138:in `exec': Failed to gzip archive.

gzip: /var/www/discourse/public/backups/default/aoda-forum-2023-09-18-202648-v20230913194832.tar.gz: No space left on device

However, df -h gives 7.5GB free. I’m backing up to AWS, and user uploads are at 5.9GB. Digging a little deeper, I did find that /var/lib/docker/aufs is using 15GB while /var/discourse uses 12GB. There are no other containers, services, etc on this machine.

I have, of course, already run launcher cleanup.

In short, my questions are:

  1. What is going on with the backups? Is it really disk space related or something else?
  2. /var/aufs/ seems to be using a lot of space. Is there a safe way to clean that up?
2

You need enough space to hold the uncompressed database and the images.

4

Did you gzip problem solve itself? There have been adding strange issues around gzip lately.

5

nope, had was to manually gzip the backup, I let the backup run normally, go into the folder where it saves it, when its about to gzip I stoped the app quickly and gzip the backup and was finally able to change host, now im just having issues with uploading videos, even as small as 40MB (yes I know its not optimized but its inconsistent)