The discussion addresses excessive disk space usage during Discourse backups, specifically when uploads are already compressed. Ed_S points out that the current process creates a .tar file and then gzips it, resulting in redundant compression and requiring nearly double the disk space for the backup operation. gerhard initially announced work on a new format to remove this double compression, but later clarified that the feature was paused due to roadmap priorities.
The community explored potential fixes. Isambard proposed disabling gzip to save CPU, but Ed_S corrected that this wouldn’t solve the disk space issue unless the compression was streamed. RGJ and Ed_S identified that using tar’s built-in --gzip or --auto-compress flags would stream data directly to compression, preventing the intermediate uncompressed tar file from hitting the disk. Ed_S also suggested zstd for better speed and compression ratios.
Despite the apparent simplicity of the fix, progress stalled due to the critical nature of backup reliability. CT075 expressed frustration over repeated backup failures caused by space limits, noting the fix had been known for over a year. pfaffman marked the issue as #pr-welcome, encouraging community contributions while acknowledging the risks of breaking restore functionality.
Ultimately, david announced a merged solution via pull request #44908. This update eliminates the 2x disk space requirement by streaming compression and spreads out CPU usage. While the team considered dropping the wrapper .gz entirely, they proceeded with caution to maintain format compatibility. david confirmed the change has been stable for some time, inviting users to open new topics if issues arise.
The backup process creates a tar file and then applies gzip to it. There are two types of things in the tar file: an already gzipped sql dump and the contents of uploads (if requested.) In my case every upload file is already compressed: gz, gzip, gif, jpeg, png, zip. So the final gzipping gains only 1% of size.
I believe it would be better to demand less free space.
A previous topic from 2016 mentions disabling backup compression, but it looks like the sql dump was at that time not compressed, which shifted the tradeoffs.
I’m aiming to save cpu time. Actually, I was thinking of using the 0 as a flag that would change the code path so that it doesn’t gzip (sadly, zero is not a valid compression level supported across all gzip versions, afaik).
Hmm that wouldn’t help me at all! (Likewise others who’ve had the same problem with limited disk space.)
If tar were being used, it could be used with z or j options. If a subshell were being used, the output of tar could be piped into gzip. But I think in fact some higher level ruby functions may be in use.
Maybe it shouldn’t be too difficult… I appreciate that making changes to backup and restore must be made with great care, but I think just inlining the compression would save a lot of space requirement without any compatibility question.
From tar --help
-a, --auto-compress use archive suffix to determine the compression
-z, --gzip, --gunzip, --ungzip filter the archive through gzip
Possibly a better approach which might simplify things - being able to deal with existing backups which might be *.gz or *.zst - is to use tar’s automatic detection:
tar --auto-compress -c -f ../file.tar.gz .
tar --auto-compress -c -f ../file.tar.zst .
More important for the unpacking, of course, where we might not know what we’re going to see.
Presently the ruby code seems to do lots of things which tar itself can do. Hopefully this can be simplified, rather than becoming more complex.
Hi, is there any news on this? From this reply it seems like we’ve known that this is a one line fix for over a year, so it’s quite frustrating to continue to wake up to backup failed DMs despite having almost 30G free on the server.
I would make the PR myself if signing the CLA didn’t require me to enter a physical mailing address.