This discussion addresses the inefficiency of Discourse’s backup process, which currently creates a tar file and then applies gzip, resulting in redundant compression for already-compressed uploads (e.g., images, zips) and SQL dumps. Ed_S highlights that this double compression yields minimal size benefits while requiring nearly twice the disk space of the original data, often causing backups to fail on servers with limited storage.
gerhard initially mentioned working on a new backup format to eliminate double compression but later stated that development is paused and not on the current roadmap update. In response, users explored technical solutions. Isambard proposed a patch to disable gzip via a compression rate flag, though Ed_S noted this would save CPU time but not disk space.
The conversation shifted to using tar’s built-in compression capabilities (--gzip or --zstd) instead of separate steps. RGJ pointed out the relevant Ruby code, and Ed_S confirmed that using tar --gzip pipes data directly without writing an uncompressed tar file to disk, effectively halving the temporary disk space requirement. Ed_S also suggested --zstd as a superior option due to its speed and better compression ratios, noting it requires the zstd package in the Docker image.
Users like CT075 and adamengst expressed frustration over recurring backup failures caused by this issue. While pfaffman suggested marking it as PR-welcome due to the small fix size, he warned against breaking backups. Finally, mcwumbly indicated that the team is tinkering with related backup infrastructure and may address this directly.
The backup process creates a tar file and then applies gzip to it. There are two types of things in the tar file: an already gzipped sql dump and the contents of uploads (if requested.) In my case every upload file is already compressed: gz, gzip, gif, jpeg, png, zip. So the final gzipping gains only 1% of size.
I believe it would be better to demand less free space.
A previous topic from 2016 mentions disabling backup compression, but it looks like the sql dump was at that time not compressed, which shifted the tradeoffs.
I’m aiming to save cpu time. Actually, I was thinking of using the 0 as a flag that would change the code path so that it doesn’t gzip (sadly, zero is not a valid compression level supported across all gzip versions, afaik).
Hmm that wouldn’t help me at all! (Likewise others who’ve had the same problem with limited disk space.)
If tar were being used, it could be used with z or j options. If a subshell were being used, the output of tar could be piped into gzip. But I think in fact some higher level ruby functions may be in use.
Maybe it shouldn’t be too difficult… I appreciate that making changes to backup and restore must be made with great care, but I think just inlining the compression would save a lot of space requirement without any compatibility question.
From tar --help
-a, --auto-compress use archive suffix to determine the compression
-z, --gzip, --gunzip, --ungzip filter the archive through gzip
Possibly a better approach which might simplify things - being able to deal with existing backups which might be *.gz or *.zst - is to use tar’s automatic detection:
tar --auto-compress -c -f ../file.tar.gz .
tar --auto-compress -c -f ../file.tar.zst .
More important for the unpacking, of course, where we might not know what we’re going to see.
Presently the ruby code seems to do lots of things which tar itself can do. Hopefully this can be simplified, rather than becoming more complex.
Hi, is there any news on this? From this reply it seems like we’ve known that this is a one line fix for over a year, so it’s quite frustrating to continue to wake up to backup failed DMs despite having almost 30G free on the server.
I would make the PR myself if signing the CLA didn’t require me to enter a physical mailing address.