백업 중복 압축을 피하여 로컬 디스크 공간 요구량 줄이기

[Ed_S]

The backup process creates a tar file and then applies gzip to it. There are two types of things in the tar file: an already gzipped sql dump and the contents of uploads (if requested.) In my case every upload file is already compressed: gz, gzip, gif, jpeg, png, zip. So the final gzipping gains only 1% of size.

@pfaffman

To create a backup you need to have almost twice the size of your database+uploads available to create the backup…

I believe it would be better to demand less free space.

A previous topic from 2016 mentions disabling backup compression, but it looks like the sql dump was at that time not compressed, which shifted the tradeoffs.

Add option to disable backup compression

[gerhard (Gerhard Schlager)]

I’m already working on a new backup format that removes the double compression. My hope is that it will be ready within a month or two.

[Ed_S]

Sounds great @gerhard!

[tumbano (Alberto)]

Any update on this? Thanks

[Ed_S]

@gerhard

working on a new backup format that removes the double compression

Not to bug you too much, but how is this progressing?

[gerhard (Gerhard Schlager)]

Development of that feature is currently paused and it isn’t on our current roadmap. I hope we will get to it in 2024.

[Isambard]

If I wrote a patch to accept a 0 in the compression rate to disable gzip, would that be something that you would accept?

[Ed_S]

(I’m guessing that you’d save CPU time that way, but not space, because the gzipped tar file would still be created.)

[Isambard]

I’m aiming to save cpu time. Actually, I was thinking of using the 0 as a flag that would change the code path so that it doesn’t gzip (sadly, zero is not a valid compression level supported across all gzip versions, afaik).

[Ed_S]

Hmm that wouldn’t help me at all! (Likewise others who’ve had the same problem with limited disk space.)

If tar were being used, it could be used with z or j options. If a subshell were being used, the output of tar could be piped into gzip. But I think in fact some higher level ruby functions may be in use.

[RGJ (Richard - Communiteq)]

@Ed_S

If a subshell were being used, the output of tar could be piped into gzip. But I think in fact some higher level ruby functions may be in use.

cough

lib/compression/tar.rb

[Ed_S]

Maybe it shouldn’t be too difficult… I appreciate that making changes to backup and restore must be made with great care, but I think just inlining the compression would save a lot of space requirement without any compatibility question.

From tar --help

-a, --auto-compress use archive suffix to determine the compression
-z, --gzip, --gunzip, --ungzip filter the archive through gzip

[Isambard]

Does -z actually do an in-place compression? I always assumed that it just ran gzip after the tar file is completed.

[Ed_S]

@Isambard

I always assumed…

Unwisely, in this case! The bytes which represent the uncompressed tar file never hit the disk.

[MentalNomad]

@Ed_S

–gzip

Are you saying we can simply add
"--gzip",

And it will stop requiring fully double the actually space used used the data?

[Ed_S]

@MentalNomad

Are you saying

Yes, that’s the change to the tar command.

[Ed_S]

@MentalNomad

"--gzip",
을 단순히 추가하면 된다는 말씀이신가요?

더 나은 선택지는 --zstd인 것 같지만, 그렇게 하려면 docker 이미지에도 ‘zstd’ 패키지를 설치해야 합니다.

[Ed_S]

기존 백업 파일이 *.gz 또는 *.zst 형식일 수 있다는 점을 고려할 때, 더 나은 접근 방식은 tar의 자동 감지 기능을 활용하는 것입니다.

tar --auto-compress -c -f ../file.tar.gz .
tar --auto-compress -c -f ../file.tar.zst .

물론 압축 해제 시에는 어떤 형식의 파일이 나타날지 알 수 없으므로 이 기능이 더욱 중요합니다.

현재 ruby 코드는 tar 자체가 처리할 수 있는 많은 작업을 수행하고 있습니다. 복잡해지기보다는 이 부분이 단순화되기를 바랍니다.

[Ed_S]

zstd는 압축 속도가 훨씬 빠릅니다. 이로 인해 압축이 거의 불가능한 데이터를 압축하는 데 시간을 쓰는 것이 덜 문제됩니다.

(zstd가 sql 덤프에도 사용된다면, 제 경우 10% 더 작아집니다.)

[CT075 (Cameron Wong)]

안녕하세요, 이 문제에 대해 새로운 소식이 있나요? 이 답변을 보면, 이 문제가 1년 넘게 한 줄로 수정 가능한 것으로 알려져 있는 것 같습니다. 서버에 거의 30G의 여유 공간이 있음에도 불구하고 계속 backup failed DM을 받는 것은 꽤 답답한 상황입니다.

CLA 서명 시 물리적 우편 주소를 입력해야 하지 않는다면, 제가 직접 PR을 만들었을 것입니다.