Add option to disable backup compression

기여 기능
21

Note to self:

Recently hit space issues again and backups failing so this post needs revisiting soon.

Note that in order to create a backup resulting in a 12GB gzip file I require ~36GB+ of space dedicated to backups, ~24GB+ of free space:

  • 12GB for the backup from the day before
  • 12GB for the new backup file
  • ~12GB+ tar archive to be compressed into gzip file (original DB file backup + original image files

So as backup sizes increase by 1GB, the backup / space requirements are actually increasing by a ~3GB.

This assumes that you are only keeping a single previous backup - where the Discourse setting maximum backups is set to 1.

The Discourse default is 5, so using defaults I would need ~84GB dedicated to backups to allow them to work.

2개의 좋아요
22

What is the lion’s share of the backup? I assume uploaded images and so on? Wouldn’t it be easier to specify the backup is database-only, and thus make it a tiny fraction of the overall size?

(Yes, we’d still need some way to back up the images independently, but at least then the urgent need for 100GB+ of space would not be present.)

23

Yes the lions share of the backup content is “uploads”.

However a backup is not complete without the “uploads”.

My personal target is to move the images / “uploads” to Amazon S3 to avoid this issue for this specific instance, however there is still some testing to be done on a high topic / post count instance before I can trust the migration to S3, some issues already highlighted in that thread (more specifically avoiding a rebake of all posts).

I have other Discourse instances that would benefit in the backups being created in a more streamlined way.

2개의 좋아요
24

I have the same problem as this thread, I have many GB of images and while I want to migrate them to S3 from what I have read the migration script seems a bit buggy still. So, I still have images locally but am running out of disk space given the high ceiling needed to allow a backup. Even if I could delete the old backup before creating the new one it would be OK for me. In fact I have been doing that manually.

Note that the backup system also seems to be failing me on the free disk space calculation, it will fill up the whole disk before giving up, and not even delete the partial files. Then the whole computer gets unhappy. There should be a calculation to not do a backup if there is no disk space for it, taking into account the space needed for the compression etc.

Edit: I am going to run a cron job which will delete the (sole) local backup every day. That should solve my immediate problem, but I think it would also be nice to have an option to immediately delete any (local) backup that was already successfully copied to S3.

25

현재 백업 압축에 사용 중인 gzip 옵션은 무엇인가요?

주제에서 다룬 내용과 달리, 저는 더 효율적인 압축 방법을 사용하여 공간을 절약하는 데 관심이 있었습니다. SQL 덤프를 대상으로 다양한 gzip 레벨과 brotli를 사용하여 간단한 테스트를 수행해 보았습니다.

2630702226 level1.sql.gz
2276305530 level1.sql.br
2216602536 level5.sql.gz
2147212204 level9.sql.gz
2036157791 level2.br
1851831279 level4.br

알 수 있듯이, 압축 시간은 비슷한 수준이었으나 효율성 측면에서는 Brotli 레벨 4가 Gzip 레벨 5를 압도합니다. 매우 빠른 도구라는 점을 고려하면 Brotli 레벨 1의 결과도 나쁘지 않았습니다.

어쨌든, 10% 이상 향상된 효과는 꽤 흥미롭다고 생각합니다.

1개의 좋아요
26

흥미롭네요. 실제 압축 시간은 얼마였나요? :thinking:

27

Zopfli는 일반적인 gzip 압축 해제 도구와 호환되기 때문에 여기에서 더 흥미롭습니다. Brotli는 다른 압축 해제기가 필요합니다.

2개의 좋아요
28

zopfli는 매우 느려서 거대한 백업 같은 용도로는 사용하지 않을 것 같습니다. 적어도 brotli는 속도에 대해 어느 정도 최적화되어 있습니다.

3개의 좋아요
29

업로드용 백업 gzip 압축 수준

gzip 압축을 비활성화하는 방법이 있을까요? 제 업로드 파일은 대부분 이미 압축된 이미지라서, 이를 다시 압축하는 것은 리소스와 시간의 낭비입니다.

1개의 좋아요
30

관련 스레드에서:

1개의 좋아요