Suggested improvements to discourse backup format

sam · January 8, 2018, 5:41am

We are double compressing database:

First time at:

discourse/discourse/blob/main/lib/backup_restore/backuper.rb#L188


      
          
          password_argument = "PGPASSWORD='#{db_conf.password}'" if db_conf.password.present?
          host_argument = "--host=#{db_conf.host}" if db_conf.host.present?
          port_argument = "--port=#{db_conf.port}" if db_conf.port.present?
          username_argument = "--username=#{db_conf.username}" if db_conf.username.present?
          
          [
            password_argument, # pass the password to pg_dump (if any)
            "pg_dump", # the pg_dump command
            "--schema=public", # only public schema
            "-T public.pg_*", # exclude tables and views whose name starts with "pg_"
            "--file='#{@dump_filename}'", # output to the dump.sql file
            "--no-owner", # do not output commands to set ownership of objects
            "--no-privileges", # prevent dumping of access privileges
            "--verbose", # specifies verbose mode
            "--compress=4", # Compression level of 4
            host_argument, # the hostname to connect to (if any)
            port_argument, # the port to connect to (if any)
            username_argument, # the username to connect as (if any)
            db_conf.database, # the name of the database to dump
          ].join(" ")

Second time at:

github.com

discourse/discourse/blob/main/lib/backup_restore/backuper.rb#L244


      
          
          log "Archiving data dump..."
          Discourse::Utils.execute_command(
            "tar",
            "--append",
            "--dereference",
            "--file",
            tar_filename,
            File.basename(@dump_filename),
            failure_message: "Failed to archive data dump.",
            chdir: File.dirname(@dump_filename),
          )
          
          add_local_uploads_to_archive(tar_filename)
          add_remote_uploads_to_archive(tar_filename) if SiteSetting.Upload.enable_s3_uploads
          
          remove_tmp_directory
          
          log "Gzipping archive, this may take a while..."
          Discourse::Utils.execute_command(
            "gzip",

Instead this really should be a simple “tar file”, with the db added to it and then a bundle of compressed files.

backup.tar

- metadata
- database.compressed
- uploads_folder.compressed

This makes it ultra cheap to read metadata (since its at the head of the tar file) and means we also stop the double compression of database.

Topic		Replies	Views
Reduce local disk space needs by not (redundantly) gzipping backups Feature backups	13	630	September 2, 2024
Backup Suggestions - Make a large flashing sign to not rename your backups Support	6	444	March 5, 2021
Failed to restore from the backup Installation backups	5	512	July 5, 2023
Backup file not downloaded with .tar.gz extension Support	8	4085	April 7, 2015
Modifying the database inside of a backup to remove a duplicate key tag so that it doesn't fail when restoring Installation	28	1328	August 30, 2023

Suggested improvements to discourse backup format

Related topics