Suggested improvements to discourse backup format

sam · January 8, 2018, 5:41am

We are double compressing database:

First time at:

discourse/discourse/blob/main/lib/backup_restore/backuper.rb#L188


      
          
          password_argument = "PGPASSWORD='#{db_conf.password}'" if db_conf.password.present?
          host_argument = "--host=#{db_conf.host}" if db_conf.host.present?
          port_argument = "--port=#{db_conf.port}" if db_conf.port.present?
          username_argument = "--username=#{db_conf.username}" if db_conf.username.present?
          
          [
            password_argument, # pass the password to pg_dump (if any)
            "pg_dump", # the pg_dump command
            "--schema=public", # only public schema
            "-T public.pg_*", # exclude tables and views whose name starts with "pg_"
            "--file='#{@dump_filename}'", # output to the dump.sql file
            "--no-owner", # do not output commands to set ownership of objects
            "--no-privileges", # prevent dumping of access privileges
            "--verbose", # specifies verbose mode
            "--compress=4", # Compression level of 4
            host_argument, # the hostname to connect to (if any)
            port_argument, # the port to connect to (if any)
            username_argument, # the username to connect as (if any)
            db_conf.database, # the name of the database to dump
          ].join(" ")

Second time at:

github.com

discourse/discourse/blob/main/lib/backup_restore/backuper.rb#L244


      
          
          log "Archiving data dump..."
          Discourse::Utils.execute_command(
            "tar",
            "--append",
            "--dereference",
            "--file",
            tar_filename,
            File.basename(@dump_filename),
            failure_message: "Failed to archive data dump.",
            chdir: File.dirname(@dump_filename),
          )
          
          add_local_uploads_to_archive(tar_filename)
          add_remote_uploads_to_archive(tar_filename) if SiteSetting.Upload.enable_s3_uploads
          
          remove_tmp_directory
          
          log "Gzipping archive, this may take a while..."
          Discourse::Utils.execute_command(
            "gzip",

Instead this really should be a simple “tar file”, with the db added to it and then a bundle of compressed files.

backup.tar

- metadata
- database.compressed
- uploads_folder.compressed

This makes it ultra cheap to read metadata (since its at the head of the tar file) and means we also stop the double compression of database.

Topic		Replies	Views
Migration version is missing from the filename – can't restore from backup Support	11	1462	June 8, 2024
Backup Suggestions - Make a large flashing sign to not rename your backups Support	6	468	March 5, 2021
Reduce local disk space needs by not (redundantly) gzipping backups Feature backups	15	751	May 6, 2025
Add option to disable backup compression Feature	29	5961	August 30, 2024
Failed to restore from backup because backup file was renamed Support	7	1036	January 6, 2019

Suggested improvements to discourse backup format

Related topics