Suggested improvements to discourse backup format

sam · 8 Gennaio 2018, 5:41am

We are double compressing database:

First time at:

discourse/discourse/blob/main/lib/backup_restore/backuper.rb#L188


      
          
          password_argument = "PGPASSWORD='#{db_conf.password}'" if db_conf.password.present?
          host_argument = "--host=#{db_conf.host}" if db_conf.host.present?
          port_argument = "--port=#{db_conf.port}" if db_conf.port.present?
          username_argument = "--username=#{db_conf.username}" if db_conf.username.present?
          
          [
            password_argument, # pass the password to pg_dump (if any)
            "pg_dump", # the pg_dump command
            "--schema=public", # only public schema
            "-T public.pg_*", # exclude tables and views whose name starts with "pg_"
            "--file='#{@dump_filename}'", # output to the dump.sql file
            "--no-owner", # do not output commands to set ownership of objects
            "--no-privileges", # prevent dumping of access privileges
            "--verbose", # specifies verbose mode
            "--compress=4", # Compression level of 4
            host_argument, # the hostname to connect to (if any)
            port_argument, # the port to connect to (if any)
            username_argument, # the username to connect as (if any)
            db_conf.database, # the name of the database to dump
          ].join(" ")

Second time at:

github.com

discourse/discourse/blob/main/lib/backup_restore/backuper.rb#L244


      
          
          log "Archiving data dump..."
          Discourse::Utils.execute_command(
            "tar",
            "--append",
            "--dereference",
            "--file",
            tar_filename,
            File.basename(@dump_filename),
            failure_message: "Failed to archive data dump.",
            chdir: File.dirname(@dump_filename),
          )
          
          add_local_uploads_to_archive(tar_filename)
          add_remote_uploads_to_archive(tar_filename) if SiteSetting.Upload.enable_s3_uploads
          
          remove_tmp_directory
          
          log "Gzipping archive, this may take a while..."
          Discourse::Utils.execute_command(
            "gzip",

Instead this really should be a simple “tar file”, with the db added to it and then a bundle of compressed files.

backup.tar

- metadata
- database.compressed
- uploads_folder.compressed

This makes it ultra cheap to read metadata (since its at the head of the tar file) and means we also stop the double compression of database.

Argomento		Risposte	Visualizzazioni
Migration version is missing from the filename – can't restore from backup Support	11	1449	Giugno 8, 2024
Backup Suggestions - Make a large flashing sign to not rename your backups Support	6	463	Marzo 5, 2021
Reduce local disk space needs by not (redundantly) gzipping backups Feature backups	15	718	Maggio 6, 2025
Add option to disable backup compression Feature	29	5935	Agosto 30, 2024
Failed to restore from backup because backup file was renamed Support	7	1020	Gennaio 6, 2019

Suggested improvements to discourse backup format

Argomenti correlati