# 使用 TarWriter 流式传输备份
**URL:** https://meta.discourse.org/t/using-tarwriter-to-stream-backup/78909
**Category:** Development
**Tags:** performance, backups
**Created:** [2018年一月24日 20:54 UTC](https://meta.discourse.org/t/using-tarwriter-to-stream-backup/78909 "2018-01-24T20:54:03Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@riking](https://meta.discourse.org/u/riking)
#### Post date: [2018年一月24日 20:54 UTC](https://meta.discourse.org/t/using-tarwriter-to-stream-backup/78909/1 "2018-01-24T20:54:03Z")
While doing research, I found that this class is already available in Discourse via `require 'rubygems/package'`: [Class: Gem::Package::TarWriter (Ruby 2.0.0)](http://ruby-doc.org/stdlib-2.0.0/libdoc/rubygems/rdoc/Gem/Package/TarWriter.html)
Using this should allow Discourse to take backups without having more than double the required disk space available, by streaming the entire archive to disk through an in-process tar and an in-process gzip.
Usage should look like the following:
```plaintext
destination = File.open(target_filename, "wb")
gz_stream = Zlib::GzipWriter.new(destination, 5)
@tar_writer = Gem::Package::TarWriter.new(gz_stream)
log "Archiving data dump..."
FileUtils.cd(File.dirname(@dump_filename)) do
@tar_writer.add_file "dump.sql.gz", 0644 do |tf|
File.open(@dump_filename) do |df|
IO.copy_stream(df, tf)
end
end
end
rel_directory = File.join(Rails.root, "public")
upload_directory = File.join(rel_directory, "uploads", @current_db)
log "Archiving uploads..."
last_progress = Time.now
files_since_progress = 0
Dir[File.join(upload_directory, "**/*")].each do |file|
stat = File.stat(file)
relative = file.delete_prefix(rel_directory)
if stat.directory?
@tar_writer.mkdir relative, stat.mode
else
files_since_progress += 1
if files_since_progress > 100 or (last_progress < 15.seconds.ago)
log "Archiving #{file}"
files_since_progress = 0
last_progress = Time.now
end
@tar_writer.add_file relative, stat.mode do |tf|
File.open(file, "rb") { |df| IO.copy_stream(df, tf) }
end
end
end
log "Finishing up archive..."
@tar_writer.close
gz_stream.close
destination.close
remove_tmp_directory
```
The above code does not have:
- proper error reporting
- progress indicators
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [2018年一月25日 00:56 UTC](https://meta.discourse.org/t/using-tarwriter-to-stream-backup/78909/2 "2018-01-25T00:56:18Z")
Not sure, I thought @sam already had some ideas on this?
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [2018年一月25日 01:00 UTC](https://meta.discourse.org/t/using-tarwriter-to-stream-backup/78909/3 "2018-01-25T01:00:16Z")
I am open to optimising this stuff, but we have to be careful around compatibility.
As a general approach I would like us
backup db as sql \> stream to gzip \> stream to tar
backup files \> stream to cheapest gzip \> stream to same tar
This can be done by shelling out or by using a Ruby gem
---
### Author:  [@riking](https://meta.discourse.org/u/riking)
#### Post date: [2018年一月25日 01:33 UTC](https://meta.discourse.org/t/using-tarwriter-to-stream-backup/78909/4 "2018-01-25T01:33:53Z")
I think that any solution relying on `tar --append` is going to be wonky and prone to extra copies.
I should mention that this came up when I was thinking about a “more fully-featured user data checkout”.
The user\_visits and topic\_timings tables should absolutely be part of the user data checkout (see: new EU regulations, “request access to [the personal data]”), but that means writing a zip file with multiple files in it, not just a single csv. So I thought about, “how does the backup system do this?”