Backing up files in Object Storage

Support
1

So, I wanted to pick at the brains of any team members floating around or other community users about a use case I have: I want to use Object Storage on Discourse but I need the attachments to also be moved to a second object storage provider so that we have a backup of the data in a second entity as part of following the 3:2:1 rule.

I know that Discourse itself can’t handle this, so I just wanted some ideas as to how I should follow up on this once the files are on object storage. If it matters, I might use Backblaze as per this topic: Using Object Storage for Uploads (S3 & Clones) and I might use S3 as the secondary location. Should some sort of script be created that communicates via the APIs of both services, or is this something I can setup on S3 (having it pull from another location).

1 Like
2

If you are using the real S3, this is something it has native support:

https://docs.aws.amazon.com/AmazonS3/latest/dev/replication.html

This is a responsibility of the Object Storage service, so I assume all “good” clones should offer similar functionality.

2 Likes
3

I was looking at that, but it doesn’t seem like I can use it to replicate outside of Amazon S3 into say Linode Object Storage or Backblaze.

The goal was to make sure that the data existed somewhere else apart from one just provider so that no provider could take your data hostage by shutting down an account in error or some such.

1 Like
4

Sounds like a job for a Lambda then. You can write a script that triggers on upload and puts the recently touched file elsewhere.

https://docs.aws.amazon.com/lambda/latest/dg/with-s3.html

6 Likes
5

Will take a look thanks!

2 Likes