So, I wanted to pick at the brains of any team members floating around or other community users about a use case I have: I want to use Object Storage on Discourse but I need the attachments to also be moved to a second object storage provider so that we have a backup of the data in a second entity as part of following the 3:2:1 rule. I know that Discourse itself can’t handle this, so I just wanted some ideas as to how I should follow up on this once the files are on object storage. If it matters, I might use Backblaze as per this topic: Configure an S3 compatible object storage provider for uploads and I might use S3 as the secondary location. Should some sort of script be created that communicates via the APIs of both services, or is this something I can setup on S3 (having it pull from another location).

If you are using the real S3, this is something it has native support: Replicating objects within and across Regions - Amazon Simple Storage Service Set up and configure replication to allow automatic, asynchronous copying of objects across Amazon S3 buckets. This is a responsibility of the Object Storage service, so I assume all “good” clones should offer similar functionality.

I was looking at that, but it doesn’t seem like I can use it to replicate outside of Amazon S3 into say Linode Object Storage or Backblaze. The goal was to make sure that the data existed somewhere else apart from one just provider so that no provider could take your data hostage by shutting down an account in error or some such.

Sounds like a job for a Lambda then. You can write a script that triggers on upload and puts the recently touched file elsewhere. Process Amazon S3 event notifications with Lambda - AWS Lambda Use Lambda to process event notifications from Amazon S3.

Backing up files in Object Storage

Product Support

Falco (Falco) December 30, 2020, 6:48pm 2

If you are using the real S3, this is something it has native support:

This is a responsibility of the Object Storage service, so I assume all “good” clones should offer similar functionality.

Topic		Replies	Views
Backup request via FTP Self-hosting backups	5	487	May 15, 2024
REQ: Support S3 backup to a service like Backblaze Product Support	7	1713	August 27, 2020
Different S3 Region for Backups and Media Product Support	4	736	January 14, 2019
Automated Backups to other Destinations Besides Amazon S3 Feature docker	18	5207	October 6, 2016
Backing up your forum Product Support	5	1060	October 14, 2022

Backing up files in Object Storage

Related topics