Backing up files in Object Storage

Product Support

AI-generated summary

Backing up files in Object Storage

TheDarkWizard wants to create a backup of attachments in a secondary object storage provider to follow the 3:2:1 rule. Since Discourse can’t handle this, they’re seeking ideas on how to achieve it. They’re considering using Backblaze and S3, and wondering if a script using APIs or an S3 setup can accomplish this.

Falco suggests that S3 has native support for replication, but TheDarkWizard notes that this doesn’t seem to work for replicating outside of Amazon S3, such as to Linode Object Storage or Backblaze.

Falco proposes using an AWS Lambda function to trigger on upload and put the file elsewhere, providing a link to AWS documentation. TheDarkWizard will explore this option.

1

So, I wanted to pick at the brains of any team members floating around or other community users about a use case I have: I want to use Object Storage on Discourse but I need the attachments to also be moved to a second object storage provider so that we have a backup of the data in a second entity as part of following the 3:2:1 rule.

I know that Discourse itself can’t handle this, so I just wanted some ideas as to how I should follow up on this once the files are on object storage. If it matters, I might use Backblaze as per this topic: Configure an S3 compatible object storage provider for uploads and I might use S3 as the secondary location. Should some sort of script be created that communicates via the APIs of both services, or is this something I can setup on S3 (having it pull from another location).

2

If you are using the real S3, this is something it has native support:

This is a responsibility of the Object Storage service, so I assume all “good” clones should offer similar functionality.

3

I was looking at that, but it doesn’t seem like I can use it to replicate outside of Amazon S3 into say Linode Object Storage or Backblaze.

The goal was to make sure that the data existed somewhere else apart from one just provider so that no provider could take your data hostage by shutting down an account in error or some such.

4

Sounds like a job for a Lambda then. You can write a script that triggers on upload and puts the recently touched file elsewhere.

5

Will take a look thanks!