Separation of S3 backups provider from primary S3 provider

Contribute Feature

AI-generated summary

markersocial suggests separating the S3 backup provider from the primary S3 provider, highlighting several benefits. These include differing performance and security needs between static assets and backups, cost efficiency for backup storage, and flexibility in choosing providers. They note that certain providers, like Scaleway, have limitations (e.g., 1,000-part multipart uploads) that conflict with AWS SDK defaults, making it difficult to use them for backups without workarounds. Separating providers would also improve security by isolating backup data and allow for more granular access control. The suggestion aims to enhance flexibility and reliability in backup strategies.

1

I think it would be a nice addition to have the flexibility to specify a different S3 provider for backups, rather than needing to use the same provider as the main S3 bucket provider.

Primary reasons:

  • Requirements for a S3 bucket storing static assets will often differ from one storing backups. E.g. Strong network performance is important for a bucket storing static assets, but not so much for a backups bucket (beyond being reliable). Security may be a higher concern for a backups bucket.

  • For a backups bucket, the ideal provider has competitive pricing per GB storage, with egress pricing not being of much concern.

  • Easier to change providers to solve issues with specific providers. E.g. Scaleway S3 works perfectly as a primary S3 bucket for me, however with backups there are issues because they only support 1,000 part multipart uploads, where as the AWS s3 SDK uses 10,000 as the maximum. I’ve solved this using pups replacing the maximum parts in the S3 gem, but it’s quite makeshift and I need to check if the path for the file has changed on each update/rebuild. For me to solve this problem, I’d need to migrate the primary bucket in order to switch the backups bucket to a different compatible provider.

  • Better security easily, if the backup bucket provider or account is compromised, the primary bucket is elsewhere. It wouldn’t help as much in the reverse situation, if the main bucket provider was compromised and wiped for example (seeing S3 static assets aren’t included in the backups) - but at least the malicious party wouldn’t get access to the backups data.

  • May be easier to provide access to employees or contractors that is limited to either the main bucket or backup backet if the provider doesn’t have advanced access/role management per api key.

Anyhow, just a suggestion - cheers!

2 Likes