Use Google Cloud Storage Instead of S3

Contribute Feature

AI-generated summary

The discussion revolves around using Google Cloud Storage (GCS) instead of Amazon S3 for storing Discourse backups. yyhmsg proposes modifying the AWS endpoint configuration in app.yml to use GCS’s compatibility mode. However, they ended up using a bash script to copy and remove backups manually.

downey agrees that having more options for cloud storage would be beneficial. tuanpembual shares their success in using GCS for backups by adding a special policy and using the default configuration with DISCOURSE_S3_BACKUP_BUCKET and DISCOURSE_BACKUP_LOCATION.

Mark_Schmucker suggests using Cloud Storage Transfer Service to sync backups from S3 to GCS, which doesn’t require programming but may increase storage costs. This approach allows for keeping multiple backups, with a minimal cost of $3/month for 10 GB of data.

1

I saw another post about this from '14 - but no answer and it seemed like they were making this harder than it should be…

I would like some pointers to see if there is an AWS endpoint configuration within app.yml which could be changed to allow use of google cloud storage instead of s3 - GCS has a compatability mode where all - as I understand it - would need to happen is to change the AWS endpoint:

In your existing tools or libraries, make the following changes:
Change the request endpoint to use the Google Cloud Storage request endpoint.
Replace the Amazon Web Services (AWS) access and secret key with the corresponding Google Cloud Storage access key and secret key (collectively called your Google developer key).

This could be added as an option under /admin/site_settings/category/files and option for AWS endpoint along with the access keys? or can it be done in the app.yml? [AWS_ENDPOINT = “/xxxx/xxx”] already?

Thanks!

2개의 좋아요
2

I ended up doing this, just to get it working:

/bin/bash
 This script will copy the latest discourse backup to google cloud
 Then it will remove the previous backup from google cloud

LATESTBACKUP=`ls /var/discourse/shared/standalone/backups/default/*.gz -t1 | head -n1`
PREVBACKUPWDIR=`ls /var/discourse/shared/standalone/backups/default/*.gz -1tr | head -1`
PREVBACKUP=`find $PREVBACKUPWDIR -exec basename {} \;`

/home/user/gsutil/gsutil cp $LATESTBACKUP gs://yyh-community-backups
/home/user/gsutil/gsutil rm gs://your-backup-bucket/$PREVBACKUP

Which works, but I still think greater use of google cloud storage could be used with the s3 compatibility mode as described above. But maybe I’m the only one who wants to use non amazon tools.

3개의 좋아요
3

It’d be great to have a few options “out of the box”!

4

안녕하세요

기본 설정으로 GCS 백업을 테스트하고 있습니다.

DISCOURSE_S3_BACKUP_BUCKET: falcoland-files/backup
DISCOURSE_BACKUP_LOCATION: s3

그리고 GCS에 특수 정책을 추가했습니다.

policy = {
    "serviceAccount:${google_service_account.discourse.email}" = [
      "roles/storage.legacyBucketOwner"
    ]
  }

이렇게 하면 GCS로 백업 전송에 성공합니다.

저처럼 이 설정을 적용하는 데 1주를 보낸 분들에게 도움이 되길 바랍니다.

GCS Storage

6개의 좋아요
Set up image uploads to Google Cloud Storage
Configure an S3 compatible object storage provider for uploads
5

아마도 지나치게 조심스러운 것일 수도 있지만, 저는 Discourse 백업을 AWS S3에 저장하고 있습니다(기본적으로 지원됨). 그리고 다음을 사용하여 S3에서 GCS로 백업을 동기화합니다:

프로그래밍이 필요하지 않습니다. 이 방법은 저장 비용을 약간 증가시키므로, 비용을 절감하려는 경우에는 도움이 되지 않습니다. 저는 최신 10개의 백업을 유지하고 있으며, 각각 10GB이고, Google에서 월 $3의 비용이 듭니다.