Digitalocean block storage VS amazon S3

[Bill]

I had got a warning in my command line about disc space before rebuilding my container while installing a Discourse plugin. I had to run some commands to remove some older versions of linux on my digitalocean droplet, but I’m still over 50% ssd capacity.

As my Discourse website has grown, I started looking at storage solutions. I’m currently on a 5$ digitalocean droplet and have read around about digitalocean block storage and amazon s3. I see the S3 settings built into the Discourse settings, so I’m guessing its easier to setup.

What do you use for your website? Which has better loading speeds, s3 vs digiocean blocks? What option is cheaper? Wouldn’t the free tier of s3 make it cheaper than digiocean blocks? Do you store your images AND backups? Or do you recommend another option outside of digiocean and amazon?

[itsbhanusharma (Bhanu Sharma)]

Hi,

There are a lot of Providers that offer S3 compatible storage solutions. You can use either of those. I’ve used a few and can recommend that Digital Ocean Spaces works great.

If You’re already hosting on DO, I’d recommend going with spaces as it’s fast and reliable.

However, there are some concerns relating to their CDN so you may have to consider a 3rd party CDN.

Check this topic for more information:

Using Object Storage for Uploads (S3 & Clones)

I just went over some common Object Storage providers so I can attest if they work or not with Discourse. Provider Service Name Works with Discourse? Amazon AWS S3 Yes Digital Ocean Spaces Yes Linode Object Storage Yes Google Cloud Storage Yes Scaleway Object Storage Yes Vultr Object Storage Yes BackBlaze Cloud Storage Yes* Self-hosted MinIO Yes Azure Blob Storage Flexify.IO Yes If you got a different service working, please add it to this wiki. Configuration In o…

[Falco]

S3 is obviously the more mature and more tested solution, but price wise DO offering may make sense for your community.

No matter what you pick, you must put a CDN in front of the bucket, so don’t forget that.

[Bill]

Would the free tier of Cloudflare work as the CDN in front of the bucket? I already use them with my domain settings for the free DDOS protection.

I noticed DigitalOcean advertises their Spaces service is cheaper than the other well known providers. So that’s definitely a plus.

Because the CDN that comes with DigitalOcean Spaces doesn’t work correctly with Discourse, I’m thinking I should just go with one of the other providers on your list. That way I don’t have to pay for DigitalOcean Spaces and a separate CDN service.

[md-misko]

@Bill

Because the CDN that comes with DigitalOcean Spaces doesn’t work correctly with Discourse, I’m thinking I should just go with one of the other providers on your list. That way I don’t have to pay for DigitalOcean Spaces and a separate CDN service.

I’ll post this here, because the Using Object Storage for Uploads (S3 Clones) is set to delete all replies after a month.

@Falco

@md-misko

@Falco can you share any specifics on this CDN brokenness?

It’s what is documented here:

Warning

Due to a known issue, file metadata headers like Content-Encoding are not passed through the CDN. Metadata headers are correctly set when fetching content directly from the origin.

A missing content-encoding means all our brotli encoded assets are broken.

That means you can use their Object Storage offering, but must shop for a caching CDN in another provider.

A missing content-encoding in the DO CDN implementation definitely is a bummer, but it only happens when setting all the S3 parameters inside app.yml. If those parameters are set in the site settings via web admin console, the DO only serves the S3 uploads via CDN, and the site assets are still served from the source.

It seems that this is by design, because the DISCOURSE_S3_CDN_URL env variable, if set in the app.yml, overrides the CDN setting for assets too, while this same setting, when declared in site settings only, doesn’t?

This is a bit inconsistent, but it allows the DO CDN to be used for S3 uploads only, without breaking the site:

There are two ways to do this:

• declare all S3 settings in site settings only

rails c
SiteSetting.s3_upload_bucket="<bucket_name>/<uploads_folder>"
SiteSetting.s3_backup_bucket="<bucket_name>/<backups_folder>"
SiteSetting.enable_s3_uploads=true
SiteSetting.s3_access_key_id="<key>"
SiteSetting.s3_secret_access_key="<secret_key>"
SiteSetting.s3_endpoint="https://<sfo2>.digitaloceanspaces.com"
SiteSetting.s3_cdn_url="https://<bucket_name>.<sfo2>.cdn.digitaloceanspaces.com/<uploads_folder>"
SiteSetting.backup_location="s3"

• shadow all S3 settings except DISCOURSE_S3_CDN_URL in the app.yml, and declare DO CDN in SiteSetting.s3_cdn_url

  DISCOURSE_USE_S3: true
  DISCOURSE_S3_REGION: <sfo2>
  DISCOURSE_S3_ENDPOINT: https://<sfo2>.digitaloceanspaces.com
  DISCOURSE_S3_ACCESS_KEY_ID: <key>
  DISCOURSE_S3_SECRET_ACCESS_KEY: <secret_key>
#  DISCOURSE_S3_CDN_URL: https://<bucket_name>.<sfo2>.cdn.digitaloceanspaces.com/<uploads_folder>
  DISCOURSE_S3_BUCKET: <bucket_name>/<uploads_folder>
  DISCOURSE_S3_BACKUP_BUCKET: <bucket_name>/<backup_folder>
  DISCOURSE_BACKUP_LOCATION: s3

rails c
SiteSetting.s3_cdn_url="https://<bucket_name>.<sfo2>.cdn.digitaloceanspaces.com/<uploads_folder>"

@falco if you agree with this I can update the original wiki topic

[Falco]

@md-misko

@falco if you agree with this I can update the original wiki topic

I don’t think that is a good idea.

1. All other providers, even AWS, require a CDN. DO offering a free one, that is broken, is just a a quirk we can ignore it and treat it like a standard Object Storage service.

2. I’m not sure that the implementation detail of setting S3 CDN URL in the yml and on the site setting resulting in different behaviors is something we want to promote, as it can be fixed anytime.

@Bill

Would the free tier of Cloudflare work as the CDN in front of the bucket?

I have not tested it, but I can assure it works with real CDN services, like KeyCDN, Cloudfront, StackPath, etc.