Extend S3 configuration for other s3 API compatible cloud storage solutions

[edouardb]

It would be great to have an extra field host and set region optional to allow other S3 API compatible services to work with discourse for backup and files storage.

Update:

@sam

For the record, digitalocean now supports this and we are open to extending core to allow for “any” s3 compatible location. Just need to figure out how this would be entered via site settings.

I guess you would select “custom” in the s3 location and then add a custom_s3_url or something like that.

#pr-welcome for this but I do want full details about the site setting changes to be made prior to implementing anything.

[codinghorror (Jeff Atwood)]

What other S3 API compatible services are there? I was not aware of such a thing.

[edouardb]

For instance Riak CS which offer an S3-Compatible api, more here riakcs

[Frank]

The world could use a few more S3 type services.

https://www.backblaze.com/ is one claims to be 25 times cheaper than S3

Cloud Storage - Online Data Storage  |  Cloud Storage  |  Google Cloud (we all like Google may not be economic).

Microsoft has something fancy in Azure file storage. They can’t just say simple stuff like store your files and call them. they have to wrap it in completely stupid ‘wow them’ tech speak. http://msdn.microsoft.com/en-us/library/azure/dn167006.aspx Probably not a true S3 competetor (probable byproduct thingy).

One of these could be a true options though.

[Frank]

One more I found… Rackspace. http://www.rackspace.com/cloud/files/ (claims simple per GB pricing…).

[edouardb]

I don’t think google storage, azure and rackspace have an S3-Compatible API. Advantage of RIAK CS solution is the api which is S3-Compatible. @codinghorror, do you think a more complete S3 configuration could be possible for next version of disource ?

[kpfleming (Kevin P. Fleming)]

OpenStack provides an S3-compatible API, so anyone using an OpenStack-based cloud has it. Rackspace should offer it as well, as they are big proponents and developers of OpenStack.

[Frank]

Thank you.

@edouardb yes Rackspace is indeed an S3 competitor and they say it right on their website.

Backblaze is too. Here is an article on their blog on what they offer compared to S3. Dated March 14 2014.

100 Petabytes of Cloud Data

[Frank]

Here is another link that compares pricing between Amazon, Microsoft, and Google’s storage products.

Amazon S3, MS Azure and Google Cloud Storage Pricing Comparison

Google has the best pricing and I still can’t understand Amazon pricing at all other than it is free for a while.

Anyways the particular product you are looking for in Google-land is:

Cloud Storage - Online Data Storage  |  Cloud Storage ...

https://cloud.google.com/storage/docs/website-configuration

Here is the pricing: Cloud Storage pricing  |  Cloud Storage  |  Google Cloud

Hope this helps.

[renoirb (Renoir Boulanger)]

There is also DreamHost who offers storage with their DreamObjects service. They support both S3 and Swift credentials.

I wonder if changing this line with a configurable auth URI and maybe adjusting S3RegionSiteSetting at app/models/s3_region_site_setting.rb would be enough.

You can see their documentation and samples here. Basically the value at absolute_base_url in s3_store.rb would have a value similar to my-bucket-name.objects.dreamhost.com.

[renoirb (Renoir Boulanger)]

I tried this hook in my container/app.yml file;

hooks:
  after_code:
    - exec:
        cd: $home
        cmd:
          - sed -i "s/s3.amazonaws/objects.dreamhost/" lib/file_store/s3_store.rb

To superseed lib/file_store/s3_store.rb method absolute_base_url with a post_code hook. I changes to what I need. But the backup cannot upload to the bucket. As if something, somewhere else, caches the URL to amazon. Regardless of this change.

[calmh (Jakob Borg)]

Also LeoFS, a distributed object storage thingy with (among others) an S3 API.

[dobon]

I had a go at making discourse work with DreamHost DreamObjects… it didn’t work. With the proliferation of S3-compatible hosts, I reckon this is definitely an idea worth looking into.

[rriemann (Robert)]

Any news so far? Someone got it working with a non-Amazon S3 service?

As the safe habour agreement has officially been questioning data transfers from the EU to the US, I would prefer a S3 provider that has its servers AND its root in the EU.

[robbo]

Google storage has worked using the same sort of API since it existed. In-fact using googles sync tool for your S3 buckets is significantly faster (last we tested anyway, we have since moved to Google cloud as it is cheaper and faster).

I don’t understand how discourse still only supports S3. Since google buckets use the same API hopefully I can hack it in but jesus… the fact I found a thread from 2014 asking for simple backups to something other than S3 and it is still not provided (or even mentioned). Should have just said “no” instead of pretending it might happen. This thread further shows you haven’t even thought about anything but S3 since you never even knew other services have compatible apis…

So hard finding any information about discourse too. Just links to threads with out of date information. Just glanced at github, issues not even enabled (I assume to try force people to post on your forums, this is just a barrier to entry for developers, the people who would contribute to an open source project).

Every time I have to do something with discourse I just run into issues because of weird philosophies, lack of information or because the original setup we have is bad (multisite). Trying to properly setup everything now with separate servers for postgres, redis and each forum and naturally I don’t want to have files on the app servers (this is a very outdated way of handling uploads) and you provide a single (bad) option instead of it.

[Mittineague]

@robbo

I don’t understand how discourse still only supports S3

You missed seeing this?

https://meta.discourse.org/t/discourse-backups-to-dropbox/51176?u=mittineague

@robbo

Should have just said “no” instead of pretending it might happen.

IMHO this is a rather unfair attitude.

The Discourse Team has a lot on it’s plate and can’t do everything at once.
But realize that not everything needs to be done by the Discourse Team, Much code has been contributed by members of the community. Until that happens a “maybe” is the appropriate status.

That said, there are a few ways to help make a feature become reality.

• Become a “customer” - work that is being compensated for is much more likely to receive attention than uncompensated work.
• Make a Pull Request or write a plugin - become a contributor
• Post a topic in Marketplace to get the interest of a community developer in doing the work

[Uninen (Ville Säävuori)]

I’m very much +1 on bringing support for the likes of Backblaze B2 in.

Would a S3-like functionality (ie. “just works”) be easy/hard/impossible to do? I’m not familiar with Rails nor the plugin system, but I’d like to take a stab at this Where should one start? Anyone interested joining forces here?

[smaffulli (Stefano Maffulli)]

I’d be very interested in having a configuration option for S3 compatible storage solutions. NextCloud for example offers S3 and compatible options, both via GUI and config file: Primary Storage — Nextcloud 11 Server Administration Manual 11 alpha documentation
Has there been any progress on this feature? Any interest in picking this up?

[tophee (Christoph)]

Just to add to the list of S3 compatible options:

MinIO | Object Storage for AI

It’s a minimalist open source server that you can run yourself. As you can see, they’re so minimalist that they don’t even care to support OpenGraph on their website.

[sam (Sam Saffron)]

For the record, digitalocean now supports this and we are open to extending core to allow for “any” s3 compatible location. Just need to figure out how this would be entered via site settings.

I guess you would select “custom” in the s3 location and then add a custom_s3_url or something like that.

#pr-welcome for this but I do want full details about the site setting changes to be made prior to implementing anything.