Extend S3 configuration for other s3 API compatible cloud storage solutions

Google storage has worked using the same sort of API since it existed. In-fact using googles sync tool for your S3 buckets is significantly faster (last we tested anyway, we have since moved to Google cloud as it is cheaper and faster).

I don’t understand how discourse still only supports S3. Since google buckets use the same API hopefully I can hack it in but jesus… the fact I found a thread from 2014 asking for simple backups to something other than S3 and it is still not provided (or even mentioned). Should have just said “no” instead of pretending it might happen. This thread further shows you haven’t even thought about anything but S3 since you never even knew other services have compatible apis…

So hard finding any information about discourse too. Just links to threads with out of date information. Just glanced at github, issues not even enabled (I assume to try force people to post on your forums, this is just a barrier to entry for developers, the people who would contribute to an open source project).

Every time I have to do something with discourse I just run into issues because of weird philosophies, lack of information or because the original setup we have is bad (multisite). Trying to properly setup everything now with separate servers for postgres, redis and each forum and naturally I don’t want to have files on the app servers (this is a very outdated way of handling uploads) and you provide a single (bad) option instead of it.

You missed seeing this?

IMHO this is a rather unfair attitude.

The Discourse Team has a lot on it’s plate and can’t do everything at once.
But realize that not everything needs to be done by the Discourse Team, Much code has been contributed by members of the community. Until that happens a “maybe” is the appropriate status.

That said, there are a few ways to help make a feature become reality.

  • Become a “customer” - work that is being compensated for is much more likely to receive attention than uncompensated work.
  • Make a Pull Request or write a plugin - become a contributor
  • Post a topic in Marketplace to get the interest of a community developer in doing the work
2 Likes

I’m very much +1 on bringing support for the likes of Backblaze B2 in.

Would a S3-like functionality (ie. “just works”) be easy/hard/impossible to do? I’m not familiar with Rails nor the plugin system, but I’d like to take a stab at this :slight_smile: Where should one start? Anyone interested joining forces here?

6 Likes

I’d be very interested in having a configuration option for S3 compatible storage solutions. NextCloud for example offers S3 and compatible options, both via GUI and config file: Primary Storage — Nextcloud 11 Server Administration Manual 11 alpha documentation
Has there been any progress on this feature? Any interest in picking this up?

1 Like

Just to add to the list of S3 compatible options:

It’s a minimalist open source server that you can run yourself. As you can see, they’re so minimalist that they don’t even care to support OpenGraph on their website.

2 Likes

For the record, digitalocean now supports this and we are open to extending core to allow for “any” s3 compatible location. Just need to figure out how this would be entered via site settings.

I guess you would select “custom” in the s3 location and then add a custom_s3_url or something like that.

#pr-welcome for this but I do want full details about the site setting changes to be made prior to implementing anything.

10 Likes

Just to be clear, though: Backblaze is not S3 compatible.

6 Likes

Just Curious to know if this functionality is available to be tested as me & my team pretty much testing our discourse setup prior to launch and are open to trying out such features. DO spaces looks like an attractive deal if it can be implemented with Discourse without much hassle. can anyone here shed some light on how is AmazonS3 implemented? (hardcoded stuff or propritary content or apis etc?) Surely gonna look at it.

Step 1:

Demonstrating in code how you would get aws-s3 gem to talk to a different provider. I have not researched that yet but that is the first blocker here.

This may not be trivial considering amazon are the ones maintaining that gem.

4 Likes

Azure has blob storage which is not s3 api compatible but is a similar service…

EDIT: Sorry probably shouldn’t be replying to a post years ago. Sorry to disrupt the thread.

I am late to this party. Any conclusion around the nature/ volume of work involved?

This is totally unknown until this is answered.

2 Likes

Maybe this will help? https://docs.minio.io/docs/how-to-use-aws-sdk-for-ruby-with-minio-server

Minio is s3 compatible storage, so it should be similar configuration with DO Spaces.

Trouble is Amazon control the aws-sdk gem and don’t appear to provide easy extensibility to which urls you hit.

You dont need to make changes in Amazon aws-sdk gem. As I understand, its higher level (storage), where s3 client is initialized. Changes are in the initialization params of sdk, not in the gem itself.

At first, it looks like here is the place to add custom endpoint https://github.com/discourse/discourse/blob/4f28c71b5082d8194129f10084738775b36b8ed3/lib/s3_helper.rb#L152

hmmm but that place just lets you choose “aws region” how would we enter a completely different HTTPS based endpoint there?

https://github.com/aws/aws-sdk-ruby/issues/1616#issuecomment-329991391

Looks like we can add a s3_endpoint setting that defaults to aws when empty?

8 Likes

Hello!

We use the Minio on our site as s3 compatible storage without any problems.

So to make it works, it is necessary:

Add to s3_options (/lib/s3_helper.rb) your s3 compatible storage endpoint, and force_path_style param depending on its settings:

opts = {region: obj.s3_region, endpoint: 'http://minio:9000', force_path_style: true}

and modify absolute_base_url setting (/app/models/site_setting.rb) to return this:

def self.absolute_base_url
  bucket = SiteSetting.enable_s3_uploads ? Discourse.store.s3_bucket_name : GlobalSetting.s3_bucket

  return "//files.you_site.tld/#{bucket}"
end

Also, you need to make the appropriate settings for your nginx server.

It’s work fine, but there is one problem, particularly with Minio, that should be solved - update_lifecycle - Minio does not support this.

5 Likes

So any ideas how to port this settings to original codebase?
Or how to add script to change this code after updates?

This is PR welcome, the correct way is to have a PR that adds the feature.