# Extend S3 configuration for other S3 API compatible services
**URL:** https://meta.discourse.org/t/extend-s3-configuration-for-other-s3-api-compatible-services/90371
**Category:** Feature
**Created:** [21.Июнь.2018 07:10:42 UTC](https://meta.discourse.org/t/extend-s3-configuration-for-other-s3-api-compatible-services/90371 "2018-06-21T07:10:42Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@rishabh](https://meta.discourse.org/u/rishabh)
#### Post date: [21.Июнь.2018 07:10:43 UTC](https://meta.discourse.org/t/extend-s3-configuration-for-other-s3-api-compatible-services/90371/1 "2018-06-21T07:10:43Z")
Hi everyone 😀
This feature was requested by a lot of community members in [this post](https://meta.discourse.org/t/extend-s3-configuration-for-other-s3-api-compatible-cloud-storage-solutions/20593).
I’m planning to add support for **DigitalOcean Spaces** because it’s fully S3 compatible but if the community prefers another cloud option, suggestions are welcome!
Here’s a link to the [DigitalOcean API](https://developers.digitalocean.com/documentation/spaces/#aws-s3-compatibility).
I’m excited to work on this feature but I’m new to Ruby and the code base so I would need a bit of guidance:
- It would be super helpful if someone could share some documentation or resources about modifying site settings and integrations as this would help me understand how Discourse integrations work.
- What’s an ideal development environment for testing the existing AWS features? VPS with a production Discourse instance? I wanted to know because I might have to deal with nginx etc.
To see if I’ve understood the requirements properly before I start working:
The goal here is to extend s3 configuration and use the existing s3 code for using another compatible backup service. The official **aws-sdk-s3** gem supports custom HTTP endpoints as shown [here](https://github.com/aws/aws-sdk-ruby/issues/1616#issuecomment-329991391), so for example, I would have to select “custom” in the s3 location and then add a custom\_s3\_url.
Is that correct?
---
### Author:  [@itsbhanusharma](https://meta.discourse.org/u/itsbhanusharma)
#### Post date: [21.Июнь.2018 07:14:08 UTC](https://meta.discourse.org/t/extend-s3-configuration-for-other-s3-api-compatible-services/90371/2 "2018-06-21T07:14:08Z")
Some insight here:
> [@Extend S3 configuration for other s3 API compatible cloud storage solutions](https://meta.discourse.org/t/extend-s3-configuration-for-other-s3-api-compatible-cloud-storage-solutions/20593/24):
>
> Step 1: Demonstrating in code how you would get aws-s3 gem to talk to a different provider. I have not researched that yet but that is the first blocker here. This may not be trivial considering amazon are the ones maintaining that gem.
For development, start with a local development sandbox.
Discourse can be a good starting point as your first ruby app as well.
---
### Author:  [@rishabh](https://meta.discourse.org/u/rishabh)
#### Post date: [21.Июнь.2018 07:20:29 UTC](https://meta.discourse.org/t/extend-s3-configuration-for-other-s3-api-compatible-services/90371/3 "2018-06-21T07:20:29Z")
> [@itsbhanusharma](#):
>
> Some insight here:
> [Extend S3 configuration for other s3 API compatible cloud storage solutions - #24 by sam](https://meta.discourse.org/t/extend-s3-configuration-for-other-s3-api-compatible-cloud-storage-solutions/20593/24)
Thanks, that post is actually my primary source of information! 😀
> [@](#):
>
> For development, start with a local development sandbox.
> Discourse can be a good starting point as your first ruby app as well.
Perfect, I’ve already got a local sandbox up and running so I’ll get to work.
Thank you!
---
### Author:  [@rishabh](https://meta.discourse.org/u/rishabh)
#### Post date: [22.Июнь.2018 16:01:52 UTC](https://meta.discourse.org/t/extend-s3-configuration-for-other-s3-api-compatible-services/90371/4 "2018-06-22T16:01:52Z")
> [@itsbhanusharma](#):
>
> Step 1:
>
> Demonstrating in code how you would get aws-s3 gem to talk to a different provider. I have not researched that yet but that is the first blocker here.
>
> This may not be trivial considering amazon are the ones maintaining that gem.
@sam I did the research and yes, this is well supported by aws-sdk-ruby even though the documentation is lacking. This is a demonstration of **aws-sdk-ruby** being used to upload an archive to **DigitalOcean Spaces**. This works really well and I hope this helps and gives everyone an idea about how compatible these services are.
**Steps:**
1. Create a [DigitalOcean Space](https://cloud.digitalocean.com/spaces/).
2. [Generate Access Keys](https://cloud.digitalocean.com/settings/api/) for your DigitalOcean Account.
3. Paste the Space name (_bucket_), key pair (_access\_key\_id, secret\_access\_key_), file name and path in the variables below. Run the code and the file should be visible in your Space.
```plaintext
require 'aws-sdk-s3'
name = 'test.tar.gz'
path = '/home/workspace/test.tar.gz'
# Before uploading, ensure that you've created a Space on DigitalOcean
bucket = 'discoursetest1'
# Configure an S3 Resource for use with Spaces
# Note: Generate Spaces Access Keys from cloud.digitalocean.com/settings/api/
s3 = Aws::S3::Resource.new(
access_key_id: '',
secret_access_key: '',
endpoint: 'https://nyc3.digitaloceanspaces.com',
region: 'nyc3',
)
# Add a file to a Space
obj = s3.bucket(bucket).object(name)
obj.upload_file(path)
```
_Note:
Read more about [DigitalOcean’s AWS S3 compatiblity here](https://developers.digitalocean.com/documentation/spaces/#aws-s3-compatibility) and the [AWS Ruby SDK here.](https://aws.amazon.com/sdk-for-ruby/)_
---
### Author:  [@rishabh](https://meta.discourse.org/u/rishabh)
#### Post date: [27.Июнь.2018 06:06:33 UTC](https://meta.discourse.org/t/extend-s3-configuration-for-other-s3-api-compatible-services/90371/5 "2018-06-27T06:06:33Z")
> [@Extend S3 configuration for other s3 API compatible cloud storage solutions](https://meta.discourse.org/t/extend-s3-configuration-for-other-s3-api-compatible-cloud-storage-solutions/20593/20):
>
> #pr-welcome for this but I do want full details about the site setting changes to be made prior to implementing anything.
@sam I’ve made this work by adding one optional field named _spaces\_endpoint_ to site settings.
This enables Spaces support for all existing S3 features like upload, delete etc.
Does this look fine to you?
```plaintext
spaces_endpoint:
default: ''
```
* * *
Here’s an image of the new Settings/Files page:
---
### Author:  [@riking](https://meta.discourse.org/u/riking)
#### Post date: [27.Июнь.2018 06:39:39 UTC](https://meta.discourse.org/t/extend-s3-configuration-for-other-s3-api-compatible-services/90371/6 "2018-06-27T06:39:39Z")
Can you put your code changes up as a PR on Github?
Also, that setting is being inserted as `Aws::S3::Resource.new(... endpoint ...)`, right?
The setting should be named `s3_endpoint`, default of `https://s3.amazonaws.com` – as it’s not specific to DO Spaces.
---
### Author:  [@arrowcircle](https://meta.discourse.org/u/arrowcircle)
#### Post date: [27.Июнь.2018 08:33:56 UTC](https://meta.discourse.org/t/extend-s3-configuration-for-other-s3-api-compatible-services/90371/7 "2018-06-27T08:33:56Z")
There are should be some changes in nginx template. Its better to serve files via nginx, not directly from spaces. Also, its strongly recommended to add nginx caching to make minimum number of read requests to spaces API. Also, for local development (and for some production cases) you can use [https://www.minio.io/](https://www.minio.io/) - its s3 api compatible storage software.
---
### Author:  [@rishabh](https://meta.discourse.org/u/rishabh)
#### Post date: [27.Июнь.2018 12:37:29 UTC](https://meta.discourse.org/t/extend-s3-configuration-for-other-s3-api-compatible-services/90371/8 "2018-06-27T12:37:29Z")
You can view the PR [here.](https://github.com/discourse/discourse/pull/6045)
Yes, that’s much better. I’ve changed the name to `s3_endpoint` now.
I need to check the endpoint for DO to set DO-specific parameters and that’s what I’ve done in `s3_endpoint()`. I was not sure where to put the code for configuring different platforms. Thanks for the help!
_Note: This PR is a Work in Progress._
---
### Author:  [@Falco](https://meta.discourse.org/u/Falco)
#### Post date: [27.Июнь.2018 12:58:06 UTC](https://meta.discourse.org/t/extend-s3-configuration-for-other-s3-api-compatible-services/90371/9 "2018-06-27T12:58:06Z")
> [@arrowcircle](#):
>
> There are should be some changes in nginx template. Its better to serve files via nginx, not directly from spaces. Also, its strongly recommended to add nginx caching to make minimum number of read requests to spaces API.
Shouldn’t everyone using S3-like services use a CDN? We even have a separate setting `DISCOURSE_S3_CDN` so people can have a s3-like only CDN.
Maybe after we merge the PR we can work out an optional template with nginx caching and another one using CDNs (Cloudflare should fit here, since it will be caching only statics).
---
### Author:  [@arrowcircle](https://meta.discourse.org/u/arrowcircle)
#### Post date: [27.Июнь.2018 13:40:04 UTC](https://meta.discourse.org/t/extend-s3-configuration-for-other-s3-api-compatible-services/90371/10 "2018-06-27T13:40:04Z")
Not everyone use CDN. There are many small installations that just dont need additional layer.
Also, its easy to add caching to nginx for any kind of s3-compatible storage.
And in case of minio, for small installations it can be used to directly serve images.
---
### Author:  [@Falco](https://meta.discourse.org/u/Falco)
#### Post date: [27.Июнь.2018 14:26:10 UTC](https://meta.discourse.org/t/extend-s3-configuration-for-other-s3-api-compatible-services/90371/11 "2018-06-27T14:26:10Z")
> [@rishabh](#):
>
> You can view the PR [here.](https://github.com/discourse/discourse/pull/6045)
I would like to avoid this if possible:
[https://github.com/rishabhnambiar/discourse/blob/a9bdbf962946cdbb9516b21b92434cd68c90831a/lib/s3\_helper.rb#L166-L168](https://github.com/rishabhnambiar/discourse/blob/a9bdbf962946cdbb9516b21b92434cd68c90831a/lib/s3_helper.rb#L166-L168)
Can we just ignore the region if the `opts[:endpoint]` is different than the default? Objective is allowing to use [any s3-compatible service](https://en.wikipedia.org/wiki/Amazon_S3#S3_API_and_competing_services) without depending on Discourse to add code.
---
### Author:  [@rishabh](https://meta.discourse.org/u/rishabh)
#### Post date: [27.Июнь.2018 18:28:21 UTC](https://meta.discourse.org/t/extend-s3-configuration-for-other-s3-api-compatible-services/90371/12 "2018-06-27T18:28:21Z")
I agree, that would be much cleaner.
From the Spaces API docs, I read that `region` and `endpoint ` were the required parameters.
But I just tested it again without the `region` parameter and it still works. So yes, we can ignore the region if the given endpoint is different from the default.
I’ve updated the PR, Thanks! 👍
---
### Author:  [@riking](https://meta.discourse.org/u/riking)
#### Post date: [27.Июнь.2018 20:39:48 UTC](https://meta.discourse.org/t/extend-s3-configuration-for-other-s3-api-compatible-services/90371/13 "2018-06-27T20:39:48Z")
> [@Falco](#):
>
> Can we just ignore the region if the `opts[:endpoint]` is different than the default? Objective is allowing to use [any s3-compatible service](https://en.wikipedia.org/wiki/Amazon_S3#S3_API_and_competing_services) without depending on Discourse to add code.
Maybe we should make `s3_region` a free-text field and rely on the admins to know the correct values?
---
### Author:  [@Falco](https://meta.discourse.org/u/Falco)
#### Post date: [27.Июнь.2018 20:43:35 UTC](https://meta.discourse.org/t/extend-s3-configuration-for-other-s3-api-compatible-services/90371/14 "2018-06-27T20:43:35Z")
My first choice was having something like select-kit in the tags input, select+create. Not sure we have a widget like that handy @j.jaffeux ?
---
### Author:  [@rishabh](https://meta.discourse.org/u/rishabh)
#### Post date: [27.Июнь.2018 21:08:27 UTC](https://meta.discourse.org/t/extend-s3-configuration-for-other-s3-api-compatible-services/90371/15 "2018-06-27T21:08:27Z")
> [@riking](#):
>
> Maybe we should make `s3_region` a free-text field and rely on the admins to know the correct values?
I’d like to remind everyone that DigitalOcean Spaces features work by just using an endpoint like `https://sgp1.digitaloceanspaces.com` or `https://nyc3.digitaloceanspaces.com`.
There is no need for a separate `region` field, at least for DigitalOcean Spaces support.
I will check if other services can also work in this manner, with a single endpoint.
---
### Author:  [@Falco](https://meta.discourse.org/u/Falco)
#### Post date: [27.Июнь.2018 21:21:15 UTC](https://meta.discourse.org/t/extend-s3-configuration-for-other-s3-api-compatible-services/90371/16 "2018-06-27T21:21:15Z")
> [@rishabh](#):
>
> I will check if other services can also work in this manner, with a single endpoint.
If both Google and Minio work without the region then let’s ship it.
---
### Author:  [@rishabh](https://meta.discourse.org/u/rishabh)
#### Post date: [28.Июнь.2018 15:46:48 UTC](https://meta.discourse.org/t/extend-s3-configuration-for-other-s3-api-compatible-services/90371/17 "2018-06-28T15:46:48Z")
> [@Falco](#):
>
> If both Google and Minio work without the region then let’s ship it.
All 3 platforms will not require an additional `region` field and the current PR will add support for DigitalOcean Spaces and Minio.
1. **Minio** : _endpoint format: server\_ip:9000 or user domain_
Requires `region` but it works perfectly with the pre-existing `s3_region` options (drop-down menu) in site settings so it does not need an extra `region` field.
2. **Google Cloud Platform** : _endpoint format: [https://storage.googleapis.com](https://storage.googleapis.com)_
Does not require a `region` parameter but I haven’t got it to work yet because of an incorrect header issue that I’m investigating but it will work without an extra `region` field once I debug the issue.
3. **DigitalOcean Spaces** : _endpoint format: [https://nyc3.digitaloceanspaces.com](https://nyc3.digitaloceanspaces.com)_
Works perfectly without an extra `region` field.
* * *
@Falco Before we ship it, the last problem is that Minio requires `{ force_path_style: true }` in s3\_options and AWS requires the setting to be `false.` This is because AWS and Minio use different addressing styles so if we force one, it breaks the other.
I know we wanted to avoid this but I don’t see how we can make both work without adding something specific like this:
```plaintext
if opts[:endpoint].includes? "minio"
opts[:force_path_style] = true
```
But even this won’t work because a lot of Minio users might use IP addresses to their servers instead of having “minio” in their endpoint. We have to think of a way to detect a Minio endpoint and we might need Minio specific code in Discourse if we want to support it.
We could support a lot of cloud options easily if we could have a free-text field for entering key:pair options for greater flexibility and avoiding issues like this. The `region` and `force_path_style` issues would be solved. @riking What do you feel about a field like this?
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [28.Июнь.2018 22:05:36 UTC](https://meta.discourse.org/t/extend-s3-configuration-for-other-s3-api-compatible-services/90371/18 "2018-06-28T22:05:36Z")
> [@rishabh](#):
>
> But even this won’t work because a lot of Minio users might use IP addresses to their servers instead of having “minio” in their endpoint.
Could you just ask what the service was? Or have a `force_path_style` setting and in the description say it’s true for minio and false for AWS?
---
### Author:  [@Falco](https://meta.discourse.org/u/Falco)
#### Post date: [28.Июнь.2018 23:21:05 UTC](https://meta.discourse.org/t/extend-s3-configuration-for-other-s3-api-compatible-services/90371/19 "2018-06-28T23:21:05Z")
> [@rishabh](#):
>
> @Falco Before we ship it, the last problem is that Minio requires `{ force_path_style: true }` in s3\_options and AWS requires the setting to be `false.` This is because AWS and Minio use different addressing styles so if we force one, it breaks the other.
@pfaffman solution, defaulting to false sounds good.
> [@pfaffman](#):
>
> Or have a `force_path_style` setting and in the description say it’s true for minio and false for AWS?
---
### Author:  [@rishabh](https://meta.discourse.org/u/rishabh)
#### Post date: [29.Июнь.2018 04:01:16 UTC](https://meta.discourse.org/t/extend-s3-configuration-for-other-s3-api-compatible-services/90371/20 "2018-06-29T04:01:16Z")
> [@Falco](#):
>
> @pfaffman solution, defaulting to false sounds good.
Sure, I’ve added that and the PR now works great with S3, DigitalOcean Spaces and Minio 😀
@Falco, one question: When `enable_s3_backups` is disabled and `enable_s3_uploads` is enabled, do we expect a remote backup on clicking ‘Backup’ ?
On my installation, a remote backup only occurs when `enable_s3_backups` is enabled.
[Следующая страница](https://meta.discourse.org/t/extend-s3-configuration-for-other-s3-api-compatible-services/90371.md?page=2)