Using Scaleway s3-compatible object storage

Falco · February 4, 2020, 8:59pm

Yeah, that was a bad guess.

I can’t find a way to end with a URL like yours by reading the code responsible for it:

discourse/discourse/blob/master/app/models/site_setting.rb#L155-L170


  def self.absolute_base_url
    url_basename = SiteSetting.s3_endpoint.split('/')[-1]
    bucket = SiteSetting.enable_s3_uploads ? Discourse.store.s3_bucket_name : GlobalSetting.s3_bucket_name


    # cf. http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region
    if SiteSetting.s3_endpoint.blank? || SiteSetting.s3_endpoint.end_with?("amazonaws.com")
      if SiteSetting.Upload.s3_region.start_with?("cn-")
        "//#{bucket}.s3.#{SiteSetting.Upload.s3_region}.amazonaws.com.cn"
      else
        "//#{bucket}.s3.dualstack.#{SiteSetting.Upload.s3_region}.amazonaws.com"
      end
    else
      "//#{bucket}.#{url_basename}"
    end
  end
end

Is this only affecting backups? Are uploads working?

pfaffman · February 4, 2020, 9:06pm

Well, that’s different from the issue that I had with GCP buckets, but you might have a look at Trouble with Google Bucket for backup.

I was able to determine which upgrade of the aws-sdk-s3 gem caused using GCP Buckets not to work, though the solution my client chose was to switch to an AWS bucket.

dino · February 4, 2020, 9:31pm

@Falco yeah, I’m stumped too.
The error with the url which contains amazonaws is specifically for the file upload, not backups. For backups I only get the generic error, so I guess both are broken because of the url issue.

Can you think of anything else?

@pfaffman thanks for the tip - I’ll see if changing the version of the gem helps.

FroggyC · February 21, 2020, 9:42pm

Hey @dino, did that end up helping out? I’m having the same problem here and I’m about to give up and switch to Amazon S3.

dino · February 22, 2020, 6:31am

Hey @FroggyC,
I actually ran out of time to debug this so I didn’t experiment with changing the gem vesions after all. I switched to using Amazon S3 using the official docs and things worked right away.
Sorry it’s not better news…

FroggyC · February 23, 2020, 10:38am

Thank you. I guess we will have to do the same. Thank you!

overheadhunter · April 3, 2020, 1:43pm

Same problem here. Is there anything we can do to help debug this?

Like for example accessing the bucket via CLI and sending in log files?

The s3_region is ignored, right? Because Scaleway uses different values than AWS.

codinghorror · April 4, 2020, 8:56am

You might try asking scaleway – the onus of compatibility is on them. If they aren’t fully AWS S3 compatible, they should fix that.

overheadhunter · April 4, 2020, 10:03am

You’re implying it is their fault, yet so far you’ve been ignoring @dino’s comment:

As long as the (untampered) s3_endpoint url isn’t used as is, it will be hard to convince Scaleway the error is on their side. Especially since other s3 clients are able to connect.

codinghorror · April 4, 2020, 10:11am

OK, prove it. Show me the documentation and log traces that demonstrate this is the case. If you can provide actual evidence the problem on our end, I will look.

overheadhunter · April 4, 2020, 1:46pm

Sure, that’s what I meant by

So how can I tell discourse to log its s3 connection attempts? Once we know for sure, what URL it wants to connect to, I can intercept the traffic and share the results.

elle · April 10, 2020, 10:21am

the reason the s3 upload/backup does not work is the region that needs to be set to fr-par (or nl-ams) which can only be set by circumventing discourse’s SiteSetting validation:

(via ./launcher enter app then rails c)

SiteSetting.find_by(name: 's3_region').update_attribute(:value, 'fr-par')

after this change uploads and backups work to scaleway object storage.

Scaleway documentation on Object Storage

of course, this is a workaround and once you reset or change this site setting via Web Admin you cannot set it back to a working state (unless using rails console again)

I guess the aws/s3 Client all allow explicitly setting a region string (as opposed to the current state of the webadmin)

It is also kind of misleading in case of the “EU (Paris)” dropdown value in discourse, since this refers to the aws naming scheme eu-west-3 (or sth. like that) and not the expected value for scaleway.

codinghorror · April 10, 2020, 5:54pm

Aha. Do we need a special “s3 compatible region” site settings field @falco? That would allow people to key in completely arbitrary (and thus ‘made up’ from Amazon’s perspective) regions?

Falco · April 10, 2020, 6:42pm

No need for that.

People on S3 clones need to set the S3 Endpoint env var, which overwrites the S3 Region.

github.com

discourse/discourse/blob/a1d660d9515fa446fea85bfea3b4c9b96ac75320/app/models/site_setting.rb#L155-L170


  def self.absolute_base_url
    url_basename = SiteSetting.s3_endpoint.split('/')[-1]
    bucket = SiteSetting.enable_s3_uploads ? Discourse.store.s3_bucket_name : GlobalSetting.s3_bucket_name


    # cf. http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region
    if SiteSetting.s3_endpoint.blank? || SiteSetting.s3_endpoint.end_with?("amazonaws.com")
      if SiteSetting.Upload.s3_region.start_with?("cn-")
        "//#{bucket}.s3.#{SiteSetting.Upload.s3_region}.amazonaws.com.cn"
      else
        "//#{bucket}.s3.dualstack.#{SiteSetting.Upload.s3_region}.amazonaws.com"
      end
    else
      "//#{bucket}.#{url_basename}"
    end
  end
end

I have an entire how to explaining that with Digital Ocean S3 Clone, but I’m waiting for Digital Ocean to fix a bug in their S3 CDN before rolling it out.

If Scaleway is in a better shape than Digital Ocean, I think I will give it a try and try to base my guide on it.

codinghorror · April 10, 2020, 6:49pm

Right but how do they know to do that? Does the description for the existing site setting mention this? I think it should. Can you make it so?

Falco · April 10, 2020, 6:58pm

Turns out most clones are broken in one way or another. For example, the Digital Ocean S3 Clone forgets the file Mime Type, so our brotli compressed JS is served as plain text and it breaks Discourse completely. So there is no big howto guides, because the experience was awful as a few months ago and everything is changing very rapidly.

I plan on doing a complete check on the ScaleWay offering next week, and write a #howto if it works.

Freez · April 16, 2020, 2:47pm

I can’t wait to read this documentation

azerqsdf · April 20, 2020, 10:50am

Very interested as well. I attempted to make it work on Rails with ActiveStorage, to no avail.

Falco · April 21, 2020, 8:17pm

I have a pending PR before I can publish the guide at

I assume almost all S3 clones who use minio under the hood, like Digital Ocean, will need this.

Falco · April 22, 2020, 10:46pm

Guide is up at Using Object Storage for Uploads (S3 Clones)