Using Scaleway s3-compatible object storage

Yeah, that was a bad guess.

I can’t find a way to end with a URL like yours by reading the code responsible for it:

Is this only affecting backups? Are uploads working?

1 Like

Well, that’s different from the issue that I had with GCP buckets, but you might have a look at Trouble with Google Bucket for backup.

I was able to determine which upgrade of the aws-sdk-s3 gem caused using GCP Buckets not to work, though the solution my client chose was to switch to an AWS bucket.

@Falco yeah, I’m stumped too.
The error with the url which contains amazonaws is specifically for the file upload, not backups. For backups I only get the generic error, so I guess both are broken because of the url issue.

Can you think of anything else?

@pfaffman thanks for the tip - I’ll see if changing the version of the gem helps.

1 Like

Hey @dino, did that end up helping out? I’m having the same problem here and I’m about to give up and switch to Amazon S3.

Hey @FroggyC,
I actually ran out of time to debug this so I didn’t experiment with changing the gem vesions after all. I switched to using Amazon S3 using the official docs and things worked right away.
Sorry it’s not better news…

1 Like

Thank you. I guess we will have to do the same. Thank you!

Same problem here. Is there anything we can do to help debug this?

Like for example accessing the bucket via CLI and sending in log files?

The s3_region is ignored, right? Because Scaleway uses different values than AWS.

You might try asking scaleway – the onus of compatibility is on them. If they aren’t fully AWS S3 compatible, they should fix that.

You’re implying it is their fault, yet so far you’ve been ignoring @dino’s comment:

As long as the (untampered) s3_endpoint url isn’t used as is, it will be hard to convince Scaleway the error is on their side. Especially since other s3 clients are able to connect.

OK, prove it. Show me the documentation and log traces that demonstrate this is the case. If you can provide actual evidence the problem on our end, I will look.

Sure, that’s what I meant by

So how can I tell discourse to log its s3 connection attempts? Once we know for sure, what URL it wants to connect to, I can intercept the traffic and share the results.

1 Like

the reason the s3 upload/backup does not work is the region that needs to be set to fr-par (or nl-ams) which can only be set by circumventing discourse’s SiteSetting validation:

(via ./launcher enter app then rails c)

SiteSetting.find_by(name: 's3_region').update_attribute(:value, 'fr-par')

after this change uploads and backups work to scaleway object storage.

Scaleway documentation on Object Storage

of course, this is a workaround and once you reset or change this site setting via Web Admin you cannot set it back to a working state (unless using rails console again)

I guess the aws/s3 Client all allow explicitly setting a region string (as opposed to the current state of the webadmin)

It is also kind of misleading in case of the “EU (Paris)” dropdown value in discourse, since this refers to the aws naming scheme eu-west-3 (or sth. like that) and not the expected value for scaleway.

4 Likes

Aha. Do we need a special “s3 compatible region” site settings field @falco? That would allow people to key in completely arbitrary (and thus ‘made up’ from Amazon’s perspective) regions?

No need for that.

People on S3 clones need to set the S3 Endpoint env var, which overwrites the S3 Region.

I have an entire how to explaining that with Digital Ocean S3 Clone, but I’m waiting for Digital Ocean to fix a bug in their S3 CDN before rolling it out.

If Scaleway is in a better shape than Digital Ocean, I think I will give it a try and try to base my guide on it.

4 Likes

Right but how do they know to do that? Does the description for the existing site setting mention this? I think it should. Can you make it so?

3 Likes

Turns out most clones are broken in one way or another. For example, the Digital Ocean S3 Clone forgets the file Mime Type, so our brotli compressed JS is served as plain text and it breaks Discourse completely. So there is no big howto guides, because the experience was awful as a few months ago and everything is changing very rapidly.

I plan on doing a complete check on the ScaleWay offering next week, and write a #howto if it works.

7 Likes

:+1:
I can’t wait to read this documentation :slight_smile:

1 Like

Very interested as well. I attempted to make it work on Rails with ActiveStorage, to no avail.

I have a pending PR before I can publish the guide at

I assume almost all S3 clones who use minio under the hood, like Digital Ocean, will need this.

4 Likes

Guide is up at Using Object Storage for Uploads (S3 Clones)

6 Likes