Digital Ocean Spaces don’t implement the AWS S3 API for the CORS rule

support
#1

I’m still struggling getting uploads moved to Spaces. When s3_cdn_url is defined, discourse points to the s3_cdn for assets and they aren’t there. I tried `rake s3:upload_assets and get

root@shadrack-rbx:/var/www/discourse# rake s3:upload_assets                                                             
installing CORS rule
Uploading: assets/vendor-3037640def3beef9cc83cef108868f2bac887cf141d032c6b7388c7879c19601.js
OPTS: {:cache_control=>"max-age=31556952, public, immutable", :content_type=>"application/ecmascript", :acl=>"public-read", :tagging=>""} # NOTE: Pfaffman added this debug line. . . 
rake aborted!
Aws::S3::Errors::InvalidArgument: 
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.30.1/lib/aws-sdk-s3/plugins/sse_cpk.rb:22:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.30.1/lib/aws-sdk-s3/plugins/dualstack.rb:26:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.30.1/lib/aws-sdk-s3/plugins/accelerate.rb:35:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/plugins/idempotency_token.rb:17:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/plugins/param_converter.rb:24:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/plugins/response_paging.rb:10:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/seahorse/client/plugins/response_target.rb:23:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/seahorse/client/request.rb:70:in `send_request'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.30.1/lib/aws-sdk-s3/client.rb:5856:in `put_object'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.30.1/lib/aws-sdk-s3/object.rb:928:in `put'
/var/www/discourse/lib/s3_helper.rb:44:in `upload'
/var/www/discourse/lib/tasks/s3.rake:38:in `block in upload'
/var/www/discourse/lib/tasks/s3.rake:37:in `open'
/var/www/discourse/lib/tasks/s3.rake:37:in `upload'
/var/www/discourse/lib/tasks/s3.rake:148:in `block (2 levels) in <top (required)>'
/var/www/discourse/lib/tasks/s3.rake:147:in `each'
/var/www/discourse/lib/tasks/s3.rake:147:in `block in <top (required)>'
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'
Tasks: TOP => s3:upload_assets
(See full trace by running task with --trace)
root@shadrack-rbx:/var/www/discourse# vi config/discourse.
discourse.conf           discourse.config.sample  discourse.pill.sample    
root@shadrack-rbx:/var/www/discourse# vi config/discourse.conf
root@shadrack-rbx:/var/www/discourse# rails c

FWIW, this config can upload images to S3 just fine.

And, if this were to work, would assets get pushed to S3 in bootstrap? I really want to get this site moved to S3 (73GB images) to move to a different server configuratoin. Spaces is appealing because I have referral credits and free CDN. But maybe I should just punt and move to AWS S3?

#2

Why do you need to push assets to S3? Can’t they be served from a push CDN like normal? Uploads assets means JS files not uploads.

3 Likes
Defining DISCOURSE_S3_CDN_URL links to assets in S3 CDN URL
#3

That’s what I thought, but what’s happening is that all the assets are linked to the S3 CDN url and they aren’t there. I’d assumed that they’d either be served by the CDN like normal (I have one site configured with regular CDN and it points to S3 cdn when that’s configured) or from local storage (I have another site configured with no regular CDN and when I have s3_cdn_url in global settings, all the assets are linked to the s3 CDN.

So, I either need to push assets to S3 or have Discourse not point to assets in S3. I don’t really care, but they need to be somewhere that they can be found, as without assets the whole site is hosed and I have to make changes in console.

#4

Making sure you saw this @pfaffman because this topic makes very little sense the way it is phrased now.

Do you mean “uploads” as in “uploaded files”?

#5

No, I really mean assets (uploads seem fine). It seems that when GlobalSettings.s3_cdn_url is defined, discourse expects assets (i.e., the javascript stuff) to be in the S3 CDN, so the site is broken because none of the assets are available.

It makes zero sense to me either. :frowning_face:

I just set SiteSettings.s3_cdn_url (not in GlobalSettings) and assets seem to still be coming from the server, not the CDN. So maybe it’s a bug with GlobalSettings.s3_cdn_url?

Yup. Re-setting the value in discourse.conf does this:

image
image.png653×545 96.2 KB

All those red things link to the CDN for the assets.

I turned it back on/off and defining it as a global makes assets link to the s3_cdn_url.

So I guess the thing to do is configure those values in a plugin and hide them from the UX that way.

#6

Looks like spaces don’t implement the AWS S3 API for the CORS rule. That is a must have since assets can contain fonts, and those really need CORS.

Maybe work on a way to disable the rules insertion and took care of that manually?

@rishabh did this already with the tombstone rule to allow our S3 compatibility with more providers, so makes sense to expand on that.

4 Likes
#7

So if I want to move this site to S3 maybe I should just give in and stick this on AWS S3 and then stick CloudFlare or KeyCDN in front of it?

#8

No.

That means that if you want to use the GlobalSettings for S3 in the current state you will need to deal with the missing feature in the S3-like service in Digital Ocean, patching around it somehow.

You already have a workaround (use a normal site setting), and we know that we need a setting to disable the CORS rule (#pr-welcome for that).

3 Likes
#9

Sorry I’m dense. I’ve been working on this off and on for months.

Ah, so as long as I don’t use GlobalSettings it’ll work just fine? (Because GlobalSettings also wants assets to be in S3?)

#10

I’m not sure if you (or anyone else) is still having problems with this, or if the below would resolve it, but since I came across this when troubleshooting my own issue, I thought I’d share:

  • If you are using Digital Ocean Spaces for S3 storage, and
  • You have themes with uploaded fonts, e.g. WOFF, that get copied over to the S3 storage, and
  • You are getting CORS errors when loading the fonts on your Discourse site, then –

Digital Ocean Spaces now have CORS Configurations settings on the Endpoint you’re using for S3 storage. Set your Discourse site URL as the Origin, GET for allowed methods, and something reasonable like 86400 for time.

8 Likes
#11

So the whole topic conclusion was a misunderstanding.

I spent a full day testing Discourse compatibility with DO offerings, and the CORS thing here was a red herring.

The problem is that DO Spaces don’t implement the tagging header in their S3 Clone.

Fix for the upload_assets task is pretty simple and is in a branch.

Problem is that we actually use tagging for real in another rake task, so if using DO Spaces you can’t really ever use the s3:expire_missing_assets task. Which can be an “okay” trade-off for some.

I plan on updating and creating some new guides on this next week.

8 Likes
#12

Uggggh can we report this to DO? So they can fix their clone?

3 Likes
#13

I’m writing up on a big experiment using DO services with Discourse (like Redis Service, PostgreSQL service, Object Store service, CDN service), so we can forward this in a concise document. I found a far worse problem already. Their S3-Clone CDN removes the Content-Encoding header of files, so you can’t serve our GZip and Brotli Javascript, and it just breaks everything.

6 Likes
#14

Long term best outcome for everyone is for all the AWS S3 cloning to be actually… accurate? So if we can follow up and nag the providers to make their clones better, that’s the superior path.

giphy
giphy640×640 442 KB

2 Likes
#15

Indeed.

That would be fantastic, but I’m not holding my breath.

OTOH, They can’t fix what they don’t know is broken. Maybe this would give them the push they need.

2 Likes
#16

I got in contact with Digital Ocean, and it appears that they are running MinIO under the hood, and minio got tagging support a month ago. They will have support for in as soon as they rollout this new version.

10 Likes
#17

This is working now.

Just follow Using Object Storage for Uploads (S3 Clones)

2 Likes
#18