I’m still struggling getting uploads moved to Spaces. When s3_cdn_url is defined, discourse points to the s3_cdn for assets and they aren’t there. I tried `rake s3:upload_assets and get
root@shadrack-rbx:/var/www/discourse# rake s3:upload_assets
installing CORS rule
Uploading: assets/vendor-3037640def3beef9cc83cef108868f2bac887cf141d032c6b7388c7879c19601.js
OPTS: {:cache_control=>"max-age=31556952, public, immutable", :content_type=>"application/ecmascript", :acl=>"public-read", :tagging=>""} # NOTE: Pfaffman added this debug line. . .
rake aborted!
Aws::S3::Errors::InvalidArgument:
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.30.1/lib/aws-sdk-s3/plugins/sse_cpk.rb:22:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.30.1/lib/aws-sdk-s3/plugins/dualstack.rb:26:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.30.1/lib/aws-sdk-s3/plugins/accelerate.rb:35:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/plugins/idempotency_token.rb:17:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/plugins/param_converter.rb:24:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/plugins/response_paging.rb:10:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/seahorse/client/plugins/response_target.rb:23:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/seahorse/client/request.rb:70:in `send_request'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.30.1/lib/aws-sdk-s3/client.rb:5856:in `put_object'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.30.1/lib/aws-sdk-s3/object.rb:928:in `put'
/var/www/discourse/lib/s3_helper.rb:44:in `upload'
/var/www/discourse/lib/tasks/s3.rake:38:in `block in upload'
/var/www/discourse/lib/tasks/s3.rake:37:in `open'
/var/www/discourse/lib/tasks/s3.rake:37:in `upload'
/var/www/discourse/lib/tasks/s3.rake:148:in `block (2 levels) in <top (required)>'
/var/www/discourse/lib/tasks/s3.rake:147:in `each'
/var/www/discourse/lib/tasks/s3.rake:147:in `block in <top (required)>'
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'
Tasks: TOP => s3:upload_assets
(See full trace by running task with --trace)
root@shadrack-rbx:/var/www/discourse# vi config/discourse.
discourse.conf discourse.config.sample discourse.pill.sample
root@shadrack-rbx:/var/www/discourse# vi config/discourse.conf
root@shadrack-rbx:/var/www/discourse# rails c
FWIW, this config can upload images to S3 just fine.
And, if this were to work, would assets get pushed to S3 in bootstrap? I really want to get this site moved to S3 (73GB images) to move to a different server configuratoin. Spaces is appealing because I have referral credits and free CDN. But maybe I should just punt and move to AWS S3?
That’s what I thought, but what’s happening is that all the assets are linked to the S3 CDN url and they aren’t there. I’d assumed that they’d either be served by the CDN like normal (I have one site configured with regular CDN and it points to S3 cdn when that’s configured) or from local storage (I have another site configured with no regular CDN and when I have s3_cdn_url in global settings, all the assets are linked to the s3 CDN.
So, I either need to push assets to S3 or have Discourse not point to assets in S3. I don’t really care, but they need to be somewhere that they can be found, as without assets the whole site is hosed and I have to make changes in console.
No, I really mean assets (uploads seem fine). It seems that when GlobalSettings.s3_cdn_url is defined, discourse expects assets (i.e., the javascript stuff) to be in the S3 CDN, so the site is broken because none of the assets are available.
It makes zero sense to me either.
I just set SiteSettings.s3_cdn_url (not in GlobalSettings) and assets seem to still be coming from the server, not the CDN. So maybe it’s a bug with GlobalSettings.s3_cdn_url?
Yup. Re-setting the value in discourse.conf does this:
That means that if you want to use the GlobalSettings for S3 in the current state you will need to deal with the missing feature in the S3-like service in Digital Ocean, patching around it somehow.
You already have a workaround (use a normal site setting), and we know that we need a setting to disable the CORS rule (#pr-welcome for that).
I’m not sure if you (or anyone else) is still having problems with this, or if the below would resolve it, but since I came across this when troubleshooting my own issue, I thought I’d share:
If you are using Digital Ocean Spaces for S3 storage, and
You have themes with uploaded fonts, e.g. WOFF, that get copied over to the S3 storage, and
You are getting CORS errors when loading the fonts on your Discourse site, then –
Digital Ocean Spaces now have CORS Configurations settings on the Endpoint you’re using for S3 storage. Set your Discourse site URL as the Origin, GET for allowed methods, and something reasonable like 86400 for time.
Problem is that we actually use tagging for real in another rake task, so if using DO Spaces you can’t really ever use the s3:expire_missing_assets task. Which can be an “okay” trade-off for some.
I plan on updating and creating some new guides on this next week.
I’m writing up on a big experiment using DO services with Discourse (like Redis Service, PostgreSQL service, Object Store service, CDN service), so we can forward this in a concise document. I found a far worse problem already. Their S3-Clone CDN removes the Content-Encoding header of files, so you can’t serve our GZip and Brotli Javascript, and it just breaks everything.
Long term best outcome for everyone is for all the AWS S3 cloning to be actually… accurate? So if we can follow up and nag the providers to make their clones better, that’s the superior path.
I got in contact with Digital Ocean, and it appears that they are running MinIO under the hood, and minio got tagging support a month ago. They will have support for in as soon as they rollout this new version.
