Digital Ocean Spaces 未针对 CORS 规则实现 AWS S3 API

[pfaffman (Jay Pfaffman)]

我仍然在努力将上传内容迁移到 Spaces。当定义了 s3_cdn_url 时，Discourse 会将资源指向 s3_cdn，但资源并不存在在那里。我尝试运行 rake s3:upload_assets，结果得到：

root@shadrack-rbx:/var/www/discourse# rake s3:upload_assets                                                             
installing CORS rule
Uploading: assets/vendor-3037640def3beef9cc83cef108868f2bac887cf141d032c6b7388c7879c19601.js
OPTS: {:cache_control=>"max-age=31556952, public, immutable", :content_type=>"application/ecmascript", :acl=>"public-read", :tagging=>""} # NOTE: Pfaffman added this debug line. . . 
rake aborted!
Aws::S3::Errors::InvalidArgument: 
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.30.1/lib/aws-sdk-s3/plugins/sse_cpk.rb:22:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.30.1/lib/aws-sdk-s3/plugins/dualstack.rb:26:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.30.1/lib/aws-sdk-s3/plugins/accelerate.rb:35:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/plugins/idempotency_token.rb:17:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/plugins/param_converter.rb:24:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/plugins/response_paging.rb:10:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/seahorse/client/plugins/response_target.rb:23:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/seahorse/client/request.rb:70:in `send_request'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.30.1/lib/aws-sdk-s3/client.rb:5856:in `put_object'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.30.1/lib/aws-sdk-s3/object.rb:928:in `put'
/var/www/discourse/lib/s3_helper.rb:44:in `upload'
/var/www/discourse/lib/tasks/s3.rake:38:in `block in upload'
/var/www/discourse/lib/tasks/s3.rake:37:in `open'
/var/www/discourse/lib/tasks/s3.rake:37:in `upload'
/var/www/discourse/lib/tasks/s3.rake:148:in `block (2 levels) in <top (required)>'
/var/www/discourse/lib/tasks/s3.rake:147:in `each'
/var/www/discourse/lib/tasks/s3.rake:147:in `block in <top (required)>'
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'
Tasks: TOP => s3:upload_assets
(See full trace by running task with --trace)
root@shadrack-rbx:/var/www/discourse# vi config/discourse.
discourse.conf           discourse.config.sample  discourse.pill.sample    
root@shadrack-rbx:/var/www/discourse# vi config/discourse.conf
root@shadrack-rbx-rbx:/var/www/discourse# rails c

顺便一提，此配置可以正常地将图片上传到 S3。

此外，如果这能正常工作，资源是否会在引导过程中被推送到 S3？我真的很想将站点迁移到 S3（包含 73GB 的图片），以便切换到不同的服务器配置。Spaces 很有吸引力，因为我有推荐积分和免费 CDN。但也许我应该直接放弃并迁移到 AWS S3？

[Falco]

Why do you need to push assets to S3? Can’t they be served from a push CDN like normal? Uploads assets means JS files not uploads.

[pfaffman (Jay Pfaffman)]

@Falco

Why do you need to push assets to S3? Can’t they be served from a push CDN like normal?

That’s what I thought, but what’s happening is that all the assets are linked to the S3 CDN url and they aren’t there. I’d assumed that they’d either be served by the CDN like normal (I have one site configured with regular CDN and it points to S3 cdn when that’s configured) or from local storage (I have another site configured with no regular CDN and when I have s3_cdn_url in global settings, all the assets are linked to the s3 CDN.

So, I either need to push assets to S3 or have Discourse not point to assets in S3. I don’t really care, but they need to be somewhere that they can be found, as without assets the whole site is hosed and I have to make changes in console.

[codinghorror (Jeff Atwood)]

@Falco

Uploads assets means JS files not uploads.

Making sure you saw this @pfaffman because this topic makes very little sense the way it is phrased now.

Do you mean “uploads” as in “uploaded files”?

[pfaffman (Jay Pfaffman)]

No, I really mean assets (uploads seem fine). It seems that when GlobalSettings.s3_cdn_url is defined, discourse expects assets (i.e., the javascript stuff) to be in the S3 CDN, so the site is broken because none of the assets are available.

@codinghorror

this topic makes very little sense the way it is phrased now.

It makes zero sense to me either.

I just set SiteSettings.s3_cdn_url (not in GlobalSettings) and assets seem to still be coming from the server, not the CDN. So maybe it’s a bug with GlobalSettings.s3_cdn_url?

Yup. Re-setting the value in discourse.conf does this:

All those red things link to the CDN for the assets.

I turned it back on/off and defining it as a global makes assets link to the s3_cdn_url.

So I guess the thing to do is configure those values in a plugin and hide them from the UX that way.

[Falco]

Looks like spaces don’t implement the AWS S3 API for the CORS rule. That is a must have since assets can contain fonts, and those really need CORS.

Maybe work on a way to disable the rules insertion and took care of that manually?

@rishabh did this already with the tombstone rule to allow our S3 compatibility with more providers, so makes sense to expand on that.

[pfaffman (Jay Pfaffman)]

So if I want to move this site to S3 maybe I should just give in and stick this on AWS S3 and then stick CloudFlare or KeyCDN in front of it?

[Falco]

No.

That means that if you want to use the GlobalSettings for S3 in the current state you will need to deal with the missing feature in the S3-like service in Digital Ocean, patching around it somehow.

You already have a workaround (use a normal site setting), and we know that we need a setting to disable the CORS rule (pr-welcome for that).

[pfaffman (Jay Pfaffman)]

Sorry I’m dense. I’ve been working on this off and on for months.

Ah, so as long as I don’t use GlobalSettings it’ll work just fine? (Because GlobalSettings also wants assets to be in S3?)

[downey (Michael Downey)]

我不确定您（或其他人）是否仍遇到此问题，或者以下方法是否能解决，但既然我在排查自己的问题时遇到了这种情况，我想分享如下：

• 如果您使用 Digital Ocean Spaces 作为 S3 存储，并且
• 您有包含已上传字体（例如 WOFF）的主题，这些字体被复制到 S3 存储中，并且
• 在您的 Discourse 站点上加载字体时出现 CORS 错误，那么——

Digital Ocean Spaces 现在在您用于 S3 存储的端点上提供了 CORS 配置设置。将您的 Discourse 站点 URL 设置为 Origin，将 allowed methods 设置为 GET，并将时间设置为一个合理的值（例如 86400）。

How to Configure CORS on DigitalOcean Spaces | DigitalOcean Documentation

[Falco]

因此，整个话题的结论其实是一场误会。

我花了一整天测试 Discourse 与 DigitalOcean（DO）服务的兼容性，而这里的 CORS 问题其实是个误导。

问题在于 DO Spaces 在其 S3 克隆实现中并未支持 tagging 头。

修复 upload_assets 任务的方法非常简单，相关代码已在分支中。

FEATURE: Digital Ocean Spaces compatibility for Object Storage

问题是，我们在另一个 rake 任务中确实实际使用了 tagging 功能。因此，如果使用的是 DO Spaces，就完全无法使用 s3:expire_missing_assets 任务。对某些用户来说，这可能是一个“可以接受”的权衡。

我计划在下周更新并创建一些新的相关指南。

[codinghorror (Jeff Atwood)]

[quote=“Falco, 帖子：11, 主题：111736”]
问题在于 DO Spaces 在其 S3 克隆实现中未支持 tagging 头部。[/quote]
唉，我们能向 DO 报告这个问题吗？这样他们就能修复他们的克隆实现。

[Falco]

我正在撰写一份关于使用 DigitalOcean 服务（如 Redis 服务、PostgreSQL 服务、对象存储服务、CDN 服务）与 Discourse 配合运行的大型实验的文档，以便将其整理成一份简洁的文档。但我已经发现了一个更严重的问题：他们的 S3 克隆版 CDN 会移除文件的 Content-Encoding 头部，导致我们无法提供 GZip 和 Brotli 压缩的 JavaScript 文件，从而彻底破坏了所有功能。

[codinghorror (Jeff Atwood)]

对所有人来说，长期的最佳结果是所有 AWS S3 克隆都真正……准确？因此，如果我们能跟进并督促提供商改进他们的克隆质量，那才是更优的路径。

[pfaffman (Jay Pfaffman)]

@codinghorror

对所有人来说，长期来看最好的结果就是所有 AWS S3 克隆都能真正……准确？

确实如此。

@codinghorror

所以，如果我们能跟进并督促提供商改进他们的克隆，那才是更优的路径。

那当然很棒，但我并不抱太大希望。

另一方面，他们无法修复自己都不知道有问题的地方。也许这能给他们所需的推动力。

[Falco]

我已与 Digital Ocean 取得联系，他们似乎正在底层运行 MinIO，而 MinIO 在一个月前已支持标签功能。一旦他们推出新版本，将很快提供对此的支持。

[Falco]

现在可以正常工作了。

只需按照 使用对象存储进行上传（S3 克隆） 操作即可。