重建问题：[无法在使用自定义端点的同时设置dual-stack。]

Don · 2024 年11 月 7 日 08:13

你好

由于以下问题，无法重建。

I, [2024-11-07T08:00:56.758061 #1]  INFO -- : cd /var/www/discourse & sudo -E -u discourse bundle exec rake s3:upload_assets
rake aborted!
ArgumentError: Cannot set dual-stack in combination with a custom endpoint. (ArgumentError)

          raise ArgumentError, "Cannot set dual-stack in combination with a custom endpoint."
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

FAILED

--------------------

Pups::ExecError: cd /var/www/discourse & sudo -E -u discourse bundle exec rake s3:upload_assets failed with return #<Process::Status: pid 3349 exit 1>

Location of failure: /usr/local/lib/ruby/gems/3.3.0/gems/pups-1.2.1/lib/pups/exec_command.rb:132:in `spawn'

exec failed with the params {"cd"=>"$home", "cmd"=>["sudo -E -u discourse bundle exec rake s3:upload_assets", "sudo -E -u discourse bundle exec rake s3:expire_missing_assets"]}

bootstrap failed with exit code 1

** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.

./discourse-doctor may help diagnose the problem.

567dc9e1f9a4e662de6024b8504915d8b6ef1ee2d9b4303af75323ba478679e4

可能相关的提交：FIX: Use dualstack S3 endpoint for direct uploads (#29611) · discourse/discourse@0568d36 · GitHub

我使用 S3 对象存储上传到 DO Spaces。问题可能出在这里。

Discourse:

Configuration

In order to store Discourse static assets in your Object Storage add this configuration on your app.yml under the hooks section:
  after_assets_precompile:
    - exec:
        cd: $home
        cmd:
          - sudo -E -u discourse bundle exec rake s3:upload_assets
          - sudo -E -u discourse bundle exec rake s3:expire_missing_assets

谢谢

Don · 2024 年11 月 7 日 09:35

我注释掉了这些行，这使得重新构建能够成功运行，但网站卡在启动画面，我认为是因为无法加载资源。安全模式也一样…

  after_assets_precompile:
    - exec:
        cd: $home
        cmd:
          - sudo -E -u discourse bundle exec rake s3:upload_assets
          - sudo -E -u discourse bundle exec rake s3:expire_missing_assets

有什么想法吗？我知道现在时机不好…

nat · 2024 年11 月 7 日 09:44

嘿 Don，我们会看看的。感谢您的报告。

Don · 2024 年11 月 7 日 10:33

谢谢 Natalie

对于快速修复，我已经用 cc01555fce59e116b76c912b4c5195e111a652b2 版本重新构建了，这个版本比这个版本 https://github.com/discourse/discourse/commit/0568d36133081e52f25f05585c1a568c3b828d79 落后一个提交。

pfaffman · 2024 年11 月 7 日 19:24

我的使用 Digital Ocean Spaces S3 设置的重建开始因以下错误而失败：

, [2024-11-07T19:09:38.615466 #1]  INFO -- : cd /var/www/discourse & sudo -E -u discourse bundle exec rake s3:upload_assets
rake aborted!
ArgumentError: Cannot set dual-stack in combination with a custom endpoint. (ArgumentError)
          raise ArgumentError, "Cannot set dual-stack in combination with a custom endpoint."
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.143.0/lib/aws-sdk-s3/endpoint_provider.rb:34:in `resolve_endpoint'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.143.0/lib/aws-sdk-s3/plugins/endpoints.rb:37:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.191.3/lib/seahorse/client/plugins/endpoint.rb:47:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/param_validator.rb:26:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.143.0/lib/aws-sdk-s3/plugins/sse_cpk.rb:24:in `call'

但是，省略将其设置为 whatever 会导致构建失败，如下所示：


I, [2024-11-07T19:15:31.060936 #1]  INFO -- : cd /var/www/discourse & sudo -E -u discourse bundle exec rake s3:upload_assets
ERROR: Ensure S3 is configured in config/discourse.conf or environment vars
I, [2024-11-07T19:15:36.056204 #1]  INFO -- :

我发现了这个：

github.com/hashicorp/terraform-provider-aws

Correctly ignore `use_dualstack_endpoint` when using custom service endpoint regardless of source of either setting

opened 06:43AM - 07 Jun 24 UTC

gdavison

bug

### Description The `use_dualstack_endpoint` parameter instructs the AWS SDK en…dpoint resolver to return a dual-stack endpoint. If a custom endpoint is configured for a service, the resolver returns an error. When a custom endpoint is supplied for a service, the value of `use_dualstack_endpoint` must be ignored. Test the following combinations: * Use dual-stack endpoint set on the provider, shared configuration file, or environment variable * No custom endpoint (returns default dual-stack endpoint), custom endpoint set on the provider, shared configuration file, or environment variable, base endpoint set in shared configuration file or environment variable (#32649 suggests adding base endpoint configuration on the provider) ### References See #37876 for the equivalent issue with FIPS endpoints ### Would you like to implement a fix? None

还有这个：

github.com/aws/aws-sdk-php

After 3.243.0 Host override cannot be combined with Dualstack, FIPS, or S3 Accelerate

opened 02:01PM - 04 Jan 23 UTC

closed 05:24PM - 05 Jan 23 UTC

kerkness

guidance response-requested

### Describe the bug After the update to version 3.243.0 and I think the start …of using Aws\EndpointV2 it would appear that the option for `use_dual_stack_endpoint` and possibly `use_fips_endpoint` are always true. This prevents multipart uploads which use accelerator and any writes that use accelerator with possible other issues as a result. When tracing through the updates made in 3.243.0 and subsequent updates I was unable to determine where the options are actually being set to true, or if the default behavior for these options is being assumed as true on the AWS end and explicitly setting them to false is being ignored. I am using Laravel 9 with league/flysystem-aws-s3-v3 but the issue appears to be isolated with changes starting with aws-sdk-php v3.243.0 and later. Explicitly setting `"aws/aws-sdk-php": "3.242.1"` in composer resolves the issue ### Expected Behavior Default values for `use_dual_stack_endpoint` and `use_fips_endpoint` is false. Setting env variables `AWS_USE_DUAL_STACK_ENDPOINT` and `AWS_USE_FIPS_ENDPOINT` to false is respected. ### Current Behavior Default values for `use_dual_stack_endpoint` and `use_fips_endpoint` appears to be true. Setting env variables `AWS_USE_DUAL_STACK_ENDPOINT` and `AWS_USE_FIPS_ENDPOINT` to false is ignored. ### Reproduction Steps Create an S3 client with default Laravel S3 config and create a multipart upload ``` $client = Storage::disk('s3')->getClient(); $result = $client->createMultipartUpload([ 'Bucket' => $bucket, 'Key' => $key, 'ContentType' => $type, 'ContentDisposition' => 'inline', ]); ``` Results in error `Host override cannot be combined with Dualstack, FIPS, or S3 Accelerate` Environment variables used ``` AWS_ACCESS_KEY_ID=***** AWS_SECRET_ACCESS_KEY=***** AWS_DEFAULT_REGION=ca-central-1 AWS_BUCKET=my-bucket-name AWS_URL="https://s3.amazonaws.com" AWS_POST_END_POINT="https://${AWS_BUCKET}.s3.amazonaws.com/" AWS_USE_PATH_STYLE_ENDPOINT=false AWS_USE_ACCELERATE_ENDPOINT=true AWS_USE_DUAL_STACK_ENDPOINT=false AWS_USE_FIPS_ENDPOINT=false ``` ### Possible Solution _No response_ ### Additional Information/Context The specific error `Host override cannot be combined with Dualstack, FIPS, or S3 Accelerate` does not appear to be common but has recently been reported with aws cli on stackoverflow indicating it may actually result from a more recent change on AWS/S3 side of things. ### SDK version used 3.243.0 ### Environment details (Version of PHP (`php -v`)? OS name and version, etc.) php 8.0.12, laravel/framerwork ^9.0, league/flysystem-aws-s3-v3 ^3.0, aws/aws-sdk-php ^3.243.0

也许 S3_REGION 应该在设置了 endpoint 时被清除？

github.com/discourse/discourse

lib/file_store/to_s3_migration.rb

main


      
                  - DISCOURSE_S3_REGION
                  and either
                  - DISCOURSE_S3_ACCESS_KEY_ID
                  - DISCOURSE_S3_SECRET_ACCESS_KEY
                  or
                  - DISCOURSE_S3_USE_IAM_PROFILE
              TEXT
            end
          
            opts = { region: ENV["DISCOURSE_S3_REGION"] }
            opts[:endpoint] = ENV["DISCOURSE_S3_ENDPOINT"] if ENV["DISCOURSE_S3_ENDPOINT"].present?
          
            if ENV["DISCOURSE_S3_USE_IAM_PROFILE"].blank?
              opts[:access_key_id] = ENV["DISCOURSE_S3_ACCESS_KEY_ID"]
              opts[:secret_access_key] = ENV["DISCOURSE_S3_SECRET_ACCESS_KEY"]
            end
          
            { client_options: opts, bucket: ENV["DISCOURSE_S3_BUCKET"] }
          end
          
          def migrate

编辑：哎呀。我以为我搜索过了。抱歉 Falco。

pfaffman · 2024 年11 月 7 日 19:40

奇怪的是，这给了我这个错误：

Gem::LoadError: can't activate webrick-1.9.0, already activated webrick-1.8.2 (Gem::LoadError)

这正是我在使用一个实例运行导入时看到的，我通过升级“修复”了它。

我猜我现在不能升级这个服务器了。

Don · 2024 年11 月 7 日 19:48

我不确定，但也许值得一试。我之前成功地进行过一次重建，然后再进行此操作。

然后就在这之后尝试使用选择的提交进行重建。第二次重建成功运行。

Falco · 2024 年11 月 7 日 20:14

@martin 通常我们在这里的做法是提供一个 GlobalSetting 来禁用该行为，以便运行不兼容的 S3 克隆的用户可以选择启用它以使其正常工作。

在我看来，当前行为应保持为默认值，但提供用于提高兼容性的开关。

请参阅 DISCOURSE_S3_HTTP_CONTINUE_TIMEOUT、DISCOURSE_S3_INSTALL_CORS_RULE 等之前的示例。

martin · 2024 年11 月 8 日 00:13

啊抱歉，各位，这是我的错。我总是记不住这些 S3 克隆。昨天我不得不修复 Minio 的这个自定义端点问题。我认为这里的修复方法是，如果设置了 DISCOURSE_S3_ENDPOINT，就永远不要使用 dualstack，因为它们不兼容，而且这里的每个人似乎都在使用它。

martin · 2024 年11 月 8 日 01:13

我已经合并了一个修复程序：

github.com/discourse/discourse

FIX: S3 custom endpoint incompatible with dualstack

main ← issue/fix-s3-endpoint-dualstack

opened 12:41AM - 08 Nov 24 UTC

martin-brennan

+22 -5

Followup 0568d36133081e52f25f05585c1a568c3b828d79 S3 itself and other S3-comp…atible providers do not allow using an S3 custom endpoint and dualstack at the same time, so this commit fixes that by not using dualstack when the endpoint is present. c.f. https://meta.discourse.org/t/rebuild-issue-cannot-set-dual-stack-in-combination-with-a-custom-endpoint/334902

pfaffman · 2024 年11 月 8 日 16:03

感谢您快速修复！我又能正常工作了。

感谢 Rafael 清晰的解释！

那是因为有很多事情需要记住！而且我认为，规范测试将需要那些烦人的 stub_request，我知道我讨厌写它们。

供参考，这听起来对我来说是正确的。而且你可能可以为此编写一些不需要 stub_request 存根的测试。

nat · 2024 年11 月 13 日 05:49

10 篇帖子已拆分为新主题：无法备份或导航到备份

merefield · 2024 年11 月 13 日 11:29

我不认为这应该被标记为#fixed。

这次更改很可能导致多个实例上的备份中断（而其他人可能还不知道他们的实例尚未备份）。

这对我来说表明，规范不够广泛，无法涵盖所有实际用例，并且不应该通过的更改却通过了？

pfaffman · 2024 年11 月 14 日 17:01

在我看来，它似乎只修复了启用了 use_s3 的网站，而没有修复仅将 S3 用于备份的网站。

虽然可以重建，但 S3 备份已损坏。

martin · 2024 年11 月 15 日 04:41

备份问题已在 Unable to backup or navigate to backups - #20 by martin 修复

j.jaffeux · 2024 年11 月 18 日 04:42

此主题在上次回复后 3 天自动关闭。不再允许新回复。

话题		回复	浏览量
Setting DISCOURSE_S3_REGION: whatever no longer works Bug	2	55	2024 年11 月 7 日
Unable to backup or navigate to backups Bug s3	23	242	2024 年11 月 18 日
Rake uploads:migrate_to_s3 error Installation s3	8	789	2020 年1 月 6 日
Configure an S3 compatible object storage provider for uploads Self-Hosting cdn , configuring , how-to , reference	265	43364	2025 年11 月 19 日
Can't rebuild due to AWS SDK gem bump and new AWS Data Integrity Protections Installation	53	933	2025 年11 月 4 日

重建问题：[无法在使用自定义端点的同时设置dual-stack。]

相关话题