Rebuild issue: [Cannot set dual-stack in combination with a custom endpoint.]

Don · November 7, 2024, 8:13am

Hello

Cannot rebuild with the following issue.

I, [2024-11-07T08:00:56.758061 #1]  INFO -- : > cd /var/www/discourse && sudo -E -u discourse bundle exec rake s3:upload_assets
rake aborted!
ArgumentError: Cannot set dual-stack in combination with a custom endpoint. (ArgumentError)

          raise ArgumentError, "Cannot set dual-stack in combination with a custom endpoint."
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

FAILED

--------------------

Pups::ExecError: cd /var/www/discourse && sudo -E -u discourse bundle exec rake s3:upload_assets failed with return #<Process::Status: pid 3349 exit 1>

Location of failure: /usr/local/lib/ruby/gems/3.3.0/gems/pups-1.2.1/lib/pups/exec_command.rb:132:in `spawn'

exec failed with the params {"cd"=>"$home", "cmd"=>["sudo -E -u discourse bundle exec rake s3:upload_assets", "sudo -E -u discourse bundle exec rake s3:expire_missing_assets"]}

bootstrap failed with exit code 1

** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.

./discourse-doctor may help diagnose the problem.

567dc9e1f9a4e662de6024b8504915d8b6ef1ee2d9b4303af75323ba478679e4

Probably related commit: FIX: Use dualstack S3 endpoint for direct uploads (#29611) · discourse/discourse@0568d36 · GitHub

I use S3 object storage upload to DO Spaces. The issue is probably with this.

Configure an S3 compatible object storage provider for uploads

Configuration

In order to store Discourse static assets in your Object Storage add this configuration on your app.yml under the hooks section:
  after_assets_precompile:
    - exec:
        cd: $home
        cmd:
          - sudo -E -u discourse bundle exec rake s3:upload_assets
          - sudo -E -u discourse bundle exec rake s3:expire_missing_assets

Thanks

Don · November 7, 2024, 9:35am

I commented out these lines which makes the rebuild run successfully but the site stuck on splash I think because cannot load assets. Safe mode same…

  after_assets_precompile:
    - exec:
        cd: $home
        cmd:
          - sudo -E -u discourse bundle exec rake s3:upload_assets
          - sudo -E -u discourse bundle exec rake s3:expire_missing_assets

Any idea? I know it’s a bad timing…

nat · November 7, 2024, 9:44am

Hey Don, we’ll have a look. Thanks for reporting.

Don · November 7, 2024, 10:33am

Thanks Natalie

For quick fix I’ve rebuild with cc01555fce59e116b76c912b4c5195e111a652b2 version, which is one commit behind this FIX: Use dualstack S3 endpoint for direct uploads (#29611) · discourse/discourse@0568d36 · GitHub

pfaffman · November 7, 2024, 7:24pm

My rebuild with a Digital Ocean Spaces S3 setup started failing with this error:

, [2024-11-07T19:09:38.615466 #1]  INFO -- : > cd /var/www/discourse && sudo -E -u discourse bundle exec rake s3:upload_assets                                                   
rake aborted!                                                                                                                                                                     
ArgumentError: Cannot set dual-stack in combination with a custom endpoint. (ArgumentError)
                                                                                         
          raise ArgumentError, "Cannot set dual-stack in combination with a custom endpoint."                                                                                     
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.143.0/lib/aws-sdk-s3/endpoint_provider.rb:34:in `resolve_endpoint'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.143.0/lib/aws-sdk-s3/plugins/endpoints.rb:37:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.191.3/lib/seahorse/client/plugins/endpoint.rb:47:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/param_validator.rb:26:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.143.0/lib/aws-sdk-s3/plugins/sse_cpk.rb:24:in `call'

But omitting setting it to whatever makes the build fail like this:

                                                                                         
I, [2024-11-07T19:15:31.060936 #1]  INFO -- : > cd /var/www/discourse && sudo -E -u discourse bundle exec rake s3:upload_assets                                                   
ERROR: Ensure S3 is configured in config/discourse.conf or environment vars              
I, [2024-11-07T19:15:36.056204 #1]  INFO -- :

I found this:

github.com/hashicorp/terraform-provider-aws

Correctly ignore `use_dualstack_endpoint` when using custom service endpoint regardless of source of either setting

opened 06:43AM - 07 Jun 24 UTC

gdavison

bug

### Description The `use_dualstack_endpoint` parameter instructs the AWS SDK en…dpoint resolver to return a dual-stack endpoint. If a custom endpoint is configured for a service, the resolver returns an error. When a custom endpoint is supplied for a service, the value of `use_dualstack_endpoint` must be ignored. Test the following combinations: * Use dual-stack endpoint set on the provider, shared configuration file, or environment variable * No custom endpoint (returns default dual-stack endpoint), custom endpoint set on the provider, shared configuration file, or environment variable, base endpoint set in shared configuration file or environment variable (#32649 suggests adding base endpoint configuration on the provider) ### References See #37876 for the equivalent issue with FIPS endpoints ### Would you like to implement a fix? None

and this:

github.com/aws/aws-sdk-php

After 3.243.0 Host override cannot be combined with Dualstack, FIPS, or S3 Accelerate

opened 02:01PM - 04 Jan 23 UTC

closed 05:24PM - 05 Jan 23 UTC

kerkness

guidance response-requested

### Describe the bug After the update to version 3.243.0 and I think the start …of using Aws\EndpointV2 it would appear that the option for `use_dual_stack_endpoint` and possibly `use_fips_endpoint` are always true. This prevents multipart uploads which use accelerator and any writes that use accelerator with possible other issues as a result. When tracing through the updates made in 3.243.0 and subsequent updates I was unable to determine where the options are actually being set to true, or if the default behavior for these options is being assumed as true on the AWS end and explicitly setting them to false is being ignored. I am using Laravel 9 with league/flysystem-aws-s3-v3 but the issue appears to be isolated with changes starting with aws-sdk-php v3.243.0 and later. Explicitly setting `"aws/aws-sdk-php": "3.242.1"` in composer resolves the issue ### Expected Behavior Default values for `use_dual_stack_endpoint` and `use_fips_endpoint` is false. Setting env variables `AWS_USE_DUAL_STACK_ENDPOINT` and `AWS_USE_FIPS_ENDPOINT` to false is respected. ### Current Behavior Default values for `use_dual_stack_endpoint` and `use_fips_endpoint` appears to be true. Setting env variables `AWS_USE_DUAL_STACK_ENDPOINT` and `AWS_USE_FIPS_ENDPOINT` to false is ignored. ### Reproduction Steps Create an S3 client with default Laravel S3 config and create a multipart upload ``` $client = Storage::disk('s3')->getClient(); $result = $client->createMultipartUpload([ 'Bucket' => $bucket, 'Key' => $key, 'ContentType' => $type, 'ContentDisposition' => 'inline', ]); ``` Results in error `Host override cannot be combined with Dualstack, FIPS, or S3 Accelerate` Environment variables used ``` AWS_ACCESS_KEY_ID=***** AWS_SECRET_ACCESS_KEY=***** AWS_DEFAULT_REGION=ca-central-1 AWS_BUCKET=my-bucket-name AWS_URL="https://s3.amazonaws.com" AWS_POST_END_POINT="https://${AWS_BUCKET}.s3.amazonaws.com/" AWS_USE_PATH_STYLE_ENDPOINT=false AWS_USE_ACCELERATE_ENDPOINT=true AWS_USE_DUAL_STACK_ENDPOINT=false AWS_USE_FIPS_ENDPOINT=false ``` ### Possible Solution _No response_ ### Additional Information/Context The specific error `Host override cannot be combined with Dualstack, FIPS, or S3 Accelerate` does not appear to be common but has recently been reported with aws cli on stackoverflow indicating it may actually result from a more recent change on AWS/S3 side of things. ### SDK version used 3.243.0 ### Environment details (Version of PHP (`php -v`)? OS name and version, etc.) php 8.0.12, laravel/framerwork ^9.0, league/flysystem-aws-s3-v3 ^3.0, aws/aws-sdk-php ^3.243.0

Maybe S3_REGION should be cleared if an endpoint is set here?

github.com

discourse/discourse/blob/main/lib/file_store/to_s3_migration.rb#L48


      
                  - DISCOURSE_S3_REGION
                  and either
                  - DISCOURSE_S3_ACCESS_KEY_ID
                  - DISCOURSE_S3_SECRET_ACCESS_KEY
                  or
                  - DISCOURSE_S3_USE_IAM_PROFILE
              TEXT
            end
          
            opts = { region: ENV["DISCOURSE_S3_REGION"] }
            opts[:endpoint] = ENV["DISCOURSE_S3_ENDPOINT"] if ENV["DISCOURSE_S3_ENDPOINT"].present?
          
            if ENV["DISCOURSE_S3_USE_IAM_PROFILE"].blank?
              opts[:access_key_id] = ENV["DISCOURSE_S3_ACCESS_KEY_ID"]
              opts[:secret_access_key] = ENV["DISCOURSE_S3_SECRET_ACCESS_KEY"]
            end
          
            { client_options: opts, bucket: ENV["DISCOURSE_S3_BUCKET"] }
          end
          
          def migrate

EDIT: Oops. I thought I’d searched. Sorry Falco.

pfaffman · November 7, 2024, 7:40pm

Strangely, that gives me this error:

Gem::LoadError: can't activate webrick-1.9.0, already activated webrick-1.8.2 (Gem::LoadError)

which I’d just seen on an instance I was using to run an import, which I “fixed” by upgrading.

I guess I just can’t upgrade this server right now.

Don · November 7, 2024, 7:48pm

I am not sure but maybe worth a try. I did a rebuild succesfully before do this.

And just after this try to rebuild with pick the commit. Second time run the rebuild successfully.

Falco · November 7, 2024, 8:14pm

@martin usually what we do here is providing a GlobalSetting to disable the behavior that people running incompatible S3 clones can then opt-in to make it work.

IMO, the current behavior should stay as the default, but provide toggles to increase compatibility.

See prior art in DISCOURSE_S3_HTTP_CONTINUE_TIMEOUT, DISCOURSE_S3_INSTALL_CORS_RULE, etc.

martin · November 8, 2024, 12:13am

Ah sorry my bad here everyone, these S3 clones always slip my mind. I actually had to fix this custom endpoint issue with Minio yesterday. I think the fix here can just to be never use dualstack if DISCOURSE_S3_ENDPOINT has been set since they are incompatible, and everyone here is seemingly using that.

martin · November 8, 2024, 1:13am

I merged a fix for this now:

github.com/discourse/discourse

FIX: S3 custom endpoint incompatible with dualstack

discourse:main ← discourse:issue/fix-s3-endpoint-dualstack

opened 12:41AM - 08 Nov 24 UTC

martin-brennan

+22 -5

Followup 0568d36133081e52f25f05585c1a568c3b828d79 S3 itself and other S3-comp…atible providers do not allow using an S3 custom endpoint and dualstack at the same time, so this commit fixes that by not using dualstack when the endpoint is present. c.f. https://meta.discourse.org/t/rebuild-issue-cannot-set-dual-stack-in-combination-with-a-custom-endpoint/334902

pfaffman · November 8, 2024, 4:03pm

Thanks for the quick fix! I’m back in business.

Thanks for the lucid explanation, Rafael!

That’s because there’s a lot of stuff to keep in mind! And the spec tests, I think, would require those annoying stub_requests which I know I hate writing.

FWIW, that sounds right to me. And you could probably write tests for that that didnt’ require stubs for stub_requests.

nat · November 13, 2024, 5:49am

10 posts were split to a new topic: Unable to backup or navigate to backups

merefield · November 13, 2024, 11:29am

I don’t think this should be marked as fixed .

This change has very likely caused a break to backups on several instances (with others not knowing that their instances aren’t backing up yet presumably).

That suggests to me the specs are not broad enough to capture all the real use cases and changes are passing that shouldn’t have?

pfaffman · November 14, 2024, 5:01pm

It sounds to me like it is fixed just for sites that have use_s3 turned on but not those that are using S3 only for backups.

Though it is again possible to rebuild, it’s just that S3 backups are broken.

martin · November 15, 2024, 4:41am

The backup problem is fixed here Unable to backup or navigate to backups - #20 by martin

joffreyjaffeux · November 18, 2024, 4:42am

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Setting DISCOURSE_S3_REGION: whatever no longer works Bug	2	27	November 7, 2024
Unable to backup or navigate to backups Bug s3	23	139	November 18, 2024
Help restoring - system hung at midnight Installation	26	2204	June 13, 2022
Intended path to migrate S3 to local Installation uploads , s3	5	748	May 13, 2023
Launcher rebuild fails if s3 settings aren't correct Bug	19	4207	November 6, 2017

Rebuild issue: [Cannot set dual-stack in combination with a custom endpoint.]

Related topics