Rebuild issue: [Cannot set dual-stack in combination with a custom endpoint.]

Hello :wave:

Cannot rebuild with the following issue.

I, [2024-11-07T08:00:56.758061 #1]  INFO -- : > cd /var/www/discourse && sudo -E -u discourse bundle exec rake s3:upload_assets
rake aborted!
ArgumentError: Cannot set dual-stack in combination with a custom endpoint. (ArgumentError)

          raise ArgumentError, "Cannot set dual-stack in combination with a custom endpoint."
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FAILED

--------------------

Pups::ExecError: cd /var/www/discourse && sudo -E -u discourse bundle exec rake s3:upload_assets failed with return #<Process::Status: pid 3349 exit 1>

Location of failure: /usr/local/lib/ruby/gems/3.3.0/gems/pups-1.2.1/lib/pups/exec_command.rb:132:in `spawn'

exec failed with the params {"cd"=>"$home", "cmd"=>["sudo -E -u discourse bundle exec rake s3:upload_assets", "sudo -E -u discourse bundle exec rake s3:expire_missing_assets"]}

bootstrap failed with exit code 1

** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.

./discourse-doctor may help diagnose the problem.

567dc9e1f9a4e662de6024b8504915d8b6ef1ee2d9b4303af75323ba478679e4

Probably related commit: FIX: Use dualstack S3 endpoint for direct uploads (#29611) Ā· discourse/discourse@0568d36 Ā· GitHub

I use S3 object storage upload to DO Spaces. The issue is probably with this.

Thanks :slight_smile:

1 Like

I commented out these lines which makes the rebuild run successfully but the site stuck on splash I think because cannot load assets. :thinking: Safe mode sameā€¦

  after_assets_precompile:
    - exec:
        cd: $home
        cmd:
          - sudo -E -u discourse bundle exec rake s3:upload_assets
          - sudo -E -u discourse bundle exec rake s3:expire_missing_assets

Any idea? I know itā€™s a bad timingā€¦ :confused:

1 Like

Hey Don, weā€™ll have a look. Thanks for reporting.

1 Like

Thanks Natalie :hugs:

For quick fix Iā€™ve rebuild with cc01555fce59e116b76c912b4c5195e111a652b2 version, which is one commit behind this FIX: Use dualstack S3 endpoint for direct uploads (#29611) Ā· discourse/discourse@0568d36 Ā· GitHub

2 Likes

My rebuild with a Digital Ocean Spaces S3 setup started failing with this error:

, [2024-11-07T19:09:38.615466 #1]  INFO -- : > cd /var/www/discourse && sudo -E -u discourse bundle exec rake s3:upload_assets                                                   
rake aborted!                                                                                                                                                                     
ArgumentError: Cannot set dual-stack in combination with a custom endpoint. (ArgumentError)
                                                                                         
          raise ArgumentError, "Cannot set dual-stack in combination with a custom endpoint."                                                                                     
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.143.0/lib/aws-sdk-s3/endpoint_provider.rb:34:in `resolve_endpoint'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.143.0/lib/aws-sdk-s3/plugins/endpoints.rb:37:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.191.3/lib/seahorse/client/plugins/endpoint.rb:47:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/param_validator.rb:26:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.143.0/lib/aws-sdk-s3/plugins/sse_cpk.rb:24:in `call'

But omitting setting it to whatever makes the build fail like this:

                                                                                         
I, [2024-11-07T19:15:31.060936 #1]  INFO -- : > cd /var/www/discourse && sudo -E -u discourse bundle exec rake s3:upload_assets                                                   
ERROR: Ensure S3 is configured in config/discourse.conf or environment vars              
I, [2024-11-07T19:15:36.056204 #1]  INFO -- :                                            
                                                                                         
                                           

I found this:

and this:

Maybe S3_REGION should be cleared if an endpoint is set here?

EDIT: Oops. I thought Iā€™d searched. Sorry Falco.

1 Like

Strangely, that gives me this error:

Gem::LoadError: can't activate webrick-1.9.0, already activated webrick-1.8.2 (Gem::LoadError)

which Iā€™d just seen on an instance I was using to run an import, which I ā€œfixedā€ by upgrading.

I guess I just canā€™t upgrade this server right now. :frowning:

I am not sure but maybe worth a try. I did a rebuild succesfully before do this.

And just after this try to rebuild with pick the commit. Second time run the rebuild successfully.

1 Like

@martin usually what we do here is providing a GlobalSetting to disable the behavior that people running incompatible S3 clones can then opt-in to make it work.

IMO, the current behavior should stay as the default, but provide toggles to increase compatibility.

See prior art in DISCOURSE_S3_HTTP_CONTINUE_TIMEOUT, DISCOURSE_S3_INSTALL_CORS_RULE, etc.

3 Likes

Ah sorry my bad here everyone, these S3 clones always slip my mind. I actually had to fix this custom endpoint issue with Minio yesterday. I think the fix here can just to be never use dualstack if DISCOURSE_S3_ENDPOINT has been set since they are incompatible, and everyone here is seemingly using that.

3 Likes

I merged a fix for this now:

4 Likes

Thanks for the quick fix! Iā€™m back in business. :tada:

Thanks for the lucid explanation, Rafael!

Thatā€™s because thereā€™s a lot of stuff to keep in mind! And the spec tests, I think, would require those annoying stub_requests which I know I hate writing.

FWIW, that sounds right to me. And you could probably write tests for that that didntā€™ require stubs for stub_requests.

1 Like

10 posts were split to a new topic: Unable to backup or navigate to backups

I donā€™t think this should be marked as fixed .

This change has very likely caused a break to backups on several instances (with others not knowing that their instances arenā€™t backing up yet presumably).

That suggests to me the specs are not broad enough to capture all the real use cases and changes are passing that shouldnā€™t have?

4 Likes

It sounds to me like it is fixed just for sites that have use_s3 turned on but not those that are using S3 only for backups.

Though it is again possible to rebuild, itā€™s just that S3 backups are broken. :crying_cat_face:

3 Likes

The backup problem is fixed here Unable to backup or navigate to backups - #20 by martin

3 Likes

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.