# Rebuild issue: \[Cannot set dual-stack in combination with a custom endpoint.\]

**URL:** https://meta.discourse.org/t/rebuild-issue-cannot-set-dual-stack-in-combination-with-a-custom-endpoint/334902
**Category:** Bug
**Tags:** s3, fixed
**Created:** [November 7, 2024, 8:13am UTC](https://meta.discourse.org/t/rebuild-issue-cannot-set-dual-stack-in-combination-with-a-custom-endpoint/334902 "2024-11-07T08:13:14Z")
**Posts on this page:** 16
**Page:** 1

<div class="post-metadata">

### Author: ![Don](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/don/32/228726_2.png) [@Don](https://meta.discourse.org/u/Don)
#### Post date: [November 7, 2024, 8:13am UTC](https://meta.discourse.org/t/rebuild-issue-cannot-set-dual-stack-in-combination-with-a-custom-endpoint/334902/1 "2024-11-07T08:13:14Z")

</div>

Hello 👋

Cannot rebuild with the following issue.

```plaintext
I, [2024-11-07T08:00:56.758061 #1] INFO -- : > cd /var/www/discourse && sudo -E -u discourse bundle exec rake s3:upload_assets
rake aborted!
ArgumentError: Cannot set dual-stack in combination with a custom endpoint. (ArgumentError)

          raise ArgumentError, "Cannot set dual-stack in combination with a custom endpoint."
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

```

```plaintext
FAILED

--------------------

Pups::ExecError: cd /var/www/discourse && sudo -E -u discourse bundle exec rake s3:upload_assets failed with return #<Process::Status: pid 3349 exit 1>

Location of failure: /usr/local/lib/ruby/gems/3.3.0/gems/pups-1.2.1/lib/pups/exec_command.rb:132:in `spawn'

exec failed with the params {"cd"=>"$home", "cmd"=>["sudo -E -u discourse bundle exec rake s3:upload_assets", "sudo -E -u discourse bundle exec rake s3:expire_missing_assets"]}

bootstrap failed with exit code 1

**FAILED TO BOOTSTRAP** please scroll up and look for earlier error messages, there may be more than one.

./discourse-doctor may help diagnose the problem.

567dc9e1f9a4e662de6024b8504915d8b6ef1ee2d9b4303af75323ba478679e4

```

Probably related commit: [FIX: Use dualstack S3 endpoint for direct uploads (#29611) · discourse/discourse@0568d36 · GitHub](https://github.com/discourse/discourse/commit/0568d36133081e52f25f05585c1a568c3b828d79)

I use [S3 object storage](https://meta.discourse.org/t/configure-an-s3-compatible-object-storage-provider-for-uploads/148916) upload to DO Spaces. The issue is probably with this.

> [@Configure an S3 compatible object storage provider for uploads](https://meta.discourse.org/t/configure-an-s3-compatible-object-storage-provider-for-uploads/148916/1):
>
> ## Configuration
> 
> In order to store Discourse static assets in your Object Storage add this configuration on your app.yml under the `hooks` section:
> 
> ```plaintext
> after_assets_precompile:
> - exec:
> cd: $home
> cmd:
> - sudo -E -u discourse bundle exec rake s3:upload_assets
> - sudo -E -u discourse bundle exec rake s3:expire_missing_assets
> 
> ```

Thanks 🙂

---

<div class="post-metadata">

### Author: ![Don](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/don/32/228726_2.png) [@Don](https://meta.discourse.org/u/Don)
#### Post date: [November 7, 2024, 9:35am UTC](https://meta.discourse.org/t/rebuild-issue-cannot-set-dual-stack-in-combination-with-a-custom-endpoint/334902/2 "2024-11-07T09:35:10Z")

</div>

I commented out these lines which makes the rebuild run successfully but the site stuck on splash I think because cannot load assets. 🤔 [Safe mode](https://meta.discourse.org/t/53504?silent=true) same…

```plaintext
  after_assets_precompile:
    - exec:
        cd: $home
        cmd:
          - sudo -E -u discourse bundle exec rake s3:upload_assets
          - sudo -E -u discourse bundle exec rake s3:expire_missing_assets

```

Any idea? I know it’s a bad timing… 😕

---

<div class="post-metadata">

### Author: ![nat](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/nat/32/235063_2.png) [@nat](https://meta.discourse.org/u/nat)
#### Post date: [November 7, 2024, 9:44am UTC](https://meta.discourse.org/t/rebuild-issue-cannot-set-dual-stack-in-combination-with-a-custom-endpoint/334902/3 "2024-11-07T09:44:18Z")

</div>

Hey Don, we’ll have a look. Thanks for reporting.

---

<div class="post-metadata">

### Author: ![Don](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/don/32/228726_2.png) [@Don](https://meta.discourse.org/u/Don)
#### Post date: [November 7, 2024, 10:33am UTC](https://meta.discourse.org/t/rebuild-issue-cannot-set-dual-stack-in-combination-with-a-custom-endpoint/334902/5 "2024-11-07T10:33:28Z")

</div>

Thanks Natalie 🤗

For _quick fix_ I’ve rebuild with [cc01555fce59e116b76c912b4c5195e111a652b2](https://github.com/discourse/discourse/commit/cc01555fce59e116b76c912b4c5195e111a652b2) version, which is one commit behind this [FIX: Use dualstack S3 endpoint for direct uploads (#29611) · discourse/discourse@0568d36 · GitHub](https://github.com/discourse/discourse/commit/0568d36133081e52f25f05585c1a568c3b828d79)

---

<div class="post-metadata">

### Author: ![pfaffman](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/pfaffman/32/120154_2.png) [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [November 7, 2024, 7:24pm UTC](https://meta.discourse.org/t/rebuild-issue-cannot-set-dual-stack-in-combination-with-a-custom-endpoint/334902/6 "2024-11-07T19:24:07Z")

</div>

My rebuild with a Digital Ocean Spaces S3 setup started failing with this error:

```plaintext
, [2024-11-07T19:09:38.615466 #1] INFO -- : > cd /var/www/discourse && sudo -E -u discourse bundle exec rake s3:upload_assets                                                   
rake aborted!                                                                                                                                                                     
ArgumentError: Cannot set dual-stack in combination with a custom endpoint. (ArgumentError)
                                                                                         
          raise ArgumentError, "Cannot set dual-stack in combination with a custom endpoint."                                                                                     
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.143.0/lib/aws-sdk-s3/endpoint_provider.rb:34:in `resolve_endpoint'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.143.0/lib/aws-sdk-s3/plugins/endpoints.rb:37:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.191.3/lib/seahorse/client/plugins/endpoint.rb:47:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/param_validator.rb:26:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.143.0/lib/aws-sdk-s3/plugins/sse_cpk.rb:24:in `call'

```

But omitting setting it to `whatever` makes the build fail like this:

```plaintext
                                                                                         
I, [2024-11-07T19:15:31.060936 #1] INFO -- : > cd /var/www/discourse && sudo -E -u discourse bundle exec rake s3:upload_assets                                                   
ERROR: Ensure S3 is configured in config/discourse.conf or environment vars              
I, [2024-11-07T19:15:36.056204 #1] INFO -- :                                            
                                                                                         
                                           

```

I found this:

[https://github.com/hashicorp/terraform-provider-aws/issues/37878](https://github.com/hashicorp/terraform-provider-aws/issues/37878)

and this:

[https://github.com/aws/aws-sdk-php/issues/2614](https://github.com/aws/aws-sdk-php/issues/2614)

Maybe S3\_REGION should be cleared if an endpoint is set here?

> <https://github.com/discourse/discourse/blob/main/lib/file_store/to_s3_migration.rb#L48>

EDIT: Oops. I thought I’d searched. Sorry Falco.

---

<div class="post-metadata">

### Author: ![pfaffman](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/pfaffman/32/120154_2.png) [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [November 7, 2024, 7:40pm UTC](https://meta.discourse.org/t/rebuild-issue-cannot-set-dual-stack-in-combination-with-a-custom-endpoint/334902/8 "2024-11-07T19:40:04Z")

</div>

> [@Don](#):
>
> For _quick fix_ I’ve rebuild with [cc01555fce59e116b76c912b4c5195e111a652b2](https://github.com/discourse/discourse/commit/cc01555fce59e116b76c912b4c5195e111a652b2) version,

Strangely, that gives me this error:

```plaintext
Gem::LoadError: can't activate webrick-1.9.0, already activated webrick-1.8.2 (Gem::LoadError)

```

which I’d just seen on an instance I was using to run an import, which I “fixed” by upgrading.

I guess I just can’t upgrade this server right now. ☹

---

<div class="post-metadata">

### Author: ![Don](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/don/32/228726_2.png) [@Don](https://meta.discourse.org/u/Don)
#### Post date: [November 7, 2024, 7:48pm UTC](https://meta.discourse.org/t/rebuild-issue-cannot-set-dual-stack-in-combination-with-a-custom-endpoint/334902/9 "2024-11-07T19:48:22Z")

</div>

I am not sure but maybe worth a try. I did a rebuild succesfully before do this.

> [@Rebuild issue: \[Cannot set dual-stack in combination with a custom endpoint.\]](https://meta.discourse.org/t/rebuild-issue-cannot-set-dual-stack-in-combination-with-a-custom-endpoint/334902/2):
>
> I commented out these lines which makes the rebuild run successfully but the site stuck on splash I think because cannot load assets. thinking [Safe mode](https://meta.discourse.org/t/53504?silent=true) same… after\_assets\_precompile: - exec: cd: $home cmd: - sudo -E -u discourse bundle exec rake s3:upload\_assets - sudo -E -u discourse bundle exec rake s3:expire\_missing\_assets Any idea? I know it’s a bad timing… confused

And just after this try to rebuild with pick the commit. Second time run the rebuild successfully.

---

<div class="post-metadata">

### Author: ![Falco](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/falco/32/179432_2.png) [@Falco](https://meta.discourse.org/u/Falco)
#### Post date: [November 7, 2024, 8:14pm UTC](https://meta.discourse.org/t/rebuild-issue-cannot-set-dual-stack-in-combination-with-a-custom-endpoint/334902/10 "2024-11-07T20:14:52Z")

</div>

@martin usually what we do here is providing a GlobalSetting to disable the behavior that people running incompatible S3 clones can then opt-in to make it work.

IMO, the current behavior should stay as the default, but provide toggles to increase compatibility.

See prior art in `DISCOURSE_S3_HTTP_CONTINUE_TIMEOUT`, `DISCOURSE_S3_INSTALL_CORS_RULE`, etc.

---

<div class="post-metadata">

### Author: ![martin](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/martin/32/491371_2.png) [@martin](https://meta.discourse.org/u/martin)
#### Post date: [November 8, 2024, 12:13am UTC](https://meta.discourse.org/t/rebuild-issue-cannot-set-dual-stack-in-combination-with-a-custom-endpoint/334902/11 "2024-11-08T00:13:25Z")

</div>

Ah sorry my bad here everyone, these S3 clones always slip my mind. I actually had to fix this custom endpoint issue with Minio yesterday. I think the fix here can just to be never use `dualstack` if `DISCOURSE_S3_ENDPOINT` has been set since they are incompatible, and everyone here is seemingly using that.

---

<div class="post-metadata">

### Author: ![martin](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/martin/32/491371_2.png) [@martin](https://meta.discourse.org/u/martin)
#### Post date: [November 8, 2024, 1:13am UTC](https://meta.discourse.org/t/rebuild-issue-cannot-set-dual-stack-in-combination-with-a-custom-endpoint/334902/12 "2024-11-08T01:13:04Z")

</div>

I merged a fix for this now:

[https://github.com/discourse/discourse/pull/29654](https://github.com/discourse/discourse/pull/29654)

---

<div class="post-metadata">

### Author: ![pfaffman](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/pfaffman/32/120154_2.png) [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [November 8, 2024, 4:03pm UTC](https://meta.discourse.org/t/rebuild-issue-cannot-set-dual-stack-in-combination-with-a-custom-endpoint/334902/14 "2024-11-08T16:03:31Z")

</div>

> [@martin](#):
>
> I merged a fix for this

Thanks for the quick fix! I’m back in business. 🎉

> [@Falco](#):
>
> See prior art

Thanks for the lucid explanation, Rafael!

> [@martin](#):
>
> these S3 clones always slip my mind.

That’s because there’s a lot of stuff to keep in mind! And the spec tests, I think, would require those annoying `stub_request`s which I know I hate writing.

> [@martin](#):
>
> I think the fix here can just to be never use `dualstack` if `DISCOURSE_S3_ENDPOINT` has been set since they are incompatible

FWIW, that sounds right to me. And you could probably write tests for that that didnt’ require stubs for `stub_request`s.

---

<div class="post-metadata">

### Author: ![nat](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/nat/32/235063_2.png) [@nat](https://meta.discourse.org/u/nat)
#### Post date: [November 13, 2024, 5:49am UTC](https://meta.discourse.org/t/rebuild-issue-cannot-set-dual-stack-in-combination-with-a-custom-endpoint/334902/16 "2024-11-13T05:49:00Z")

</div>

10 posts were split to a new topic: [Unable to backup or navigate to backups](https://meta.discourse.org/t/unable-to-backup-or-navigate-to-backups/335899)

---

<div class="post-metadata">

### Author: ![merefield](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/merefield/32/176214_2.png) [@merefield](https://meta.discourse.org/u/merefield)
#### Post date: [November 13, 2024, 11:29am UTC](https://meta.discourse.org/t/rebuild-issue-cannot-set-dual-stack-in-combination-with-a-custom-endpoint/334902/17 "2024-11-13T11:29:06Z")

</div>

I don’t think this should be marked as #fixed .

This change has very likely caused [a break to backups on several instances](https://meta.discourse.org/t/unable-to-backup-or-navigate-to-backups/335899) (with others not knowing that their instances aren’t backing up yet presumably).

That suggests to me the specs are not broad enough to capture all the real use cases and changes are passing that shouldn’t have?

---

<div class="post-metadata">

### Author: ![pfaffman](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/pfaffman/32/120154_2.png) [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [November 14, 2024, 5:01pm UTC](https://meta.discourse.org/t/rebuild-issue-cannot-set-dual-stack-in-combination-with-a-custom-endpoint/334902/18 "2024-11-14T17:01:08Z")

</div>

It sounds to me like it is fixed just for sites that have `use_s3` turned on but not those that are using S3 only for backups.

Though it is again possible to rebuild, it’s just that S3 backups are broken. 😿

---

<div class="post-metadata">

### Author: ![martin](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/martin/32/491371_2.png) [@martin](https://meta.discourse.org/u/martin)
#### Post date: [November 15, 2024, 4:41am UTC](https://meta.discourse.org/t/rebuild-issue-cannot-set-dual-stack-in-combination-with-a-custom-endpoint/334902/19 "2024-11-15T04:41:58Z")

</div>

The backup problem is fixed here [Unable to backup or navigate to backups - #20 by martin](https://meta.discourse.org/t/unable-to-backup-or-navigate-to-backups/335899/20)

---

<div class="post-metadata">

### Author: ![j.jaffeux](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/j.jaffeux/32/60297_2.png) [@j.jaffeux](https://meta.discourse.org/u/j.jaffeux)
#### Post date: [November 18, 2024, 4:42am UTC](https://meta.discourse.org/t/rebuild-issue-cannot-set-dual-stack-in-combination-with-a-custom-endpoint/334902/20 "2024-11-18T04:42:46Z")

</div>

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.