### Author:  [@watchmanmonitor](https://meta.discourse.org/u/watchmanmonitor)
#### Post date: [28 mei 2017 om 04:51 UTC](https://meta.discourse.org/t/launcher-rebuild-fails-if-s3-settings-arent-correct/63490/1 "2017-05-28T04:51:38Z")
One of my sites won’t come up after a recent update.
It’s a digital ocean droplet setup via the standard docker, and has been up & running for about 1.5 years.
Today it failed to restart, here’s the full rebuild attempt after an `apt-get upgrade -y` & reboot
There are no plugins installed.
It was set up to backup to s3 (my other sites aren’t) and this is of-interest.
```plaintext
16287:C 28 May 02:33:33.481 * DB saved on disk
16287:C 28 May 02:33:33.482 * RDB: 18 MB of memory used by copy-on-write
155:M 28 May 02:33:33.532 * Background saving terminated with success
rake aborted!
Aws::S3::Errors::PermanentRedirect: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/seed-fu-2.3.5/lib/seed-fu/runner.rb:46:in `eval'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-2.5.3/lib/aws-sdk-core/plugins/s3_sse_cpk.rb:19:in `call'
```
here’s the full rebuild attempt
>
My other three sites are fine, not sure what’s going on with this one, any pointers would be appreciated.
---
### Author:  [@tophee](https://meta.discourse.org/u/tophee)
#### Post date: [28 mei 2017 om 06:01 UTC](https://meta.discourse.org/t/launcher-rebuild-fails-if-s3-settings-arent-correct/63490/2 "2017-05-28T06:01:57Z")
Just a wild guess: have you seen this topic below?
> [@Rebuilds fail for anyone with discourse-narrative-bot in app.yml](https://meta.discourse.org/t/rebuilds-fail-for-anyone-with-discourse-narrative-bot-in-app-yml/63410):
>
> Since discourse-narrative-bot is now a core plugin, and contained within the core repository, there is now no need for the git clone https://github.com/discourse/discourse-narrative-bot.git line in app.yml. Anyone who tries to rebuild with that line in their config will get an error message exec failed with the params {"cd"=\>"$home/plugins", "cmd"=\>["git clone https://github.com/discourse/docker\_manager.git", "git clone https://github.com/discourse/discourse-narrative-bot.git"]} This will a…
---
### Author:  [@watchmanmonitor](https://meta.discourse.org/u/watchmanmonitor)
#### Post date: [28 mei 2017 om 07:01 UTC](https://meta.discourse.org/t/launcher-rebuild-fails-if-s3-settings-arent-correct/63490/3 "2017-05-28T07:01:50Z")
Thanks. I’d checked for this. However, there were no plugins installed.
---
### Author:  [@mpalmer](https://meta.discourse.org/u/mpalmer)
#### Post date: [28 mei 2017 om 08:11 UTC](https://meta.discourse.org/t/launcher-rebuild-fails-if-s3-settings-arent-correct/63490/4 "2017-05-28T08:11:24Z")
> [@watchmanmonitor](#):
>
> Aws::S3::Errors::PermanentRedirect: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.
Whilst this error is far from perfectly helpful, I’m guessing that AWS has deprecated (and now removed) whatever style of endpoint URL you’ve configured, and now everything is awful. Check your S3-related site settings and compare them against the S3 documentation for correctness.
---
### Author:  [@watchmanmonitor](https://meta.discourse.org/u/watchmanmonitor)
#### Post date: [28 mei 2017 om 12:59 UTC](https://meta.discourse.org/t/launcher-rebuild-fails-if-s3-settings-arent-correct/63490/5 "2017-05-28T12:59:34Z")
The value is set in the ui, and the site is down, so I can’t get to the value using normal means.
I suppose I can decompress a recent backup and edit the value in the pg dump. I’d have to know which key to empty out.
What are the odds that a change could be committed which lets `launcher` continue past this error?
---
### Author:  [@watchmanmonitor](https://meta.discourse.org/u/watchmanmonitor)
#### Post date: [28 mei 2017 om 15:00 UTC](https://meta.discourse.org/t/launcher-rebuild-fails-if-s3-settings-arent-correct/63490/6 "2017-05-28T15:00:36Z")
The settings in question are
```plaintext
74 backup_frequency 3 1 2015-09-21 12:28:56.795021 2015-09-21 12:28:56.795021
75 s3_backup_bucket 1 backups-discuss 2016-05-18 14:53:39.422497 2016-05-18 14:53:39.422497
76 s3_disable_cleanup 5 t 2016-05-18 14:53:40.325988 2016-05-18 14:53:40.325988
77 s3_upload_bucket 1 backups-discuss 2016-05-18 15:10:17.459986 2016-07-25 21:50:42.532383
78 s3_secret_access_key 1 REDACTEDAi8w/JfYNGNt9KrYA5MEuoREDACTED 2016-05-18 15:10:18.783565 2016-05-18 15:10:18.783565
79 s3_access_key_id 1 REDACTEDGNC6OS3REDACTED 2016-05-18 15:10:19.543714 2016-05-18 15:10:19.543714
80 enable_s3_uploads 5 t 2016-05-18 15:10:27.6396 2016-05-18 15:10:27.6396
```
Any idea what’s wrong here?
---
### Author:  [@watchmanmonitor](https://meta.discourse.org/u/watchmanmonitor)
#### Post date: [28 mei 2017 om 18:50 UTC](https://meta.discourse.org/t/launcher-rebuild-fails-if-s3-settings-arent-correct/63490/7 "2017-05-28T18:50:12Z")
We’ve restored the DO host to the day before I updated.
Here’s what the settings look like.. what’s been entered incorrectly?
---
### Author:  [@watchmanmonitor](https://meta.discourse.org/u/watchmanmonitor)
#### Post date: [30 mei 2017 om 18:08 UTC](https://meta.discourse.org/t/launcher-rebuild-fails-if-s3-settings-arent-correct/63490/9 "2017-05-30T18:08:38Z")
> [@mpalmer](#):
>
> Check your S3-related site settings and compare them against the S3 documentation for correctness.
I restored the entire image so our site is up, but I have no reason to think I won’t encounter a failure on the next `one-click` or `./launcher rebuild app` failure.
I’m looking here:
> [@Configure automatic backups for Discourse](https://meta.discourse.org/t/configure-automatic-backups-for-discourse/14855):
>
> bookmark This guide explains how to configure automatic backups for Discourse, including storage options on local servers and S3-compatible storage. Learn how to set up automatic backups for your Discourse platform. This guide covers configuring automatic backups, storing them on local servers or S3-compatible storage, and managing storage retention options like Amazon Glacier. Configuring automatic backups Navigate to /admin settings. Select the Backup section. Set backup\_frequency to …
I don’t see a smoking gun as to the problem. Any pointers would be appreciated.
---
### Author:  [@watchmanmonitor](https://meta.discourse.org/u/watchmanmonitor)
#### Post date: [2 juni 2017 om 14:44 UTC](https://meta.discourse.org/t/launcher-rebuild-fails-if-s3-settings-arent-correct/63490/10 "2017-06-02T14:44:13Z")
before trying an update again, I’ve been waiting for any feedback on what’s set wrong here
I can’t help but wonder if this error:
`Aws::Errors::MissingCredentialsError: unable to sign request without credentials set`
[https://meta.discourse.org/t/upgrade-error-6-1-2017/63756/8](https://meta.discourse.org/t/upgrade-error-6-1-2017/63756/8)
is related
---
### Author:  [@Timothy\_Vail](https://meta.discourse.org/u/Timothy_Vail)
#### Post date: [2 juni 2017 om 15:03 UTC](https://meta.discourse.org/t/launcher-rebuild-fails-if-s3-settings-arent-correct/63490/11 "2017-06-02T15:03:47Z")
Yes, I wonder what that is and how to fix it.
---
### Author:  [@watchmanmonitor](https://meta.discourse.org/u/watchmanmonitor)
#### Post date: [2 juni 2017 om 17:44 UTC](https://meta.discourse.org/t/launcher-rebuild-fails-if-s3-settings-arent-correct/63490/12 "2017-06-02T17:44:01Z")
> [@mpalmer](#):
>
> Check your S3-related site settings and compare them against the S3 documentation for correctness.
Why is a rebuild process trying to talk to our s3 backup endpoint at all?
Why is a failure to do so resulting in a failure to rebuild?
---
### Author:  [@watchmanmonitor](https://meta.discourse.org/u/watchmanmonitor)
#### Post date: [3 juni 2017 om 17:18 UTC](https://meta.discourse.org/t/launcher-rebuild-fails-if-s3-settings-arent-correct/63490/14 "2017-06-03T17:18:15Z")
ok, I’ve spent enough time troublehshooting this to be certain:
If the s3 backup settings are incorrect, and `enable_s3_backups` is true, then a normal rebuild will fail.
I manage the site for a friend who entered his s3 keys incorrectly from day 1 and I’ve never had a problem rebuilding his container before.
An update to discourse/discourse\_docker which skips trying to talk to s3, or at least gracefully accepts failure, would mean that people aren’t blindsided by an s3 backup failure during a rebuild
---
### Author:  [@mpalmer](https://meta.discourse.org/u/mpalmer)
#### Post date: [4 juni 2017 om 01:36 UTC](https://meta.discourse.org/t/launcher-rebuild-fails-if-s3-settings-arent-correct/63490/15 "2017-06-04T01:36:17Z")
The backtrace indicates that the rebuild attempted to upload an avatar to S3, because it was told that was where avatars are stored for this site. Chances are everyone’s avatar uploads have been exploding but nobody noticed, so in a way it’s nice you’ve been told it’s broken now, so it can be fixed… :troll:
---
### Author:  [@watchmanmonitor](https://meta.discourse.org/u/watchmanmonitor)
#### Post date: [4 juni 2017 om 02:59 UTC](https://meta.discourse.org/t/launcher-rebuild-fails-if-s3-settings-arent-correct/63490/16 "2017-06-04T02:59:07Z")
I can’t wait to hear how.
---
### Author:  [@watchmanmonitor](https://meta.discourse.org/u/watchmanmonitor)
#### Post date: [4 juni 2017 om 19:57 UTC](https://meta.discourse.org/t/launcher-rebuild-fails-if-s3-settings-arent-correct/63490/17 "2017-06-04T19:57:39Z")
I see, `enable s3 uploads` was, in fact, enabled, and the s3 setup was not correct.
The s3 config on this site was done a long time ago, and I haven’t run in to this problem before.
I wonder what changed about the upload process lately, but more to the point:
Is the some agreement that if an upload to s3 fails during upgrade or bootstrap, the rebuild should be able to continue?
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [4 juni 2017 om 19:59 UTC](https://meta.discourse.org/t/launcher-rebuild-fails-if-s3-settings-arent-correct/63490/18 "2017-06-04T19:59:36Z")
Maybe, but this might be a really rare problem for anyone to have?
---
### Author:  [@watchmanmonitor](https://meta.discourse.org/u/watchmanmonitor)
#### Post date: [4 juni 2017 om 20:02 UTC](https://meta.discourse.org/t/launcher-rebuild-fails-if-s3-settings-arent-correct/63490/19 "2017-06-04T20:02:06Z")
It’ll be rarer still once discourse warns of [s3 failures](https://meta.discourse.org/t/notify-if-s3-backups-are-failing/63884).
The problem is that once failed, I was left without a way to get the system up & running again.. I had to roll back to a system level snapshot from a few days ago, so I could get around the inability to rebuild.
---
### Author:  [@JagWaugh](https://meta.discourse.org/u/JagWaugh)
#### Post date: [4 juni 2017 om 20:48 UTC](https://meta.discourse.org/t/launcher-rebuild-fails-if-s3-settings-arent-correct/63490/20 "2017-06-04T20:48:22Z")
> [@watchmanmonitor](#):
>
> The problem is that once failed, I was left without a way to get the system up & running again
I think _that_ is a major problem, from a design standpoint. In my books, a rebuild should only fail if something is screwed up on my box (i.e. something broken, or outdated), or I can’t reach git to get the newest version. If my box isn’t broken, then rebuild shouldn’t break it. If rebuild notices that I’ve messed up my S3 config then it should say “I can proceed if you want, but your uploads and backups are/will be missing. Do you want to abort or proceed, Meatbag?”
---
### Author:  [@watchmanmonitor](https://meta.discourse.org/u/watchmanmonitor)
#### Post date: [5 juni 2017 om 02:47 UTC](https://meta.discourse.org/t/launcher-rebuild-fails-if-s3-settings-arent-correct/63490/21 "2017-06-05T02:47:25Z")
> [@codinghorror](#):
>
> really rare problem
I think it’s really rare bc it’s brand new in the past month.
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [6 november 2017 om 08:29 UTC](https://meta.discourse.org/t/launcher-rebuild-fails-if-s3-settings-arent-correct/63490/22 "2017-11-06T08:29:42Z")