Digital Ocean Spaces (S3) "unable to sign request without credentials set"

[DavidO (David)]

Trying to setup Digital Ocean Spaces for S3 storage.

I get this when I try to run backups…

Same when I upload photos…

I have this setup…

…but I’m not sure how to test if those actually work. I don’t know these systems very well. This is just for a small friends group forum.

Initially I had the below live, but I had to comment it back out on the app.yml because it wouldn’t rebuild with those lines live. Not sure if it’s still necessary or not at this point…

@Discourse

after_assets_precompile:
    - exec:
        cd: $home
        cmd:
          - sudo -E -u discourse bundle exec rake s3:upload_assets
          - sudo -E -u discourse bundle exec rake s3:expire_missing_assets

I’ve manually uploaded a file from the Digital Ocean Spaces admin area, set it to public, and I’m able to pull it up through a browser.

I also manually created the “*****-backups” folder in DO Spaces.

Here’s an example of how little I know. I’m stuck here and I don’t really know why. Somehow I’ve backed myself out of something and I’m not sure how to get back.

For some reason what’s usually in /var/discourse now seems to be in /var/www/discourse

I’m just stuck at this point though. I’ve searched through the forum and not found anything that will solve it. I’ve spent a day in the same loop. This is along the same lines, but it’s actual AWS S3 and not DO Spaces (S3)…

AWS S3 upload - unable to sign request without credentials set

Tried now, to set up IAM User with credentials and give it s3:* permissions, and now the upload works, but discourse can´t display it afterwards: [The image failed to load, resulting in a black screen with the text "The image could not be loaded." displayed. (Captioned by AI)] As I´m working in a company, and there are SCP (Service control policies) active for our accounts, it seems that public access is generally blocked [The image shows a settings page where public access is blocked for a…

Any assistance would be appreciated. Thank you.

[pfaffman (Jay Pfaffman)]

You’re missing the use_s3 setting that’s recommended on the topic you linked.

You should copy paste those settings from the topic you linked rather than trying to type them. You need colon not equal.

If you’re seeing stuff at /var/www/discourse you’re inside the container. You need to exit to get back to the os.

[DavidO (David)]

Thanks for the feedback.

@pfaffman

You’re missing the use_s3 setting that’s recommended on the topic you linked.

This is in there. I guess it just doesn’t show with that other command. I had the CDN commented out because it said there were issues with it, so I was trying to isolate that out. Especially because I’m not 100% sure that the URL is the correct one since the example was a custom domain instead of the default Spaces one. Maybe that was a bad idea to try to isolate it out. I was trying a lot of things.
Here’s the full setting…

@pfaffman

You should copy paste those settings from the topic you linked rather than trying to type them. You need colon not equal.

Not 100% positive, but I’m pretty sure I did copy/paste. I usually do.

I did have an error when trying to rebuild, so I went back in and commented these back out…

I’m going to try to enable those again and run it.

@pfaffman

If you’re seeing stuff at /var/www/discourse you’re inside the container. You need to exit to get back to the os.

Ah, ‘exit’

That’s what I was looking for. Thank you.

What am I missing on this S3 part?

[DavidO (David)]

I removed the commenting to make it live…

EDIT: I just realized that this has a typo in that last line. Not sure how I messed that up.

@Discourse

  after_assets_precompile:
    - exec:
        cd: $home
        cmd:
          - sudo -E -u discourse bundle exec rake s3:upload_assets
          - sudo -E -u discourse bundle exec rake s3:expire_missing_assets

I saved and tried to rebuild and I get this…

That security warning was a new one. I ran to hopefully resolve that…

chmod o-rwx containers/app.yml

I verified the docker version…

docker --version

Looking in to how to update that. I thought rebuilding it would update that, but maybe not.

[DavidO (David)]

@Discourse

 after_assets_precompile:
    - exec:
        cd: $home
        cmd:
          - sudo -E -u discourse bundle exec rake s3:upload_assets
          - sudo -E -u discourse bundle exec rake s3:expire_missing_assets

That didn’t solve it. Even with the old commented out and that copied/pasted in…

I still get an error on rebuild…

”did not find expected key while parsing a block mapping at line 109 column 3”

I’m not sure what key it’s referring to.

Access and secret key settings are in the settings. I’m not sure how to test to make sure those settings actually work, but they’re in there.

Seems like I’m close, but…

[DavidO (David)]

After rebuild, it was just stuck in thinking…

Left it that way a few hours. No change.

Commented out the CDN and it gave me the “line 109” error again. Commented out the bottom area also and rebuilt. I can get into the forum, but I can’t upload any attachments still.

On rebuild, it seems to hang here for a while.

Not sure it matters since it eventually gets by it without any action.

With the CDN and bottom area commented out, I can get the forum to work, but I still have the original issue…

@DavidO

Same when I upload photos…

Not sure what I’m missing.

[DavidO (David)]

After rebuild, it was just stuck in thinking. At first I thought it might be transferring everything or working in the background, so I let it…

Left it that way a few hours. No change.

I checked Spaces and no files were uploaded, so I guess it was just hung on something.

Commented out the CDN and it gave me the “line 109” error again.

Commented out the bottom area also and rebuilt. I can get into the forum, but I can’t upload any attachments still.

On rebuild, it seems to hang here for a while but it eventually gets by it without any action, so I’m not sure that matters at all..

With the CDN and bottom area commented out, I can get the forum to work, but I still have the original issue…

@DavidO

Same when I upload photos…

Not sure what I’m missing. I gave up left all of the S3 settings commented out for now.

SIDENOTE: Is there a command to update the docker? I think I normally do this through the admin area, but it’s not prompting me to update from 24.0.2. Not sure how big of an issue that is. I think the current is 24.0.7.

[DavidO (David)]

I’m not sure if the admin area settings override the app.yml settings or vice versa, but here are those settings JIC…

I’m not clear if the S3 and CDN URLs are supposed to include the bucket name or not. Also not positive that they’re supposed to be identical links.

Hoping someone notices something simple. If you’ve read this far, thanks again!

[pfaffman (Jay Pfaffman)]

Are you using the social ocean cdn?

@Discourse

Only problem is that their CDN offering is awfully broken, so you need to use a different CDN for the files

@DavidO

Left it that way a few hours

If it hadn’t loaded in 30 seconds, it probably won’t. But you don’t need to wait that long. You can look at the network tab in the browser development tools and see what’s not loading.

[DavidO (David)]

Not sure what social ocean CDN is, but I’m just trying to use the CDN that’s built into Spaces.

[pfaffman (Jay Pfaffman)]

Sorry, that was “Digital Ocean”.

@DavidO

but I’m just trying to use the CDN that’s built into Spaces.

The message I quoted says that you shouldn’t do that because it doesn’t work.

If you want more help from me my email address is in my profile

[DavidO (David)]

OK. This is so odd.

SOMETHING is working now. I thought I had it all commented out (turned off) and uploaded a photo yesterday to make sure it worked again. I checked the path to it this morning and it’s using the… CDN I guess?

https://XXXXXXX-storage.sfo3.digitaloceanspaces.com/optimized/2X/9/9b9fdb3eb27831650752b848363ea2a1a66ce137_2_375x500.jpeg

I checked in Spaces and it has some images in there…

It appears that images are working correctly now. The only thing I can think of is that I unchecked “S3 use IAM profile” in the settings. I’m guessing that’s the key that it was looking for that it stopped looking for. Not sure why I had that checked in the first place. Probably because I really have no clue what I’m doing.

Still having an issue running backups, but maybe that’s a Droplet space issue. Looks like it doesn’t have enough space to stage it locally before uploading. I’ll dig into that separately now that it appears to be at least connected to the Space.

Thanks Jay, I’ll reach out later this week when I can mess with this friends forum again. Off to pay some bills for a bit.

[DavidO (David)]

I had a few minutes while waiting for my SO to get ready, so I thought I’d check space. After thinking more about it, I figured that had to be the issue. I checked with…

df -h

Oh yea, really full. Ran…

docker system prune -a

Ran…

./launcher cleanup

Thanks again Jay.

Backup and limited storage

How big is a backup? You need room for two copies. You can try ./launcher cleanup To free some docker space. If you include more data on your backup it will take more storage, so if you’re shook storage, storing less seems like a no brainer

Sure enough, cleared up over 20GB.

Ran the backup manually and it worked!!!

Now I’m trying to migrate all files from local storage to S3…

rake uploads:migrate_to_s3

Wish me luck!

[DavidO (David)]

Checked a few things this morning while waiting for a meeting to start.

Rake didn’t seem to work, or maybe is still in progress. I can’t tell yet. Space usage went down, but when I checked some old photos manually they still seem to have the old local URL. I’ll have to dig into that more.

Not sure what the difference of rake vs reback is. I assume one is migrating the files and the other is updating the paths in the posts. More to learn.

It’s still using a lot of space in /import (1.4GB) and /uploads (9.2GB). A lot for our small package. I’m trying to offload that to cheaper storage and those are by far the biggest chunks.

Some FYI for anyone that’s setting up the same, this is what it’s using for endpoint URLs…

CDN
https://XXXXXX-storage.sfo3.cdn.digitaloceanspaces.com/original/2X/0/15740f130209eb1a6a4f147076eaa20e70a9772d.zip

ORIGIN
https://XXXXXX-storage.sfo3.digitaloceanspaces.com/original/2X/0/15740f130209eb1a6a4f147076eaa20e70a9772d.zip

The only difference is the .cdn in the URL. I need to update my settings.

[DavidO (David)]

@DavidO

Rake didn’t seem to work, or maybe is still in progress. I can’t tell yet. Space usage went down, but when I checked some old photos manually they still seem to have the old local URL. I’ll have to dig into that more.

Not sure what the difference of rake vs reback is. I assume one is migrating the files and the other is updating the paths in the posts. More to learn.

From Copilot…
__________

To rebake posts after changing the CDN link in Discourse, you need to run the rebake rake task so that all cooked post HTML gets updated with the new asset URLs.

Here’s the process:

---

1. Enter the container

cd /var/discourse

./launcher enter app

---

2. Run the rebake task

rake posts:rebake

This will:

• Re-render all posts using the current settings (including your new s3_cdn_url).

• Update cooked HTML so images and uploads point to the CDN.

---

Optional: Speed it up

If you have a large site, you can run it in batches:

rake posts:rebake[1000]

This rebakes 1,000 posts at a time.

---

3. Verify

Check a few posts with images or uploads to confirm they now use the CDN URL.

__________
Seems to be running. I’ll check back later.

Off to make the donuts.

[DavidO (David)]

@DavidO

Optional: Speed it up

If you have a large site, you can run it in batches:

rake posts:rebake[1000]

This rebakes 1,000 posts at a time.

This runs the rebake, but not just 1,000 at a time. It just runs them all and ignores the [1000] part. Maybe my syntax isn’t quite correct there.

Another reminder for myself as much as anyone else. I was looking for this URL in the menu because I had it backwards ( .cdn.sfo3. instead of vice versa - which obviously broke the SSL). I couldn’t find the setting when searching. Then I found a window that I already had it open. I updated it and tried to save and got this…

I forget that once the settings are added in the app.yml, they disappear from the admin settings menu. Now I remember.

The rebake seemed to have worked, I just had the wrong URL. That’s updating now.

I’m still sitting at under 50 files in my Space storage though, so the file rake isn’t working.

[DavidO (David)]

OK. I thought this was a small breakthrough here. I finally figured out what this issue was…

“did not find expected key while parsing a block mapping at line 106 column3”

When looking through a few threads, I saw this…

@RGJ

Remove the space here

I pulled mine up and what do you know, spaces.

I’m sure I put them in there thinking nothing of it. I removed them. I think it was mainly the one before the “hooks:” that it was referring to.

It rebuilt fine. Thanks @RGJ

However, it still doesn’t like something.

I’m still not seeing my item count rise in Spaces, so it doesn’t appear to be uploading the /uploads files to ‘S3’ yet. This was something that had been perplexing me, so I’m glad that’s resolved. But I’m back to the loading screen.

Also, with that enabled, I just get the loading screen.

Commented it back out…

Rebuilt again. Same.

Commented out the Spaces S3 settings.

Rebuilt again.

Working

• Forum loading
• Images loading
  • New ones using Spaces - Not CDN
  • Old ones using local storage

Not Working

• Not uploading files to Spaces
• Not using CDN link for image loading
• If the DISCOURSE_S3 and after_assets_precompile are not commented out, the forum just hangs on loading in a browser

Out of ideas. Going in circles.

I’ll have to let this sit for a while. At least it will keep space from being an issue anymore since new uploads and backups are going to Spaces now.

[DavidO (David)]

It appears to be solved!

Updated S3 CDN URL to… https://XXXXX.sfo3.cdn.digitaloceanspaces.com

That setting gets the path correct, but I was thrown off a bit because the bucket name needs to be in the S3 CDN URL but NOT the S3 URL, so they don’t seem consistent with that treatment.

Then I ran…

rake uploads:migrate_to_s3 --trace

Before, it was showing in the 200s of items in the Spaces CP, but then I noticed…

4361 items 49.3 GiB…

…then on refresh…

4640 items 49.4 GiB…

…it’s alive!!! I think it worked this time because I was maybe not in the app before, so I had run this to get in (IIRC)…

./launcher enter app

I also ran this…

rake posts:rebake

…but that was before I realized that it was still uploading, so I think I’ll come back after it seems to stop uploading and rebake them again.

I need to let this do its thing for a while. I’ll do that and report back if there is any other cleanup.

[DavidO (David)]

Although it’s slowed in the item count increase, it’s still uploading and growing. It went up another 125 items overnight and a tombstone folder just showed up for the first time.

Documenting for any other newbs working through this.

[DavidO (David)]

This is what finally worked:

These settings had to be in the app.yml config in order for it to finally run the rakes correctly. When they were just in the /admin/site_settings/ area, it wouldn’t run the rakes correctly. I was getting an errror:

ERROR: Ensure S3 is configured in config/discourse.conf or environment vars

The settings are from here… Configure an S3 compatible object storage provider for uploads

Specifically, I used non-custom Spaces settings with two spaces before each line. …

DISCOURSE_USE_S3: true
DISCOURSE_S3_REGION: whatever
DISCOURSE_S3_ENDPOINT: https://nyc3.digitaloceanspaces.com
DISCOURSE_S3_ACCESS_KEY_ID: myaccesskey
DISCOURSE_S3_SECRET_ACCESS_KEY: mysecretkey
DISCOURSE_S3_CDN_URL: https://nyc3.cdn.digitaloceanspaces.com
DISCOURSE_S3_BUCKET: my-files
DISCOURSE_S3_BACKUP_BUCKET: my-files/my-backups
DISCOURSE_BACKUP_LOCATION: s3
DISCOURSE_S3_INSTALL_CORS_RULE: false

NOTE: DISCOURSE_S3_REGION should literally say “whatever” and not something that could be construed as an actual S3 region (ie. us-west-1).

Also these settings, in the hooks section of the ## Plugins go here area…

after_assets_precompile:
- exec:
cd: $home
cmd:
- sudo -E -u discourse bundle exec rake s3:upload_assets
- sudo -E -u discourse bundle exec rake s3:expire_missing_assets

This is where I ran into an issue last time. Once that’s in place and I rebuilt, I was getting this loading screen when I tried to bring up our forum in a browser.

This is only because it’s looking for assets that haven’t yet been updated. This is where it’s time to run the rakes, from inside the app…

./launcher enter app

rake s3:upload_assets

Once that ran, the items number jumped up and an “assets” folder showed up in our bucket…

Then the forum started loading correctly. These are the only S3 related settings that I have showing in my /admin/site-settings area…

I ran this to upload all of the files and it updated the URLs from the old path to the new (non-CDN).

rake uploads:migrate_to_s3 --trace

The items number more than tripled after that finished.

The only thing I noticed was that old post image paths were using the S3 ENDPOINT URL versus newly posted images were using the S3 CDN URL.

https:/my-storage.sfo3.digitaloceanspaces.com/
vs
https:/my-storage.sfo3.cdn.digitaloceanspaces.com/

I didn’t really care because at least it was all using S3.

But I tried running…

rake posts:rebake

It killed it because it ran out of memory. I tried running this instead…

rake posts:rebake_uncooked_posts

It ran out of memory a few times and restarted itself, then ran out of memory again. Ran it again. Eventually it got through cooking them all.

Once it ran all the way through to 100%, all of the links were finally using the S3 CDN URL path, old and new posts. Over 34062 items at 65.9 GB of storage (a backup is <9 GB).

Really happy that it’s all working off of S3 now, including backups. Speaking of, probably a good time to run one!

Hope this helps someone.

Good luck!