`rake uploads:migrate_to_s3` fails with `Aws::S3::Errors::SignatureDoesNotMatch`

Installation
1

Uploading local files to DigitalOcean Spaces (S3) with rake uploads:migrate_to_s3 starts and migrates some files, then suddenly fails for one file with error Aws::S3::Errors::SignatureDoesNotMatch.

When this file is manually uploaded using the scipt mentioned in Extend S3 configuration for other S3 API compatible services - #4 by rishabh and the script is rerun, it uploads more files but fails again for some other file.

Can someone from the team please have a look at this? I have nearly 50GB of files that i need to move from local storage to S3 in the near future (configuration and upload for new files work without issues).

I would be happy even if it’s just a workaround for now: I have looked at the code to_s3_migration.rb and can identify the problematic files, but when I upload them to S3 using Aws::S3::Resource from the script mentioned above, the file metadata is missing and the original name of the file is lost on download.

Search on the web returns lots of issues for SignatureDoesNotMatch: one of them mentions that version of the signing algorithm that the gem is using might be wrong, but this is beyond my current skill to check.

This is configuration for rake:

DISCOURSE_S3_ACCESS_KEY_ID="id" \
DISCOURSE_S3_SECRET_ACCESS_KEY="secret" \
DISCOURSE_S3_REGION="sfo2" \
DISCOURSE_S3_BUCKET="mybucket/uploads" \
DISCOURSE_S3_ENDPOINT="https://sfo2.digitaloceanspaces.com" \
DISCOURSE_S3_CDN_URL="https://mybucket.sfo2.cdn.digitaloceanspaces.com" \
rake uploads:migrate_to_s3 --trace
1 Like
2

It turned out that DO Spaces doesn’t like consecutive spaces in the file metadata and fails with the completely unrelated error. Perhaps it is worth checking if AWS S3 has the same constraint.

Anyway, the quick fix was to conflate multiple spaces into one:

https://github.com/discourse/discourse/blob/e474cda3219d1bfd65ee7386cddece99a9e64374/lib/file_store/to_s3_migration.rb#L215

              %Q{attachment; filename="#{upload.original_filename.gsub(/\s+/, ' ')}"}
2 Likes
3

is this an official amendment ? what about other errors in migration to s3 ?

4

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.