Uploading local files to DigitalOcean Spaces (S3) with rake uploads:migrate_to_s3 starts and migrates some files, then suddenly fails for one file with error Aws::S3::Errors::SignatureDoesNotMatch. When this file is manually uploaded using the scipt mentioned in Extend S3 configuration for other S3 API compatible services - #4 by rishabh and the script is rerun, it uploads more files but fails again for some other file. Can someone from the team please have a look at this? I have nearly 50GB of files that i need to move from local storage to S3 in the near future (configuration and upload for new files work without issues). I would be happy even if it’s just a workaround for now: I have looked at the code to_s3_migration.rb and can identify the problematic files, but when I upload them to S3 using Aws::S3::Resource from the script mentioned above, the file metadata is missing and the original name of the file is lost on download. Search on the web returns lots of issues for SignatureDoesNotMatch: one of them mentions that version of the signing algorithm that the gem is using might be wrong, but this is beyond my current skill to check. This is configuration for rake: DISCOURSE_S3_ACCESS_KEY_ID="id" \ DISCOURSE_S3_SECRET_ACCESS_KEY="secret" \ DISCOURSE_S3_REGION="sfo2" \ DISCOURSE_S3_BUCKET="mybucket/uploads" \ DISCOURSE_S3_ENDPOINT="https://sfo2.digitaloceanspaces.com" \ DISCOURSE_S3_CDN_URL="https://mybucket.sfo2.cdn.digitaloceanspaces.com" \ rake uploads:migrate_to_s3 --trace

It turned out that DO Spaces doesn’t like consecutive spaces in the file metadata and fails with the completely unrelated error. Perhaps it is worth checking if AWS S3 has the same constraint. Anyway, the quick fix was to conflate multiple spaces into one: lib/file_store/to_s3_migration.rb e474cda32 bucket: bucket, content_type: MiniMime.lookup_by_filename(name)&.content_type, key: key, } if !FileHelper.is_supported_image?(name) upload = Upload.find_by(url: "/#{file}") if upload&.original_filename options[:content_disposition] = %Q{attachment; filename="#{upload.original_filename}"} end if upload&.secure options[:acl] = "private" end end etag ||= Digest::MD5.file(path).hexdigest if @dry_run %Q{attachment; filename="#{upload.original_filename.gsub(/\s+/, ' ')}"}

`rake uploads:migrate_to_s3` fails with `Aws::S3::Errors::SignatureDoesNotMatch`

Product Support Self-hosting

eextra (Sylwia) March 13, 2020, 8:26am 3

is this an official amendment ? what about other errors in migration to s3 ?

Topic		Replies	Views
Cannot upload or backup to Amazon S3 Self-hosting backups , s3	3	2233	October 14, 2015
S3 uploads migrations to DigitalOcean S3 Spaces fails Product Support	0	622	April 15, 2020
`uploads:migrate_to_s3` fails with 'Seahorse::Client::NetworkingError: certificate verify failed' on Digital Ocean Spaces Self-hosting s3	1	2325	February 11, 2020
Migrating old uploads to S3 Self-hosting s3	8	1776	May 11, 2021
Rake migrate_to_s3 errors Self-hosting s3	3	846	February 4, 2020

`rake uploads:migrate_to_s3` fails with `Aws::S3::Errors::SignatureDoesNotMatch`

Related topics