Can't rebuild due to AWS SDK gem bump and new AWS Data Integrity Protections

Installation
28

Hi - I’m Pat Patterson, Chief Technical Evangelist at Backblaze; I arrived on this thread because I have a self-hosted proof-of-concept Discourse forum, and I happened to bump into this exact issue today while configuring my forum to use Backblaze B2 for backups and uploads.

Setting AWS_REQUEST_CHECKSUM_CALCULATION & AWS_RESPONSE_CHECKSUM_CALCULATION to WHEN_REQUIRED is a helpful workaround for basic cases of uploading and downloading files, but it’s helpful to know that it doesn’t cover a number of scenarios, including:

  • Deleting files - Discourse is using the DeleteObjects S3 operation to delete multiple files in a single API call, as it should.
  • Uploading files to buckets with object lock enabled.

The problem is that a checksum (either the Content-MD5 header or one of the new checksum headers) is required (rather than just supported) for these operations, and this causes the current AWS SDKs to provide the new checksum header. As far as I know, there is no way to override this and have the SDK provide Content-MD5 as it used to.

Our engineers are working on resolving all this; in the meantime, the best mitigation is to use version 1.177.0 or earlier of the aws-sdk-s3 gem.

I did try to downgrade the AWS SDK gem versions in my PoC deployment by editing the Gemfile and replacing

gem "aws-sdk-s3", require: false
gem "aws-sdk-sns", require: false

with

gem "aws-sdk-core", "~> 3.215.1", require: false
gem "aws-sdk-kms", "~> 1.96.0", require: false
gem "aws-sdk-s3", "~> 1.177.0", require: false
gem "aws-sdk-sns", "~> 1.92.0", require: false

but my bundle-fu is not strong, and I only succeeded in breaking my deployment with the error:

/var/www/discourse/config/initializers/100-sidekiq.rb:69:in `<main>': undefined method `logger=' for module Sidekiq (NoMethodError)

  Sidekiq.logger = Logger.new(nil)
         ^^^^^^^^^
	from /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/railties-7.2.2.1/lib/rails/engine.rb:689:in `load'
	from /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/railties-7.2.2.1/lib/rails/engine.rb:689:in `block in load_config_initializer'
...

I guess I missed some vital step.

Without wishing to cast shade at our friends at DO, they did this by updating their service to simply ignore the new checksum headers rather than rejecting the API call due to the unsupported checksum.

Their incident report says:

Note that Spaces does not currently verify data integrity checksums sent by the AWS CLI and AWS SDKs as part of upload requests

We decided that simply accepting and storing data that may not match the checksum that the API client supplied was a bad thing.

5 Likes