Can't rebuild due to AWS SDK gem bump and new AWS Data Integrity Protections

[pfaffman (Jay Pfaffman)]

@AntiMetaman

I rebuilt the app and it worked!

If you rebuilt the app then the changes made in the container are gone. Perhaps the backblaze folks fixed things while you were doing that.

[AntiMetaman]

I just commented out the S3 and rebuilt. I guess I’ll stick to local storage from now on.

[pfaffman (Jay Pfaffman)]

Local storage works fine. It’s just deleting that does not work.

[tedpearson]

@PatPatterson

This is because, in the S3 API definition, the DeleteObjects operation has httpChecksum.requestChecksumRequired set to true, in contrast to, say, PutObject, which sets it to false. So the SDK is honoring WHEN_REQUIRED, because the checksum is required.

All of the AWS SDKs are now generated from the API definition, with minimal extra code to cover edge situations. The code sees that DeleteObjects requires a checksum, and the way to do that now is to use one of the new checksums, rather than Content-MD5.

Unfortunately, AWS didn’t see fit to provide a ‘use Content-MD5 like you used to’ configuration. They don’t tend to consider the third-party object storage ecosystem when they make these sorts of changes, since they don’t really need to. The only time that stuff like this gets fixed is when it accidentally hoses one of their own services in some corner case.

Thanks for posting and confirming there is no current workaround without downgrading the aws SDKs. I ran into issues with the b2 s3 apis from another project using the golang aws sdk, trying to back up a VictoriaMetrics database.

Do you have any sense from your engineering team how or when they may choose to resolve this issue, or have any way to track/get updates on a supported workaround? I’m trying to decide if I should fork and recompile a project with a dependency change or just wait a little while for an update!

Thanks!

[bjmc (Brendan)]

The following procedure worked for me to downgrade the AWS Gems:

 # To get into the container:
./launcher enter app
# Apparently needed to unfreeze Gemfile.lock:
bundle config set frozen false
# Set an older version of the sdk-s3 gem:
sed -i 's/gem "aws-sdk-s3", require: false/gem "aws-sdk-s3", "1.177.0", require: false/' Gemfile
# Downgrade the S3 gem to match what's now in the Gemfile:
bundle update aws-sdk-s3
# Also downgrade aws-sdk-core gem:
bundle add aws-sdk-core --version 3.215.

After making these changes, I was able to successfully save a backup to B2. I’m sure this is only a temporarily workaround that will be lost on the next Discourse update, so I’m hoping @PatPatterson and the Backblaze team can provide a more permanent compatibility fix soon.

[pfaffman (Jay Pfaffman)]

Hopefully you won’t need to do this, but it’s possible to put that sed command in the app.yml so that it would happen automatically when you rebuild. I think just sticking it below where the plugins get cloned would do it. But it could be more complicated than I think.

[becl]

Hi AntiMetaman,

I have spent the last week trying to get backups working with s3 storage and finally got the backup to work with this thread. Honestly, I am not sure which part worked.

I followed what you did with the gem uninstall and install but now I get an error trying to upload images. My error log shows this

not entitled /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/seahorse/client/plugins/raise_response_errors.rb:17:in `call’ /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aw

Before I have to trash my whole server and start again (honestly, at this point I’m not even sure I want to continue) do you know how I can reverse the gem uninstall and install so I can try and see if that fixes the issue?