The AWS SDK maintainers broke the compatibility. It is up to your S3 clone provider to get up to speed and implement better compatibility so you can remove the workarounds.
Just to clarify, this only affects JS/map/CSS files in assets, and wonât affect uploads, right?
I mean, will it impact clearing orphaned files?
By the way, I assume this issue might affect updating Discourse from the admin panel?
Actually, updating via the admin panel failed for me, which is why I performed a rebuild.
Yes, it is only for assets.
This rake task? No.
But the whole AWS SDK change may have also broken that for people on non-compatible clones.
I"m pretty sure thatâs wrong. So we probably need to turn off clean up uploads
too. Thatâll just cause errors when youâre running, though. It wonât cause you to not be able to rebuild.
Sounds likely. Maybe we need some âskip_s3_deleteâ setting to solve this until the other providers catch up? And maybe automatically set it for providers we know are broken?
Is that rake task the only way that expired assets get removed?
Iâm just wondering, why not add an option to keep the assets on the Discourse core server (I mean, without storing them on S3)?
As long as it doesnât affect uploads or the process of clearing orphaned files, it seems like a viable solution.
Yes. It is not like this is a big deal. Normal sites updating every once in a while wonât see much difference.
People can setup their own lifecycle rules if they care about this stuff.
Isnât setting up clean up uploads = false
already that?
Lol no. Discourse officially supports S3. While I went out of my way starting the whole Configure an S3 compatible object storage provider for uploads wiki and adding a few toggles to increase clone compatibility we have zero plans on investing more time into that today.
If the community wants to send a couple PRs that increase compatibility and are default off that is pr-welcome, but donât expect to see official support in core for every clone any time soon.
FWIW, it looks like Digital Ocean has no trouble deleting backups nor expiring missing assets.
For providers that are broken, itâd take a long time before unneeded assets caused much of a problem. Keeping a whole bunch of backups including a huge database and all uploads, could be quite a problem if youâre paying for storage.
Hi - Iâm Pat Patterson, Chief Technical Evangelist at Backblaze; I arrived on this thread because I have a self-hosted proof-of-concept Discourse forum, and I happened to bump into this exact issue today while configuring my forum to use Backblaze B2 for backups and uploads.
Setting AWS_REQUEST_CHECKSUM_CALCULATION
& AWS_RESPONSE_CHECKSUM_CALCULATION
to WHEN_REQUIRED
is a helpful workaround for basic cases of uploading and downloading files, but itâs helpful to know that it doesnât cover a number of scenarios, including:
- Deleting files - Discourse is using the
DeleteObjects
S3 operation to delete multiple files in a single API call, as it should. - Uploading files to buckets with object lock enabled.
The problem is that a checksum (either the Content-MD5
header or one of the new checksum headers) is required (rather than just supported) for these operations, and this causes the current AWS SDKs to provide the new checksum header. As far as I know, there is no way to override this and have the SDK provide Content-MD5
as it used to.
Our engineers are working on resolving all this; in the meantime, the best mitigation is to use version 1.177.0
or earlier of the aws-sdk-s3
gem.
I did try to downgrade the AWS SDK gem versions in my PoC deployment by editing the Gemfile and replacing
gem "aws-sdk-s3", require: false
gem "aws-sdk-sns", require: false
with
gem "aws-sdk-core", "~> 3.215.1", require: false
gem "aws-sdk-kms", "~> 1.96.0", require: false
gem "aws-sdk-s3", "~> 1.177.0", require: false
gem "aws-sdk-sns", "~> 1.92.0", require: false
but my bundle
-fu is not strong, and I only succeeded in breaking my deployment with the error:
/var/www/discourse/config/initializers/100-sidekiq.rb:69:in `<main>': undefined method `logger=' for module Sidekiq (NoMethodError)
Sidekiq.logger = Logger.new(nil)
^^^^^^^^^
from /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/railties-7.2.2.1/lib/rails/engine.rb:689:in `load'
from /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/railties-7.2.2.1/lib/rails/engine.rb:689:in `block in load_config_initializer'
...
I guess I missed some vital step.
Without wishing to cast shade at our friends at DO, they did this by updating their service to simply ignore the new checksum headers rather than rejecting the API call due to the unsupported checksum.
Their incident report says:
Note that Spaces does not currently verify data integrity checksums sent by the AWS CLI and AWS SDKs as part of upload requests
We decided that simply accepting and storing data that may not match the checksum that the API client supplied was a bad thing.
thanks for posting!
Yeah, and AWS SDK maintainers are only giving us the cold shoulder on this
Iâm glad to see that the B2 team has also noticed this issue.
Hereâs my temporary solution:
I commented out all the configurations related to S3 in app.yml
and successfully rebuilt Discourse. This does not affect access to previously uploaded files on my site, and any attachments uploaded during this period will be stored locally without causing any issues.
btw, I still wondering, why not add an option to keep the assets on the Discourse core server (I mean, without storing them on S3?
???
That is how Discourse works out of the box, you need to opt-in explicitly with shipping assets to an object storage service.
I mean, it might be good to have an option to keep Discourse core assets (JS, CSS, etc.) on the local server. At the same time, only user-uploaded files would be stored in S3
You can do that by not enabling âuse_s3â, but they are small, Iâd just push them and not worry about the wasted space.
Please help me understand correctly.
Do you mean to set DISCOURSE_USE_S3: false
in app.yml
?
I would like to do this because my Discourse server is in Asia, and B2 only has servers in the US.
Also, this time, the AWS SDK issue is related to asset management, right?
So if I store assets on the local server, it may avoid the issue (if I understand the situation correctly).
The issue is related to removing assets from S3. If youâll remove just the line that tries to remove the unused assets it will work right now. And it sounds like soon the problem will be resolved. This is the easiest solution. Itâs what I recommend. Thatâs my best free answer.
Thank you, thatâs very helpful for me!
This is because, in the S3 API definition, the DeleteObjects operation has httpChecksum.requestChecksumRequired
set to true
, in contrast to, say, PutObject, which sets it to false
. So the SDK is honoring WHEN_REQUIRED
, because the checksum is required.
All of the AWS SDKs are now generated from the API definition, with minimal extra code to cover edge situations. The code sees that DeleteObjects
requires a checksum, and the way to do that now is to use one of the new checksums, rather than Content-MD5
.
Unfortunately, AWS didnât see fit to provide a âuse Content-MD5
like you used toâ configuration. They donât tend to consider the third-party object storage ecosystem when they make these sorts of changes, since they donât really need to. The only time that stuff like this gets fixed is when it accidentally hoses one of their own services in some corner case.

B2 only has servers in the US
Not that it particularly helps if youâre in Asia, but, for the record, we also have data centers in Europe (Amsterdam, Netherlands) and, since the beginning of this year, Canada (Toronto).
I canât make any promises, but we are definitely considering an Asia location.
Also, if you use a CDN, it doesnât matter too much where the origin server is.
Youâre right!
Backblaze does not support x-amz-checksum-crc32
, and Discourseâs newer AWS SDK version may have enabled this by default.
so i went into the app:
./launcher enter app
and uninstalled the current version of AWS SDK:
gem uninstall aws-sdk-s3 aws-sdk-core aws-sdk-kms
and installed the one that the backblaze guy said would work:
gem install aws-sdk-core -v â~> 3.215.1â
gem install aws-sdk-kms -v â~> 1.96.0â
gem install aws-sdk-s3 -v â~> 1.177.0â
then I rebuilt the app and it worked!