Can't rebuild due to AWS SDK gem bump and new AWS Data Integrity Protections

[Yt.w]

Hello, Anyone can help me?

I, [2025-02-24T17:35:41.516926 #1]  INFO -- : Terminating async processes
I, [2025-02-24T17:35:41.516987 #1]  INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/15/bin/postmaster -D /etc/postgresql/15/main pid: 38
105:signal-handler (1740418541) Received SIGTERM scheduling shutdown...
I, [2025-02-24T17:35:41.517067 #1]  INFO -- : Sending TERM to exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf pid: 105
2025-02-24 17:35:41.517 UTC [38] LOG:  received fast shutdown request
2025-02-24 17:35:41.518 UTC [38] LOG:  aborting any active transactions
2025-02-24 17:35:41.521 UTC [38] LOG:  background worker "logical replication launcher" (PID 52) exited with exit code 1
105:M 24 Feb 2025 17:35:41.523 # User requested shutdown...
105:M 24 Feb 2025 17:35:41.523 * Saving the final RDB snapshot before exiting.
2025-02-24 17:35:41.525 UTC [47] LOG:  shutting down
2025-02-24 17:35:41.538 UTC [47] LOG:  checkpoint starting: shutdown immediate
2025-02-24 17:35:41.550 UTC [47] LOG:  checkpoint complete: wrote 0 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.005 s, sync=0.001 s, total=0.025 s; sync files=0, longest=0.000 s, average=0.000 s; distance=0 kB, estimate=1122 kB
2025-02-24 17:35:41.581 UTC [38] LOG:  database system is shut down
105:M 24 Feb 2025 17:35:41.657 * DB saved on disk
105:M 24 Feb 2025 17:35:41.657 # Redis is now ready to exit, bye bye...


FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && sudo -E -u discourse bundle exec rake s3:upload_assets failed with return #<Process::Status: pid 2845 exit 1>
Location of failure: /usr/local/lib/ruby/gems/3.3.0/gems/pups-1.2.1/lib/pups/exec_command.rb:132:in `spawn'
exec failed with the params {"cd"=>"$home", "cmd"=>["sudo -E -u discourse bundle exec rake s3:upload_assets", "sudo -E -u discourse bundle exec rake s3:expire_missing_assets"]}
bootstrap failed with exit code 1
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.
c08b2ad74e9f1171f7677979af4e41a46be92877a0783350b2f020359895a431

[Jagster (Jakke Lehtonen)]

The actual error happened much earlier. You should find it.

[Yt.w]

I, [2025-02-24T18:17:21.167441 #1]  INFO -- : > cd /var/www/discourse && sudo -E -u discourse bundle exec rake s3:upload_assets
rake aborted!
Aws::S3::Errors::InvalidArgument: Unsupported header 'x-amz-checksum-crc32' received for this API call. (Aws::S3::Errors::InvalidArgument)
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/seahorse/client/plugins/raise_response_errors.rb:17:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.182.0/lib/aws-sdk-s3/plugins/sse_cpk.rb:24:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.182.0/lib/aws-sdk-s3/plugins/dualstack.rb:21:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.182.0/lib/aws-sdk-s3/plugins/accelerate.rb:43:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/checksum_algorithm.rb:169:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:16:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/invocation_id.rb:16:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/idempotency_token.rb:19:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/param_converter.rb:26:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/seahorse/client/plugins/request_callback.rb:89:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/response_paging.rb:12:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/seahorse/client/plugins/response_target.rb:24:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/telemetry.rb:39:in `block in call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/telemetry/no_op.rb:29:in `in_span'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/telemetry.rb:53:in `span_wrapper'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/telemetry.rb:39:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/seahorse/client/request.rb:72:in `send_request'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.182.0/lib/aws-sdk-s3/client.rb:17315:in `put_object'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.182.0/lib/aws-sdk-s3/object.rb:2994:in `block in put'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/user_agent.rb:69:in `metric'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.182.0/lib/aws-sdk-s3/object.rb:2993:in `put'
/var/www/discourse/lib/s3_helper.rb:82:in `upload'
/var/www/discourse/lib/tasks/s3.rake:41:in `block in upload'
/var/www/discourse/lib/tasks/s3.rake:41:in `open'
/var/www/discourse/lib/tasks/s3.rake:41:in `upload'
/var/www/discourse/lib/tasks/s3.rake:197:in `block (2 levels) in <main>'
/var/www/discourse/lib/tasks/s3.rake:197:in `each'
/var/www/discourse/lib/tasks/s3.rake:197:in `block in <main>'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/exe/rake:27:in `<top (required)>'
/usr/local/bin/bundle:25:in `load'
/usr/local/bin/bundle:25:in `<main>'
Tasks: TOP => s3:upload_assets
(See full trace by running task with --trace)
I, [2025-02-24T18:17:32.242989 #1]  INFO -- : Installing CORS rules...
skipping
Skipping: assets/break_string-cc617154cd957804f2f6a1f3bc68258c9cdca3d4b9a322bf777d145fed04790e.js
Skipping: assets/break_string-cc617154cd957804f2f6a1f3bc68258c9cdca3d4b9a322bf777d145fed04790e.br.js
Skipping: assets/break_string-cc617154cd957804f2f6a1f3bc68258c9cdca3d4b9a322bf777d145fed04790e.gz.js
Skipping: assets/break_string-cc617154cd957804f2f6a1f3bc68258c9cdca3d4b9a322bf777d145fed04790e.js.map
Skipping: assets/service-worker-9312562dd27728d5fd952130f6bb72f4fd253fec8ef551a6e7c3c8ecd9b256da.js
Skipping: assets/service-worker-9312562dd27728d5fd952130f6bb72f4fd253fec8ef551a6e7c3c8ecd9b256da.br.js
Skipping: assets/service-worker-9312562dd27728d5fd952130f6bb72f4fd253fec8ef551a6e7c3c8ecd9b256da.gz.js
Skipping: assets/service-worker-9312562dd27728d5fd952130f6bb72f4fd253fec8ef551a6e7c3c8ecd9b256da.js.map
Skipping: assets/locales/i18n-3b40e842fd72b9bcc74ea83e094c823cd9ca535e4ecc5e78722e6f99d3656137.js
Skipping: assets/locales/i18n-3b40e842fd72b9bcc74ea83e094c823cd9ca535e4ecc5e78722e6f99d3656137.br.js
Skipping: assets/locales/i18n-3b40e842fd72b9bcc74ea83e094c823cd9ca535e4ecc5e78722e6f99d3656137.gz.js
Skipping: assets/scripts/discourse-test-listen-boot-c65930f97c9935680e942f8e32df616cc91ab7c9371b86db6e5ddf9ad868ae22.js
Skipping: assets/scripts/discourse-test-listen-boot-c65930f97c9935680e942f8e32df616cc91ab7c9371b86db6e5ddf9ad868ae22.br.js
Skipping: assets/scripts/discourse-test-listen-boot-c65930f97c9935680e942f8e32df616cc91ab7c9371b86db6e5ddf9ad868ae22.gz.js
Uploading: assets/locales/ar-a9eaeaec3fdb08b678d93b1eb6bd3dc9331b5e2f441daff74cbd5eea246f1176.js

I, [2025-02-24T18:17:32.244254 #1]  INFO -- : Terminating async processes
I, [2025-02-24T18:17:32.244314 #1]  INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/15/bin/postmaster -D /etc/postgresql/15/main pid: 38
105:signal-handler (1740421052) Received SIGTERM scheduling shutdown...
I, [2025-02-24T18:17:32.244356 #1]  INFO -- : Sending TERM to exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf pid: 105
2025-02-24 18:17:32.244 UTC [38] LOG:  received fast shutdown request
2025-02-24 18:17:32.312 UTC [38] LOG:  aborting any active transactions
2025-02-24 18:17:32.316 UTC [38] LOG:  background worker "logical replication launcher" (PID 52) exited with exit code 1
105:M 24 Feb 2025 18:17:32.317 # User requested shutdown...
105:M 24 Feb 2025 18:17:32.317 * Saving the final RDB snapshot before exiting.
2025-02-24 18:17:32.317 UTC [47] LOG:  shutting down
2025-02-24 18:17:32.319 UTC [47] LOG:  checkpoint starting: shutdown immediate
2025-02-24 18:17:32.349 UTC [47] LOG:  checkpoint complete: wrote 25 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.023 s, sync=0.002 s, total=0.033 s; sync files=16, longest=0.001 s, average=0.001 s; distance=77 kB, estimate=915 kB
2025-02-24 18:17:32.382 UTC [38] LOG:  database system is shut down
105:M 24 Feb 2025 18:17:32.479 * DB saved on disk
105:M 24 Feb 2025 18:17:32.479 # Redis is now ready to exit, bye bye...

could you kindly teach me how to trouble shooting on this?

[Jagster (Jakke Lehtonen)]

I don’t know but your AWS settings looks like to be broken somehow. At least.

[Yt.w]

@Falco I guess the newest version of discourse do not support BackBlaze any more

[Arkshine]

I found this tweet, not sure if directly related: https://x.com/jkostolansky/status/1880623631145030054

Announcement: S3 default integrity change

[Yt.w]

yes, I guess that’s the issue
Aws::S3::Errors::InvalidArgument: Unsupported header 'x-amz-checksum-crc32' received for this API call. (Aws::S3::Errors::InvalidArgument)

are there any way to downgrade aws-sdk-s3 gem manually?

[pfaffman (Jay Pfaffman)]

@Yt.w

exec failed with the params {"cd"=>"$home", "cmd"=>["sudo -E -u discourse bundle exec rake s3:upload_assets", "sudo -E -u discourse bundle exec rake s3:expire_missing_assets"]}

Probably something with your S3 setup? Keys bad? Credit card expired?

Have you had S3 configured for a while?

[Yt.w]

before upgrade to the newest version there are no any issue.

I checked B2 that I did not meet any cap

[Yt.w]

S3-Compatible API

x-amz-checksum-crc32 is not supported by b2. I guess this the reason

[Falco]

Try adding to your ENV

AWS_REQUEST_CHECKSUM_CALCULATION: WHEN_REQUIRED
AWS_RESPONSE_CHECKSUM_VALIDATION: WHEN_REQUIRED

[Yt.w]

Hello, it still give me error

env:
  DISCOURSE_ENABLE_CORS: true
  AWS_REQUEST_CHECKSUM_CALCULATION: WHEN_REQUIRED
  AWS_RESPONSE_CHECKSUM_VALIDATION: WHEN_REQUIRED
  
  DISCOURSE_USE_S3: true

rake aborted!
Aws::S3::Errors::InvalidArgument: Unsupported header 'x-amz-checksum-crc32' received for this API call. (Aws::S3::Errors::InvalidArgument)

FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && sudo -E -u discourse bundle exec rake s3:expire_missing_assets failed with return #<Process::Status: pid 2900 exit 1>
Location of failure: /usr/local/lib/ruby/gems/3.3.0/gems/pups-1.2.1/lib/pups/exec_command.rb:132:in `spawn'
exec failed with the params {"cd"=>"$home", "cmd"=>["sudo -E -u discourse bundle exec rake s3:upload_assets", "sudo -E -u discourse bundle exec rake s3:expire_missing_assets"]}
bootstrap failed with exit code 1
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.

[Falco]

@Yt.w

Pups::ExecError: cd /var/www/discourse && sudo -E -u discourse bundle exec rake s3:expire_missing_assets failed with return #<Process::Status: pid 2900 exit 1>

It’s reported that the AWS SDK is not respecting those ENVs for DELETE operations, so you need to remove the

- sudo -E -u discourse bundle exec rake s3:expire_missing_assets

line for now.

[pfaffman (Jay Pfaffman)]

So should someone update the b2 section of Configure an S3 compatible object storage provider for uploads?

[Falco]

After confirming it works? Definitely.

Also, it will affect most providers, so most clones will need adjustments.

[Yt.w]

Thank you, I will try in the maintenance time,
does delete
- sudo -E -u discourse bundle exec rake s3:expire_missing_assets
give any side effect?

[Falco]

Yes, assets won’t be removed from the bucket anymore.

[Yt.w]

So if I don’t remove assets manually, in theory, it will expand infinitely?

[Falco]

Yes, that is exactly what is going to happen.

[Yt.w]

It doesn’t sound like a super critical issue, but is there any way to fix it?
There wasn’t any issue in the previous version, right?