S3 (not AWS) backups stopped working, presumably since an update

I have a bucket set but it seems to be trying to use default?

I use GarageHQ as a lightweight MinIO alternative.

I’m on 3.6.0.beta1-dev
( ed6beea336 )

[2025-09-14 10:40:34] Removing tmp ‘/var/www/discourse/tmp/backups/default/2025-09-14-104012’ directory…
[2025-09-14 10:40:34] Uploading archive…
[2025-09-14 10:40:35] EXCEPTION: Bucket not found: default

[2025-09-14 10:40:35] /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.227.0/lib/seahorse/client/plugins/raise_response_errors.rb:17:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.182.0/lib/aws-sdk-s3/plugins/sse_cpk.rb:24:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.182.0/lib/aws-sdk-s3/plugins/dualstack.rb:21:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.182.0/lib/aws-sdk-s3/plugins/accelerate.rb:43:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.227.0/lib/aws-sdk-core/plugins/checksum_algorithm.rb:169:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.227.0/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:16:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.227.0/lib/aws-sdk-core/plugins/invocation_id.rb:16:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.227.0/lib/aws-sdk-core/plugins/idempotency_token.rb:19:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.227.0/lib/aws-sdk-core/plugins/param_converter.rb:26:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.227.0/lib/seahorse/client/plugins/request_callback.rb:89:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.227.0/lib/aws-sdk-core/plugins/response_paging.rb:12:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.227.0/lib/seahorse/client/plugins/response_target.rb:24:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.227.0/lib/aws-sdk-core/plugins/telemetry.rb:39:in `block in call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.227.0/lib/aws-sdk-core/telemetry/no_op.rb:29:in `in_span'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.227.0/lib/aws-sdk-core/plugins/telemetry.rb:53:in `span_wrapper'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.227.0/lib/aws-sdk-core/plugins/telemetry.rb:39:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.227.0/lib/seahorse/client/request.rb:72:in `send_request'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.182.0/lib/aws-sdk-s3/client.rb:3710:in `create_multipart_upload'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.182.0/lib/aws-sdk-s3/multipart_file_uploader.rb:67:in `initiate_upload'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.182.0/lib/aws-sdk-s3/multipart_file_uploader.rb:58:in `upload'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.182.0/lib/aws-sdk-s3/file_uploader.rb:42:in `block in upload'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.227.0/lib/aws-sdk-core/plugins/user_agent.rb:90:in `metric'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.182.0/lib/aws-sdk-s3/file_uploader.rb:40:in `upload'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.182.0/lib/aws-sdk-s3/customizations/object.rb:477:in `block in upload_file'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.227.0/lib/aws-sdk-core/plugins/user_agent.rb:90:in `metric'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.182.0/lib/aws-sdk-s3/customizations/object.rb:476:in `upload_file'
/var/www/discourse/lib/backup_restore/s3_backup_store.rb:48:in `upload_file'
/var/www/discourse/lib/backup_restore/backuper.rb:351:in `upload_archive'
/var/www/discourse/lib/backup_restore/backuper.rb:41:in `run'
/var/www/discourse/script/spawn_backup_restore.rb:9:in `backup'
/var/www/discourse/script/spawn_backup_restore.rb:31:in `block in <main>'
/var/www/discourse/script/spawn_backup_restore.rb:4:in `fork'
/var/www/discourse/script/spawn_backup_restore.rb:4:in `<main>'

The last backup was yesterday but now it’s not longer working, other services can backup to my local S3 without issue still

Anyone know how i can fix this?

Yeah. AWS broke the S3 library for lots of other services.

Can't rebuild due to AWS SDK gem bump and new AWS Data Integrity Protections has some work-arounds.

I created this aws-revert-template.yml to revert to an older version. I don’t know if it works anymore; my file is dated 2025-08-06T05:00:00Z

Hmm, i’m not fully convinced the is the problem though

unless i’m mistaken i was on 3.6 for longer than 2 days, and yet i got a backup 2 days and 10 days ago.

Not much more. I’m thinking that you’re mistaken.

Here’s the commit you’re on:
UX: control event display through a site setting (#34795) · discourse/discourse@ed6beea · GitHub

But also:

Is your bucket named “default”?

yes I’m on the commit now, I wasn’t initially when I posted this issue, I updated to rule out a bug, I was on 3.5 when I posted this issue

no, as stated in the OP this is why I don’t believe it’s the same issue you are referring to, my bucket is specified , ive tried both in the UI settings of the backup section and via env vars in the yaml, it still tries to use default regardless

to clarify nothing has changed on the discourse or garage end since it stopped working except for an update to 3.5 and then to 3.6

1 Like

perhaps someone with actual S3 could check to see if it’s correctly selecting a bucket? same for some other 3rd parties like backblaze, r2 etc?

So i created a bucket named “default” and it works as intended, so yes, it’s not correctly picking the bucket name i tell it.

So this seems like a bug in Discourse.

Except thousands of people and CDCK hosting would also likely be affected and they are not. It’s much more likely that you deleted or added a character to a line in your app.yml.

You can do stuff like this:

grep -i S3 /var/discourse/containers/*
docker exec -it app bash -c 'grep s3 /var/www/discourse/config/discourse.conf

And if you configured s3 in settings instead of with ENV variables, then you can have a look at Configure an S3 compatible object storage provider for uploads to see what those look like.

nothing is wrong with my setup nothing was changed except the discourse update. Ive tried configuration in the yaml and configuration in the UI, the UI settings literally show it as “cyanlabs-community” and yet the backup still tries to save to “default”

despite having s3 configured in the UI the discourse.conf file does not contain any s3 references as per your second command

This though means the bucket is not there, can you try creating a brand new bucket and creds and see if uploading there works?

1 Like

Thanks but I feel like i am going around in circles here.

the bucket default did not exist at the start of this conversation, i have since created it and it uses that bucket despite having the bucket cyanlabs-community set in the settings.

both cyanlabs-community and default exist in the s3 instance, but Discourse will not use cyanlabs-community no matter what i try

new bucket cyanlabsdiscourse

result of backup

[2025-09-22 15:14:59] Finalizing backup...
[2025-09-22 15:14:59] Finalizing database dump file: cyanlabs-official-community-2025-09-22-151437-v20250916082012.sql.gz
[2025-09-22 15:14:59] Removing tmp '/var/www/discourse/tmp/backups/default/2025-09-22-151437' directory...
[2025-09-22 15:14:59] Uploading archive...
[2025-09-22 15:15:37] Executing the after_create_hook for the backup...
[2025-09-22 15:15:37] Deleting old backups...
[2025-09-22 15:15:37] Cleaning stuff up...
[2025-09-22 15:15:37] Removing archive from local storage...
[2025-09-22 15:15:37] Removing '.tar' leftovers...
[2025-09-22 15:15:37] Marking backup as finished...
[2025-09-22 15:15:37] Refreshing disk stats...
[2025-09-22 15:15:37] Notifying 'CyanLabs' of the end of the backup...
[2025-09-22 15:15:40] Finished!

cyanlabsdiscourse bucket

default bucket

Whats interesting is it still attempts to make the connection at bucketname.s3.domain.tld, e.g cyanlabsdiscourse.s3.domain.tld as without adding a DNS record it wouldn’t work at that sub sub domain. so the setting is being honoured for the url but it seems to be ignored by the bucket selection

I feel like i have tried everything, for now i will just use the default bucket i guess

AWS can be mysterious! Sorry you were not able to get your preferred bucket name working. Hard to help without being able to replicate it, so I guess you will have to live with using default as the bucket name.