Unable to delete backups - Aws::S3::Errors::AccessDenied in logs

I run a number of Discourse instances, all self-hosted. All of them have backups in Amazon S3, and I set up a custom IAM user and Policy for each forum, which gives access to just one S3 Bucket.

Some of the forums are not deleting old backups, whereas others are doing it fine. It doesn’t seem to be related to the site settings, which are all pretty much the same. I tend to limit the number of retained backups to 10-20, and backup is done nightly.

The instances that are misbehaving have the following in common:

  • The AWS credentials work fine for creation of new backups and listing of existing ones, but raise ‘access denied’ errors when trying to delete.
  • All instances are on latest tests-passed (updated today) 3.3.0.beta5-dev
  • They don’t autodelete old backups, even those which are well beyond the age that they should have been deleted.
  • They don’t seem to respond to any combination of site settings eg maximum backups, remove older backups, s3 disable cleanup or anything.
  • Deleting the old backups in the Discourse admin UI doesn’t work - there is a Aws::S3::Errors::AccessDenied in the logs.
  • Deleting manually in AWS does (obviously) work, but that is a workaround.

Just looking to understand how I might go about debugging this further.

The JSON policies for the IAM users are all identical apart from the name of the bucket. The policies clearly do work for creation and uploading of new backups.