Backblaze S3 issue: duplicated uploads after delete

c00bt00p · March 15, 2021, 3:13pm

I am using Backblaze as S3 storage and have the clean up orphan uploads setting turned on. The problem is, instead of deleting the orphan upload, there created a duplicated file with 0 byte.
see below for example. the (2) indicates the number of files with the same name. if expanded, you will see the original file still exists, also with an 0 bytes file. Has anyone had the similar issue? is it the problem of Backblaze or the setting? Thanks.

Screen Shot 2021-03-15 at 8.00.38 AM

Falco · March 15, 2021, 3:28pm

Discourse uses the S3 API, and since it works fine with AWS S3 it leaves this to being a Backblaze problem. Maybe contact their support? I will add a note about this in Using Object Storage for Uploads (S3 & Clones)

schleifer · March 18, 2021, 7:52am

There’s an ellipsis which suggests the name of the 0-byte file is being truncated. What is the full name there? I’d wager the bucket has a file lifecycle configured and that file is a “hide marker”, as they call it.

When the lifecycle expires both versions should disappear.

c00bt00p · March 19, 2021, 11:34pm

The truncated part is “hidden”. I think you are right. It looks like the duplicated files are all recently uploaded ones. I will wait some time and see if they are gone. Thank you.

tenzan · September 15, 2021, 2:16am

Hi,
Can you please update us if the issue has been resolved?
More than a year passed since your last post, so hoping this is OK now…

rishabhlakhotia · December 12, 2021, 1:23am

The issue is not resolved yet. I just now manually deleted all the orphan files from BackBlaze manually. I think I’ll move to S3 at this point because regularly cleaning up the storage is a task in itself.

znedw · June 17, 2022, 11:22pm

Just an update with my experience…
I have a free B2 account (i.e. the first 10gb) and I don’t have a payment method added.

The backup files are rotated and the ‘hidden’ files are removed permanently after a few days. I never have more than 7 files at once (Discourse has max 5 daily backups rotation on). Since my backups are never more than ~500mb each I’ve never had to pay for anything or delete any ‘orphan’ files manually.

rahim123 · December 6, 2022, 3:32pm

Has there been any change with this issue? Has anybody tried contacting Backblaze support about this?

Bathinda · June 3, 2023, 7:46am

But the issue seems to be with ‘Uploads’!!
As the ‘Backups’ aren’t so many/too many files to manage. They are manageable, even manually.

V1ktor · June 15, 2023, 1:32pm

It seems this issue continues.

@Falco could you elaborate on what S3 operations Discourse performs to clean up orphans?

Falco · June 15, 2023, 2:02pm

github.com

discourse/discourse/blob/4b22e67c8bc1814133f58bc32f84e42372654abc/lib/s3_helper.rb#L89-L101


      
          def remove(s3_filename, copy_to_tombstone = false)
            s3_filename = s3_filename.dup
          
          
  # copy the file in tombstone
            if copy_to_tombstone && @tombstone_prefix.present?
              self.copy(get_path_for_s3_upload(s3_filename), File.join(@tombstone_prefix, s3_filename))
            end
          
          
  # delete the file
            s3_filename.prepend(multisite_upload_path) if Rails.configuration.multisite
            delete_object(get_path_for_s3_upload(s3_filename))
          rescue Aws::S3::Errors::NoSuchKey, Aws::S3::Errors::NotFound
          end

V1ktor · June 16, 2023, 8:13pm

Thanks. I see copy_to_tombstone is involved here. I couldn’t find any docs about tombstone and orphaned files, so I can only assume how it works based on what I read in the forum. Please correct me if I’m wrong:

When orphaned file is identified (clean orphan uploads grace period hours), it is copied to tombstone folder (copy_object).
It is then deleted (or supposed to be deleted) with delete_object.
When the time comes (purge deleted uploads grace period days) it is deleted from tombstone folder.

Is this correct?

Based on what I see in the Backblaze, it does appear to be copied to tombstone folder. It’s just not deleted, but instead an empty hidden version is created.

V1ktor · June 18, 2023, 2:46am

So I did contact support, and it seems there is a solution to the orphan problem. Here’s what Backblaze support said:

A few things are going on here. Looking at your account here, you have your bucket lifecycle rules set to Keep all files . If you were to change your lifecycle rules to Keep only the last version of the file , the hidden file will be deleted after 24 hours of behind hidden, and clearing up storage space.

Now, another layer to add in here is when using an S3-compatible service, any time an object delete is called, it will be hidden. And from here, depending on the lifecycle rules of the bucket, it will be deleted or remain hidden. In your case, the files remain hidden and are not being deleted due to Keep ALL files.

For an object to be deleted when an object delete is called, is to have the file version ID with the accompanying delete call, which I don’t think the integration is doing if files are just being hidden.

In order for these files to be deleted from the bucket, you’ll need to sign into your Backblaze account, go to your bucket, and update its lifecycle setting to Keep only the last version of the file. This will delete the hidden file from the bucket after 24 hours.

By by default, Backblaze sets “Keep all files” lifecycle rule for newly created buckets. That’s probably why everyone has issues with orphans. Changing lifecycle rule to “Keep only the last version of the file” deletes orphans after 24 hours once it’s hidden.

This might be worth mentioning in this thread:

Falco · June 19, 2023, 2:15pm

Nice find! Can you please edit it to the wiki?

V1ktor · June 20, 2023, 12:53pm

Didn’t realize it was a wiki. Updated.

Topic		Replies	Views
Problem with Backblaze for backup- Failed to list backups from S3: Signature validation failed support	4	1198	April 29, 2021
Rebaking old posts won't pull new S3 CDN URL after S3 bucket rename installation	14	1477	January 12, 2021
Cleaning up Uploads and Purging Uploads from S3 admins reference	1	524	April 16, 2024
REQ: Support S3 backup to a service like Backblaze support	8	1230	August 27, 2020
Uploads not getting orphaned and purged support	9	1031	April 19, 2024

Backblaze S3 issue: duplicated uploads after delete

Related Topics