Backup Upload to S3 Fails on scaleway (multipart upload)

markersocial · January 30, 2021, 10:53am

Running on 2.6.1 stable with a s3 clone (Scaleway). Here is the log:

[2021-01-30 09:52:59] Uploading archive...
[2021-01-30 09:55:36] EXCEPTION: failed to abort multipart upload: The specified multipart upload does not exist. The upload ID might be invalid, or the multipart upload might have been aborted or completed.
[2021-01-30 09:55:36] /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/aws-sdk-s3-1.83.2/lib/aws-sdk-s3/multipart_file_uploader.rb:94:in `rescue in abort_upload'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/aws-sdk-s3-1.83.2/lib/aws-sdk-s3/multipart_file_uploader.rb:82:in `abort_upload'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/aws-sdk-s3-1.83.2/lib/aws-sdk-s3/multipart_file_uploader.rb:78:in `upload_parts'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/aws-sdk-s3-1.83.2/lib/aws-sdk-s3/multipart_file_uploader.rb:51:in `upload'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/aws-sdk-s3-1.83.2/lib/aws-sdk-s3/file_uploader.rb:38:in `upload'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/aws-sdk-s3-1.83.2/lib/aws-sdk-s3/customizations/object.rb:342:in `upload_file'
/var/www/discourse/lib/backup_restore/s3_backup_store.rb:39:in `upload_file'
/var/www/discourse/lib/backup_restore/backuper.rb:301:in `upload_archive'
/var/www/discourse/lib/backup_restore/backuper.rb:41:in `run'
/var/www/discourse/script/spawn_backup_restore.rb:9:in `backup'
/var/www/discourse/script/spawn_backup_restore.rb:34:in `block in <main>'
/var/www/discourse/script/spawn_backup_restore.rb:4:in `fork'
/var/www/discourse/script/spawn_backup_restore.rb:4:in `<main>'
[2021-01-30 09:55:36] Deleting old backups...
[2021-01-30 09:55:36] Cleaning stuff up...
[2021-01-30 09:55:36] Removing archive from local storage...
[2021-01-30 09:55:36] Removing '.tar' leftovers...
[2021-01-30 09:55:36] Marking backup as finished...
[2021-01-30 09:55:36] Refreshing disk stats...

I have the same configuration on a couple of other servers and backups are working fine (though the backup sizes are significantly smaller, so multipart upload likely isn’t a factor). It looks like they support multipart uploads also: S3 Object Storage - Multipart Upload - Scaleway - so not sure what the problem is.

Freez · February 11, 2021, 2:23pm

Same thing for me, with Scaleway S3

My first backup failed 9 days ago:

EXCEPTION: multipart upload failed:
Your request was not formed properly;
Your request was not formed properly

Then, every one or two days:

EXCEPTION: failed to abort multipart upload:
The specified multipart upload does not exist.
The upload ID might be invalid, or
the multipart upload might have been aborted or completed.

@Falco what can we do to understand what is going on?
Thank you

Falco · February 11, 2021, 2:26pm

Replied to this yesterday. Contact scaleway

Freez · February 19, 2021, 4:33pm

Done
But their support needs more information
They need a listing of the parts, to see if we get a response or an error message for each parts.

Is their way to enable logging (http requests?) in aws gems?

Thank you

Falco · February 19, 2021, 5:17pm

It’s their lucky day, as Discourse is open source they can see exactly what is happening!

Backtrace shows error coming from

which is

github.com

discourse/discourse/blob/main/lib/backup_restore/s3_backup_store.rb#L39


      
          end
          
          def delete_file(filename)
            obj = s3_helper.object(filename)
          
            if obj.exists?
              obj.delete
              reset_cache
            end
          end
          
          def download_file(filename, destination_path, failure_message = nil)
            s3_helper.download_file(filename, destination_path, failure_message)
          end
          
          def upload_file(filename, source_path, content_type)
            obj = s3_helper.object(filename)
            raise BackupFileExists.new if obj.exists?
          
            obj.upload_file(source_path, content_type: content_type)
            reset_cache

Which is a call to this method in the SDK:

https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Object.html#upload_file-instance_method

Per documentation, it will default to using a MultiPart upload when a file is larger than 15MB.

Looking at the source code, looks like this method is used:

github.com

aws/aws-sdk-ruby/blob/version-3/gems/aws-sdk-s3/lib/aws-sdk-s3/multipart_file_uploader.rb#L71-L80


      
              )
            )
          end
          
          def upload_parts(upload_id, source, options)
            pending = PartList.new(compute_parts(upload_id, source, options))
            completed = PartList.new
            errors = upload_in_threads(pending, completed, options)
            if errors.empty?
              completed.to_a.sort_by { |part| part[:part_number] }

And line 74 is getting an error.

Freez · March 16, 2021, 5:35pm

It’s working again, without any modifications
7/10 are successful now

Falco · March 16, 2021, 5:50pm

Great news. The good thing about implementing against the de facto S3 spec is that they fix on their side and we need no changes.

markersocial · March 18, 2021, 8:07am

Still giving me the same error unfortunately
Hopefully they just have not fully rolled out the fixes yet.

Freez · March 23, 2021, 9:20am

Backup process 100% successful now!
Issue “closed” for me.

Topic		Replies	Views
Restore Failure - S3 (compatible) backup Support	4	1240	August 16, 2020
Using Scaleway s3-compatible object storage Support	27	3699	April 6, 2021
If backup to s3 upload fails, backup is stored locally but ignores the backup limit Bug	3	468	April 28, 2021
Can't restore backup with images on S3 Support	3	523	December 8, 2022
Occasional backup to S3 failure: "multipart upload failed: no tomes available" Installation backups , s3	0	292	December 5, 2023

Backup Upload to S3 Fails on scaleway (multipart upload)

Related topics