Backup Upload to S3 Fails on scaleway (multipart upload)

Support
1

Running on 2.6.1 stable with a s3 clone (Scaleway). Here is the log:

[2021-01-30 09:52:59] Uploading archive...
[2021-01-30 09:55:36] EXCEPTION: failed to abort multipart upload: The specified multipart upload does not exist. The upload ID might be invalid, or the multipart upload might have been aborted or completed.
[2021-01-30 09:55:36] /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/aws-sdk-s3-1.83.2/lib/aws-sdk-s3/multipart_file_uploader.rb:94:in `rescue in abort_upload'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/aws-sdk-s3-1.83.2/lib/aws-sdk-s3/multipart_file_uploader.rb:82:in `abort_upload'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/aws-sdk-s3-1.83.2/lib/aws-sdk-s3/multipart_file_uploader.rb:78:in `upload_parts'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/aws-sdk-s3-1.83.2/lib/aws-sdk-s3/multipart_file_uploader.rb:51:in `upload'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/aws-sdk-s3-1.83.2/lib/aws-sdk-s3/file_uploader.rb:38:in `upload'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/aws-sdk-s3-1.83.2/lib/aws-sdk-s3/customizations/object.rb:342:in `upload_file'
/var/www/discourse/lib/backup_restore/s3_backup_store.rb:39:in `upload_file'
/var/www/discourse/lib/backup_restore/backuper.rb:301:in `upload_archive'
/var/www/discourse/lib/backup_restore/backuper.rb:41:in `run'
/var/www/discourse/script/spawn_backup_restore.rb:9:in `backup'
/var/www/discourse/script/spawn_backup_restore.rb:34:in `block in <main>'
/var/www/discourse/script/spawn_backup_restore.rb:4:in `fork'
/var/www/discourse/script/spawn_backup_restore.rb:4:in `<main>'
[2021-01-30 09:55:36] Deleting old backups...
[2021-01-30 09:55:36] Cleaning stuff up...
[2021-01-30 09:55:36] Removing archive from local storage...
[2021-01-30 09:55:36] Removing '.tar' leftovers...
[2021-01-30 09:55:36] Marking backup as finished...
[2021-01-30 09:55:36] Refreshing disk stats...

I have the same configuration on a couple of other servers and backups are working fine (though the backup sizes are significantly smaller, so multipart upload likely isn’t a factor). It looks like they support multipart uploads also: S3 Object Storage - Multipart Upload - Scaleway - so not sure what the problem is.

1 Like
2

Same thing for me, with Scaleway S3 :frowning:

My first backup failed 9 days ago:

EXCEPTION: multipart upload failed:
Your request was not formed properly;
Your request was not formed properly

Then, every one or two days:

EXCEPTION: failed to abort multipart upload:
The specified multipart upload does not exist.
The upload ID might be invalid, or
the multipart upload might have been aborted or completed.

@Falco what can we do to understand what is going on?
Thank you

1 Like
3

Replied to this yesterday. Contact scaleway

2 Likes
4

Done
But their support needs more information :slight_smile:
They need a listing of the parts, to see if we get a response or an error message for each parts.

Is their way to enable logging (http requests?) in aws gems?

Thank you

1 Like
5

It’s their lucky day, as Discourse is open source they can see exactly what is happening!

Backtrace shows error coming from

which is

Which is a call to this method in the SDK:

https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Object.html#upload_file-instance_method

Per documentation, it will default to using a MultiPart upload when a file is larger than 15MB.

Looking at the source code, looks like this method is used:

And line 74 is getting an error.

6 Likes
6

It’s working again, without any modifications :grimacing:
7/10 are successful now

3 Likes
7

Great news. The good thing about implementing against the de facto S3 spec is that they fix on their side and we need no changes.

3 Likes
8

Still giving me the same error unfortunately :confused:
Hopefully they just have not fully rolled out the fixes yet.

9

Backup process 100% successful now!
Issue “closed” for me.