העלאת גיבוי ל-S3 נכשלת ב-scaleway (העלאה מרובת חלקים)

סיכום מבוסס בינה מלאכותית

Users markersocial and Freez report failures when uploading Discourse backups to Scaleway S3 using multipart uploads, with errors like “The specified multipart upload does not exist” or “Your request was not formed properly.” Falco confirms multipart uploads are unavoidable for large files (>15MB) and suggests contacting Scaleway support. Freez reaches out to Scaleway, who request HTTP-level logs to debug — Falco notes Discourse’s open-source nature allows Scaleway to inspect the exact SDK calls. Later, Freez reports intermittent success (7/10 backups working), then full success Freez, while markersocial still experiences failures, suggesting Scaleway’s fixes may not be fully rolled out. Falco notes the advantage of S3 compatibility: fixes on Scaleway’s side require no Discourse changes.

Running on 2.6.1 stable with a s3 clone (Scaleway). Here is the log:

[2021-01-30 09:52:59] Uploading archive...
[2021-01-30 09:55:36] EXCEPTION: failed to abort multipart upload: The specified multipart upload does not exist. The upload ID might be invalid, or the multipart upload might have been aborted or completed.
[2021-01-30 09:55:36] /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/aws-sdk-s3-1.83.2/lib/aws-sdk-s3/multipart_file_uploader.rb:94:in `rescue in abort_upload'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/aws-sdk-s3-1.83.2/lib/aws-sdk-s3/multipart_file_uploader.rb:82:in `abort_upload'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/aws-sdk-s3-1.83.2/lib/aws-sdk-s3/multipart_file_uploader.rb:78:in `upload_parts'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/aws-sdk-s3-1.83.2/lib/aws-sdk-s3/multipart_file_uploader.rb:51:in `upload'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/aws-sdk-s3-1.83.2/lib/aws-sdk-s3/file_uploader.rb:38:in `upload'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/aws-sdk-s3-1.83.2/lib/aws-sdk-s3/customizations/object.rb:342:in `upload_file'
/var/www/discourse/lib/backup_restore/s3_backup_store.rb:39:in `upload_file'
/var/www/discourse/lib/backup_restore/backuper.rb:301:in `upload_archive'
/var/www/discourse/lib/backup_restore/backuper.rb:41:in `run'
/var/www/discourse/script/spawn_backup_restore.rb:9:in `backup'
/var/www/discourse/script/spawn_backup_restore.rb:34:in `block in <main>'
/var/www/discourse/script/spawn_backup_restore.rb:4:in `fork'
/var/www/discourse/script/spawn_backup_restore.rb:4:in `<main>'
[2021-01-30 09:55:36] Deleting old backups...
[2021-01-30 09:55:36] Cleaning stuff up...
[2021-01-30 09:55:36] Removing archive from local storage...
[2021-01-30 09:55:36] Removing '.tar' leftovers...
[2021-01-30 09:55:36] Marking backup as finished...
[2021-01-30 09:55:36] Refreshing disk stats...

I have the same configuration on a couple of other servers and backups are working fine (though the backup sizes are significantly smaller, so multipart upload likely isn’t a factor). It looks like they support multipart uploads also: Managing multipart uploads | Scaleway Documentation - so not sure what the problem is.

Same thing for me, with Scaleway S3 :frowning:

My first backup failed 9 days ago:

EXCEPTION: multipart upload failed:
Your request was not formed properly;
Your request was not formed properly

Then, every one or two days:

EXCEPTION: failed to abort multipart upload:
The specified multipart upload does not exist.
The upload ID might be invalid, or
the multipart upload might have been aborted or completed.

@Falco what can we do to understand what is going on?
Thank you

Replied to this yesterday. Contact scaleway

Done
But their support needs more information :slight_smile:
They need a listing of the parts, to see if we get a response or an error message for each parts.

Is their way to enable logging (http requests?) in aws gems?

Thank you

It’s their lucky day, as Discourse is open source they can see exactly what is happening!

Backtrace shows error coming from

which is

Which is a call to this method in the SDK:

Per documentation, it will default to using a MultiPart upload when a file is larger than 15MB.

Looking at the source code, looks like this method is used:

And line 74 is getting an error.

It’s working again, without any modifications :grimacing:
7/10 are successful now

Great news. The good thing about implementing against the de facto S3 spec is that they fix on their side and we need no changes.

Still giving me the same error unfortunately :confused:
Hopefully they just have not fully rolled out the fixes yet.

Backup process 100% successful now!
Issue “closed” for me.