Migrating old uploads to S3

While my new uploads are uploading very correctly on Amazon S3, this command is failing to shift the old one from local to S3:

DISCOURSE_S3_BUCKET="bucketname" DISCOURSE_S3_REGION="ap-south-1" DISCOURSE_S3_ACCESS_KEY_ID="AKIchangedabitUNM2" DISCOURSE_S3_SECRET_ACCESS_KEY="rX6Pf3C_changedabit_YKKtdVrnbw6B" DISCOURSE_S3_CDN_URL="https://bucketname.s3.ap-south-1.amazonaws.com" rake uploads:migrate_to_s3

While 15 days ago, this command worked like a charm.
Today its giving this error:

Checking if default already migrated...
7 of 14 uploads are not migrated to S3. S3 migration failed for db 'default'.
1 posts are not remapped to new S3 upload URL. S3 migration failed for db 'default'.
Looking for missing uploads on: default

0 post uploads are missing.

No posts require rebaking
Migrating uploads to S3 for 'default'...
Some uploads were not migrated to the new scheme. Please run these commands in the rails console

SiteSetting.migrate_to_new_scheme = true
Jobs::MigrateUploadScheme.new.execute(nil)

I also gave these 2 commands in rails console, but they seemed to do no benefit:

SiteSetting.migrate_to_new_scheme = true
Jobs::MigrateUploadScheme.new.execute(nil)

Note: My uploads are configured to be in a subfolder on the bucket bucketname/uploads.

3 Likes

Any idea here @vinothkannans?

Is it returning any errors? I’m unable to see any recent big changes in the code of the uploads:migrate_to_s3 rake task.

3 Likes

I had a similar problem. It had to do with the path of the uploads not matching what that test expects (like it’s missing ‘default’ in the path of something).

What I did was (in broad and likely unhelpful terms) to see what search is doing to find those uploads that it thinks are bad and fix them

3 Likes

I happen to solve it accidentally and in a way that I’m still not sure how it got solved.

I changed the command a couple of times mostly appending or removing the word ‘dualstack’ in the command and in CDN url field in Discourse Settings: https://bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com
But still it didn’t seem to work. Then randomly I gave the two same commands (told in the OP) in the rails console once again.
And 1 step I did randomly around this time was append the word /uploads to the bucketname in the command.

And ‘randomly’ (I need another random word here :slight_smile: ), I succeeded.
So, I can’t say who was the hero of the resolution here.
Thanks for listening anyway.

2 Likes

Whatever error was coming/showing, I already included in my original post. But I agree that that error/msg wasn’t very helpful in diagnosis.

Alas, today again, I thought since I couldn’t understand the problem in migration, I created 1 upload locally. Then tried to shift that again to S3 with this command:

DISCOURSE_S3_BUCKET="bxyzbucket1/uploads" DISCOURSE_S3_REGION="ap-south-1" DISCOURSE_S3_ACCESS_KEY_ID="AKIAchangedBRF" DISCOURSE_S3_SECRET_ACCESS_KEY="H0P6Oo8changed1AuLkRDCQK8" DISCOURSE_S3_CDN_URL="https://bxyzbucket1.s3.ap-south-1.amazonaws.com/uploads" rake uploads:migrate_to_s3

And though it shifted the 1 new upload to S3, but with this error included:

1 of 14 uploads are not migrated to S3. S3 migration failed for db 'default'.
1 posts are not remapped to new S3 upload URL. S3 migration failed for db 'default'.

What does this error signify? Can it pose some problem if there are thousands of images to be shifted from ‘Local >> S3’? @vinothkannans

Can you please run the following command in rails console and post the output here

Upload.by_users.where("url NOT LIKE '//%' AND url NOT LIKE '#{GlobalSetting.relative_url_root}/uploads/default/original/_X/%'")
3 Likes

Sorry that I was away from this operation for some time.
I’m running the rails command sent by you, but no output is coming:

I still get the error: " S3 migration failed for db ‘default’"
And although post uploads/images are showing ok in my bucket and on site, but site logos etc (under Admin Settings> Branding) are missing and I’m not even able to update them with fresh ones. They just stay blank.

I also ran (with no errors): rake posts:rebake_uncooked_posts and rake posts:rebake

@vinothkannans Even after a year, when I happen to repeat the process for some reason, still same error is coming.