While my new uploads are uploading very correctly on Amazon S3, this command is failing to shift the old one from local to S3: DISCOURSE_S3_BUCKET="bucketname" DISCOURSE_S3_REGION="ap-south-1" DISCOURSE_S3_ACCESS_KEY_ID="AKIchangedabitUNM2" DISCOURSE_S3_SECRET_ACCESS_KEY="rX6Pf3C_changedabit_YKKtdVrnbw6B" DISCOURSE_S3_CDN_URL="https://bucketname.s3.ap-south-1.amazonaws.com" rake uploads:migrate_to_s3 While 15 days ago, this command worked like a charm. Today its giving this error: Checking if default already migrated... 7 of 14 uploads are not migrated to S3. S3 migration failed for db 'default'. 1 posts are not remapped to new S3 upload URL. S3 migration failed for db 'default'. Looking for missing uploads on: default 0 post uploads are missing. No posts require rebaking Migrating uploads to S3 for 'default'... Some uploads were not migrated to the new scheme. Please run these commands in the rails console SiteSetting.migrate_to_new_scheme = true Jobs::MigrateUploadScheme.new.execute(nil) I also gave these 2 commands in rails console, but they seemed to do no benefit: SiteSetting.migrate_to_new_scheme = true Jobs::MigrateUploadScheme.new.execute(nil) Note: My uploads are configured to be in a subfolder on the bucket bucketname/uploads.

[תמונה] Bathinda: this command is failing to shift the old one from local to S3. While 15 days ago, this command worked like a charm. Is it returning any errors? I’m unable to see any recent big changes in the code of the uploads:migrate_to_s3 rake task.

I had a similar problem. It had to do with the path of the uploads not matching what that test expects (like it’s missing ‘default’ in the path of something). What I did was (in broad and likely unhelpful terms) to see what search is doing to find those uploads that it thinks are bad and fix them

I happen to solve it accidentally and in a way that I’m still not sure how it got solved. I changed the command a couple of times mostly appending or removing the word ‘dualstack’ in the command and in CDN url field in Discourse Settings: https://bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com But still it didn’t seem to work. Then randomly I gave the two same commands (told in the OP) in the rails console once again. And 1 step I did randomly around this time was append the word /uploads to the bucketname in the command. And ‘randomly’ (I need another random word here :slight_smile: ), I succeeded. So, I can’t say who was the hero of the resolution here. Thanks for listening anyway.

Whatever error was coming/showing, I already included in my original post. But I agree that that error/msg wasn’t very helpful in diagnosis. Alas, today again, I thought since I couldn’t understand the problem in migration, I created 1 upload locally. Then tried to shift that again to S3 with this command: DISCOURSE_S3_BUCKET="bxyzbucket1/uploads" DISCOURSE_S3_REGION="ap-south-1" DISCOURSE_S3_ACCESS_KEY_ID="AKIAchangedBRF" DISCOURSE_S3_SECRET_ACCESS_KEY="H0P6Oo8changed1AuLkRDCQK8" DISCOURSE_S3_CDN_URL="https://bxyzbucket1.s3.ap-south-1.amazonaws.com/uploads" rake uploads:migrate_to_s3 And though it shifted the 1 new upload to S3, but with this error included: 1 of 14 uploads are not migrated to S3. S3 migration failed for db 'default'. 1 posts are not remapped to new S3 upload URL. S3 migration failed for db 'default'. What does this error signify? Can it pose some problem if there are thousands of images to be shifted from ‘Local >> S3’? @vinothkannans

Can you please run the following command in rails console and post the output here Upload.by_users.where("url NOT LIKE '//%' AND url NOT LIKE '#{GlobalSetting.relative_url_root}/uploads/default/original/_X/%'")

[תמונה] vinothkannans: Upload.by_users.where("url NOT LIKE '//%' AND url NOT LIKE '#{GlobalSetting.relative_url_root}/uploads/default/original/_X/%'") Sorry that I was away from this operation for some time. I’m running the rails command sent by you, but no output is coming: [image] I still get the error: " S3 migration failed for db ‘default’" And although post uploads/images are showing ok in my bucket and on site, but site logos etc (under Admin Settings> Branding) are missing and I’m not even able to update them with fresh ones. They just stay blank. I also ran (with no errors): rake posts:rebake_uncooked_posts and rake posts:rebake

@vinothkannans Even after a year, when I happen to repeat the process for some reason, still same error is coming.

Migrating old uploads to S3

תמיכה התקנה

codinghorror (Jeff Atwood) 22 בנובמבר,‏ 2019,‏ 4:59pm 2

Any idea here @vinothkannans?

Rake migrate_to_s3 errors

נושא		תגובות	צפיות
Uploads migration to S3 fails Product Support	26	3681	7 במאי,‏ 2021
Rake migrate_to_s3 errors Self-hosting s3	3	846	4 בפברואר,‏ 2020
Migrating Site to S3 Self-hosting s3	2	821	25 במאי,‏ 2020
Migrate_to_s3 fails Self-hosting s3	25	2477	21 בפברואר,‏ 2022
Migrating uploads to S3 fails Self-hosting s3	1	823	25 באוקטובר,‏ 2024

Migrating old uploads to S3

נושאים קשורים