Migración de cargas antiguas a S3

Mientras que mis nuevas subidas se están cargando correctamente en Amazon S3, este comando falla al mover la antigua desde local a S3:

DISCOURSE_S3_BUCKET="bucketname" DISCOURSE_S3_REGION="ap-south-1" DISCOURSE_S3_ACCESS_KEY_ID="AKIchangedabitUNM2" DISCOURSE_S3_SECRET_ACCESS_KEY="rX6Pf3C_changedabit_YKKtdVrnbw6B" DISCOURSE_S3_CDN_URL="https://bucketname.s3.ap-south-1.amazonaws.com" rake uploads:migrate_to_s3

Hace 15 días, este comando funcionaba a la perfección.
Hoy está mostrando este error:

Verificando si la migración predeterminada ya se realizó...
7 de 14 subidas no se migraron a S3. La migración a S3 falló para la base de datos 'default'.
1 publicación no se volvió a mapear a la nueva URL de subida de S3. La migración a S3 falló para la base de datos 'default'.
Buscando subidas faltantes en: default

No faltan subidas de publicaciones.

Ninguna publicación requiere rebaking
Migrando subidas a S3 para 'default'...
Algunas subidas no se migraron al nuevo esquema. Ejecute estos comandos en la consola de Rails

SiteSetting.migrate_to_new_scheme = true
Jobs::MigrateUploadScheme.new.execute(nil)

También ejecuté estos 2 comandos en la consola de Rails, pero parecen no haber tenido ningún beneficio:

SiteSetting.migrate_to_new_scheme = true
Jobs::MigrateUploadScheme.new.execute(nil)

Nota: Mis subidas están configuradas para estar en una subcarpeta del bucket bucketname/uploads.

3 Me gusta

Any idea here @vinothkannans?

Is it returning any errors? I’m unable to see any recent big changes in the code of the uploads:migrate_to_s3 rake task.

3 Me gusta

I had a similar problem. It had to do with the path of the uploads not matching what that test expects (like it’s missing ‘default’ in the path of something).

What I did was (in broad and likely unhelpful terms) to see what search is doing to find those uploads that it thinks are bad and fix them

3 Me gusta

I happen to solve it accidentally and in a way that I’m still not sure how it got solved.

I changed the command a couple of times mostly appending or removing the word ‘dualstack’ in the command and in CDN url field in Discourse Settings: https://bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com
But still it didn’t seem to work. Then randomly I gave the two same commands (told in the OP) in the rails console once again.
And 1 step I did randomly around this time was append the word /uploads to the bucketname in the command.

And ‘randomly’ (I need another random word here :slight_smile: ), I succeeded.
So, I can’t say who was the hero of the resolution here.
Thanks for listening anyway.

2 Me gusta

Whatever error was coming/showing, I already included in my original post. But I agree that that error/msg wasn’t very helpful in diagnosis.

Alas, today again, I thought since I couldn’t understand the problem in migration, I created 1 upload locally. Then tried to shift that again to S3 with this command:

DISCOURSE_S3_BUCKET="bxyzbucket1/uploads" DISCOURSE_S3_REGION="ap-south-1" DISCOURSE_S3_ACCESS_KEY_ID="AKIAchangedBRF" DISCOURSE_S3_SECRET_ACCESS_KEY="H0P6Oo8changed1AuLkRDCQK8" DISCOURSE_S3_CDN_URL="https://bxyzbucket1.s3.ap-south-1.amazonaws.com/uploads" rake uploads:migrate_to_s3

And though it shifted the 1 new upload to S3, but with this error included:

1 of 14 uploads are not migrated to S3. S3 migration failed for db 'default'.
1 posts are not remapped to new S3 upload URL. S3 migration failed for db 'default'.

What does this error signify? Can it pose some problem if there are thousands of images to be shifted from ‘Local >> S3’? @vinothkannans

Can you please run the following command in rails console and post the output here

Upload.by_users.where("url NOT LIKE '//%' AND url NOT LIKE '#{GlobalSetting.relative_url_root}/uploads/default/original/_X/%'")
3 Me gusta

Sorry that I was away from this operation for some time.
I’m running the rails command sent by you, but no output is coming:

I still get the error: " S3 migration failed for db ‘default’"
And although post uploads/images are showing ok in my bucket and on site, but site logos etc (under Admin Settings> Branding) are missing and I’m not even able to update them with fresh ones. They just stay blank.

I also ran (with no errors): rake posts:rebake_uncooked_posts and rake posts:rebake

@vinothkannans Incluso después de un año, cuando repito el proceso por alguna razón, sigue apareciendo el mismo error.