Migrando uploads antigos para o S3

Enquanto meus novos uploads estão sendo carregados corretamente no Amazon S3, este comando está falhando ao mover os antigos do local para o S3:

DISCOURSE_S3_BUCKET="bucketname" DISCOURSE_S3_REGION="ap-south-1" DISCOURSE_S3_ACCESS_KEY_ID="AKIchangedabitUNM2" DISCOURSE_S3_SECRET_ACCESS_KEY="rX6Pf3C_changedabit_YKKtdVrnbw6B" DISCOURSE_S3_CDN_URL="https://bucketname.s3.ap-south-1.amazonaws.com" rake uploads:migrate_to_s3

Há 15 dias, este comando funcionava perfeitamente.
Hoje, está retornando este erro:

Verificando se o padrão já foi migrado...
7 de 14 uploads não foram migrados para o S3. A migração para o S3 falhou para o banco de dados 'default'.
1 postagem não foi remapeada para a nova URL de upload do S3. A migração para o S3 falhou para o banco de dados 'default'.
Procurando uploads ausentes em: default

0 uploads de postagens estão ausentes.

Nenhuma postagem requer rebaking
Migrando uploads para o S3 para 'default'...
Alguns uploads não foram migrados para o novo esquema. Por favor, execute estes comandos no console do rails

SiteSetting.migrate_to_new_scheme = true
Jobs::MigrateUploadScheme.new.execute(nil)

Também executei esses 2 comandos no console do rails, mas eles parecem não ter trazido nenhum benefício:

SiteSetting.migrate_to_new_scheme = true
Jobs::MigrateUploadScheme.new.execute(nil)

Nota: Meus uploads estão configurados para estar em uma subpasta no bucket bucketname/uploads.

3 curtidas

Any idea here @vinothkannans?

Is it returning any errors? I’m unable to see any recent big changes in the code of the uploads:migrate_to_s3 rake task.

3 curtidas

I had a similar problem. It had to do with the path of the uploads not matching what that test expects (like it’s missing ‘default’ in the path of something).

What I did was (in broad and likely unhelpful terms) to see what search is doing to find those uploads that it thinks are bad and fix them

3 curtidas

I happen to solve it accidentally and in a way that I’m still not sure how it got solved.

I changed the command a couple of times mostly appending or removing the word ‘dualstack’ in the command and in CDN url field in Discourse Settings: https://bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com
But still it didn’t seem to work. Then randomly I gave the two same commands (told in the OP) in the rails console once again.
And 1 step I did randomly around this time was append the word /uploads to the bucketname in the command.

And ‘randomly’ (I need another random word here :slight_smile: ), I succeeded.
So, I can’t say who was the hero of the resolution here.
Thanks for listening anyway.

2 curtidas

Whatever error was coming/showing, I already included in my original post. But I agree that that error/msg wasn’t very helpful in diagnosis.

Alas, today again, I thought since I couldn’t understand the problem in migration, I created 1 upload locally. Then tried to shift that again to S3 with this command:

DISCOURSE_S3_BUCKET="bxyzbucket1/uploads" DISCOURSE_S3_REGION="ap-south-1" DISCOURSE_S3_ACCESS_KEY_ID="AKIAchangedBRF" DISCOURSE_S3_SECRET_ACCESS_KEY="H0P6Oo8changed1AuLkRDCQK8" DISCOURSE_S3_CDN_URL="https://bxyzbucket1.s3.ap-south-1.amazonaws.com/uploads" rake uploads:migrate_to_s3

And though it shifted the 1 new upload to S3, but with this error included:

1 of 14 uploads are not migrated to S3. S3 migration failed for db 'default'.
1 posts are not remapped to new S3 upload URL. S3 migration failed for db 'default'.

What does this error signify? Can it pose some problem if there are thousands of images to be shifted from ‘Local >> S3’? @vinothkannans

Can you please run the following command in rails console and post the output here

Upload.by_users.where("url NOT LIKE '//%' AND url NOT LIKE '#{GlobalSetting.relative_url_root}/uploads/default/original/_X/%'")
3 curtidas

Sorry that I was away from this operation for some time.
I’m running the rails command sent by you, but no output is coming:

I still get the error: " S3 migration failed for db ‘default’"
And although post uploads/images are showing ok in my bucket and on site, but site logos etc (under Admin Settings> Branding) are missing and I’m not even able to update them with fresh ones. They just stay blank.

I also ran (with no errors): rake posts:rebake_uncooked_posts and rake posts:rebake

@vinothkannans Mesmo depois de um ano, quando acabo de repetir o processo por algum motivo, o mesmo erro continua aparecendo.