将旧上传迁移到 S3

支持 安装
1

虽然我新上传的文件在 Amazon S3 上上传非常顺利，但以下命令却无法将旧文件从本地迁移到 S3：

DISCOURSE_S3_BUCKET="bucketname" DISCOURSE_S3_REGION="ap-south-1" DISCOURSE_S3_ACCESS_KEY_ID="AKIchangedabitUNM2" DISCOURSE_S3_SECRET_ACCESS_KEY="rX6Pf3C_changedabit_YKKtdVrnbw6B" DISCOURSE_S3_CDN_URL="https://bucketname.s3.ap-south-1.amazonaws.com" rake uploads:migrate_to_s3

而在 15 天前，该命令运行得非常完美。
今天它报出了以下错误：

正在检查默认数据库是否已迁移...
14 个上传中有 7 个未迁移到 S3。数据库 'default' 的 S3 迁移失败。
1 个帖子未重新映射到新的 S3 上传 URL。数据库 'default' 的 S3 迁移失败。
正在查找 'default' 上缺失的上传

0 个帖子上传缺失。

没有帖子需要重新烘焙
正在将 'default' 的上传迁移到 S3...
部分上传未迁移到新方案。请在 Rails 控制台中运行以下命令：

SiteSetting.migrate_to_new_scheme = true
Jobs::MigrateUploadScheme.new.execute(nil)

我还在 Rails 控制台中执行了上述两条命令，但它们似乎没有任何效果：

SiteSetting.migrate_to_new_scheme = true
Jobs::MigrateUploadScheme.new.execute(nil)

注意：我的上传配置为存储在桶 bucketname/uploads 的子文件夹中。

3 个赞
2

Any idea here @vinothkannans?

3

Is it returning any errors? I’m unable to see any recent big changes in the code of the uploads:migrate_to_s3 rake task.

3 个赞
4

I had a similar problem. It had to do with the path of the uploads not matching what that test expects (like it’s missing ‘default’ in the path of something).

What I did was (in broad and likely unhelpful terms) to see what search is doing to find those uploads that it thinks are bad and fix them

3 个赞
5

I happen to solve it accidentally and in a way that I’m still not sure how it got solved.

I changed the command a couple of times mostly appending or removing the word ‘dualstack’ in the command and in CDN url field in Discourse Settings: https://bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com
But still it didn’t seem to work. Then randomly I gave the two same commands (told in the OP) in the rails console once again.
And 1 step I did randomly around this time was append the word /uploads to the bucketname in the command.

And ‘randomly’ (I need another random word here :slight_smile: ), I succeeded.
So, I can’t say who was the hero of the resolution here.
Thanks for listening anyway.

2 个赞
6

Whatever error was coming/showing, I already included in my original post. But I agree that that error/msg wasn’t very helpful in diagnosis.

Alas, today again, I thought since I couldn’t understand the problem in migration, I created 1 upload locally. Then tried to shift that again to S3 with this command:

DISCOURSE_S3_BUCKET="bxyzbucket1/uploads" DISCOURSE_S3_REGION="ap-south-1" DISCOURSE_S3_ACCESS_KEY_ID="AKIAchangedBRF" DISCOURSE_S3_SECRET_ACCESS_KEY="H0P6Oo8changed1AuLkRDCQK8" DISCOURSE_S3_CDN_URL="https://bxyzbucket1.s3.ap-south-1.amazonaws.com/uploads" rake uploads:migrate_to_s3

And though it shifted the 1 new upload to S3, but with this error included:

1 of 14 uploads are not migrated to S3. S3 migration failed for db 'default'.
1 posts are not remapped to new S3 upload URL. S3 migration failed for db 'default'.

What does this error signify? Can it pose some problem if there are thousands of images to be shifted from ‘Local >> S3’? @vinothkannans

7

Can you please run the following command in rails console and post the output here

Upload.by_users.where("url NOT LIKE '//%' AND url NOT LIKE '#{GlobalSetting.relative_url_root}/uploads/default/original/_X/%'")
3 个赞
8

Sorry that I was away from this operation for some time.
I’m running the rails command sent by you, but no output is coming:

image
image1221×161 10.3 KB

I still get the error: " S3 migration failed for db ‘default’"
And although post uploads/images are showing ok in my bucket and on site, but site logos etc (under Admin Settings> Branding) are missing and I’m not even able to update them with fresh ones. They just stay blank.

I also ran (with no errors): rake posts:rebake_uncooked_posts and rake posts:rebake

10

@vinothkannans 即使过了一年，当我因某种原因重复该过程时，仍然会出现同样的错误。