设置文件和图片上传到S3

[Denis_Didkovsky (DenisD)]

上面的帖子是否仍然是允许管理员进行设置的有效指南？如果不是，是否有其他地方可以找到 2024 年的正确 S3 指南？据称这是 Discourse 团队官方支持的。

我们按照上面的指南进行了实现，但图片在 Discourse 实例上显示为损坏，尽管它们已上传到 S3。

我们的流程

1. 按照上面的指南设置 S3 存储桶本身以及 S3 中的策略（所有步骤都已准确遵循）

2. 按照上面的指南在 app.yml 和管理员中设置环境变量（相同）

3. 重新构建应用程序

4. 完全没有使用 CDN，因为我们认为 CDN 是可选的，并且在没有 CDN 的情况下也应该可以工作。

结果：

1. 图片已上传到 Discourse 的存储桶
2. 帖子中的链接已由 Discourse 正确替换为类似以下的亚马逊链接：
   xxx-bucket.s3.dualstack.us-east-1.amazonaws.com/original/1X/a1b21eb5de071799d4b5e5215619d11d28602dfe.jpeg
3. 链接无法访问（假设是因为官方指南中的 S3 策略）

对官方指南中 S3 策略的假设更改。Principal：此策略未指定 Principal，这意味着它适用于任何拥有正确权限的已认证用户。如果要允许匿名（公开）访问，我们需要指定\"Principal\": \"*\"。这也可能不是一个好的解决方案，因为它会使存储桶公开。
这是假设，因为我们希望有人能在 2024 年更新指南以使其正确。

[aosus]

Hey,
.we are using Cloudflare R2 for s3 storage(it works now) and i wanted to move discourse to another server.
However profile pictures were missing when restoring from a backup even when checking the “backup uploads” option.

so i ran the s3 migrate command on the old server to migrate images that are still local, and it’s failing with this trace, it’s not clear what’s the reason.

rake uploads:migrate_to_s3 --trace
** Invoke uploads:migrate_to_s3 (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute uploads:migrate_to_s3
Please note that migrating to S3 is currently not reversible! 
[CTRL+c] to cancel, [ENTER] to continue

Migrating uploads to S3 for 'default'...
Uploading files to S3...
 - Listing local files
..... => 5123 files
 - Listing S3 files
....... => 6871 files
 - Syncing files to S3
.......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................^[[B............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
Updating the URLs in the database...
Removing old optimized images...
Flagging all posts containing lightboxes for rebake...
828 posts were flagged for a rebake
rake aborted!
FileStore::ToS3MigrationError: 4898 of 5838 uploads are not migrated to S3. S3 migration failed for db 'default'. (FileStore::ToS3MigrationError)
/var/www/discourse/lib/file_store/to_s3_migration.rb:132:in `raise_or_log'
/var/www/discourse/lib/file_store/to_s3_migration.rb:73:in `migration_successful?'
/var/www/discourse/lib/file_store/to_s3_migration.rb:383:in `migrate_to_s3'
/var/www/discourse/lib/file_store/to_s3_migration.rb:59:in `migrate'
/var/www/discourse/lib/tasks/uploads.rake:126:in `migrate_to_s3'
/var/www/discourse/lib/tasks/uploads.rake:106:in `block in migrate_to_s3_all_sites'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-6.1.0/lib/rails_multisite/connection_management/null_instance.rb:49:in `with_connection'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-6.1.0/lib/rails_multisite/connection_management/null_instance.rb:36:in `each_connection'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-6.1.0/lib/rails_multisite/connection_management.rb:21:in `each_connection'
/var/www/discourse/lib/tasks/uploads.rake:104:in `migrate_to_s3_all_sites'
/var/www/discourse/lib/tasks/uploads.rake:100:in `block in <main>'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/task.rb:281:in `block in execute'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/task.rb:281:in `each'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/task.rb:281:in `execute'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/task.rb:199:in `synchronize'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/task.rb:199:in `invoke_with_call_chain'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/task.rb:188:in `invoke'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/application.rb:188:in `invoke_task'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/application.rb:138:in `block (2 levels) in top_level'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/application.rb:138:in `each'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/application.rb:138:in `block in top_level'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/application.rb:147:in `run_with_threads'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/application.rb:132:in `top_level'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/application.rb:83:in `block in run'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/application.rb:214:in `standard_exception_handling'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/application.rb:80:in `run'
bin/rake:13:in `<top (required)>'
/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.5.18/lib/bundler/cli/exec.rb:58:in `load'
/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.5.18/lib/bundler/cli/exec.rb:58:in `kernel_load'
/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.5.18/lib/bundler/cli/exec.rb:23:in `run'
/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.5.18/lib/bundler/cli.rb:455:in `exec'
/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.5.18/lib/bundler/vendor/thor/lib/thor/command.rb:28:in `run'
/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.5.18/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.5.18/lib/bundler/vendor/thor/lib/thor.rb:527:in `dispatch'
/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.5.18/lib/bundler/cli.rb:35:in `dispatch'
/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.5.18/lib/bundler/vendor/thor/lib/thor/base.rb:584:in `start'
/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.5.18/lib/bundler/cli.rb:29:in `start'
/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.5.18/exe/bundle:28:in `block in <top (required)>'
/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.5.18/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.5.18/exe/bundle:20:in `<top (required)>'
/usr/local/bin/bundle:25:in `load'
/usr/local/bin/bundle:25:in `<main>'

it seems these pictures are still stored locally for some reason, even though i ran this migration script before.

[pfaffman (Jay Pfaffman)]

我猜你更改了 s3 设置，并且丢失的图片位于与新图片不同的存储桶或路径中。

你可以查看

Upload.pluck(:url)

来找出问题所在，也许吧。

[Walter_Jenkins (Aviddabbler)]

我最近遇到了同样的问题，并通过更新具有此角色的 IAM 权限解决了该问题

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
               "s3:List*",
               "s3:Get*",
               "s3:AbortMultipartUpload",
               "s3:DeleteObject",
               "s3:PutObject",
               "s3:PutObjectAcl",
               "s3:PutObjectVersionAcl",
               "s3:PutLifecycleConfiguration",
               "s3:CreateBucket",
               "s3:PutBucketCORS"
      ],
      "Resource": [
        "arn:aws:s3:::my-bucket",
        "arn:aws:s3:::my-bucket/*"
      ]
    },
    {
       "Effect": "Allow",
       "Action": [
           "s3:ListAllMyBuckets",
           "s3:ListBucket"
       ],
       "Resource": "*"
    }
  ]
}

我还必须启用 ACL 对象所有权

这是我搜索 S3 时的完整配置

[ChallaPradyumna]

在存储桶上设置 CORS 以及上述多部分上传的设置。

[
    {
        "AllowedHeaders": [
            "content-type",
            "x-amz-acl",
            "x-amz-meta-sha1-checksum"
        ],
        "AllowedMethods": [
            "GET",
            "HEAD",
            "PUT"
        ],
        "AllowedOrigins": [
            "*"
        ],
        "ExposeHeaders": [],
        "MaxAgeSeconds": 3000
    }
]

[Rhod (Rodolfo)]

是否可以更换 S3 服务提供商？

我想知道，如果我尝试更换 S3 服务提供商，除了已上传图片的问题之外，是否会有其他问题。我对此并没有什么大问题，因为网站尚未上线；我仍在开发中。

是否可以将图片先保存到本地，然后再使用新的服务提供商激活 S3？

[pfaffman (Jay Pfaffman)]

您可以在 rails 中设置 siteSettings.include_s3_uploads_in_backups=true，然后进行备份，更改 S3 设置，并恢复备份。

[Rhod (Rodolfo)]

好的，我明白了。非常感谢！

[ericomeehan (Eric Meehan)]

@Discourse

单向门

与 Discourse 中的许多配置决策不同，请注意使用 S3 是一个“单向门”；也就是说，这是一个不易撤销的操作。没有安全或受维护的方式可以将文件移出 S3 以便本地上传。特别是，migrate_to_s3 会将比帖子图片更多的文件移至 S3；这些文件没有反向路径。（有关更多详细信息，请参阅 Migrate_from_s3 problems）。

链接的帖子提到了一个用于删除 migrate_from_s3 任务的 PR，而且我没有理由相信它后来又被添加回来，但我想确认一下情况是否仍然如此。我正在考虑迁移到 MinIO，并且有点担心将其作为依赖项。

@pfaffman

您可以在 rails 中设置 siteSettings.include_s3_uploads_in_backups=true，然后进行备份，更改 S3 设置，并恢复备份。

我想知道这是否能为我上面提到的情况提供一个变通方法。包含该设置，进行备份，禁用 S3 设置，然后恢复备份。

[pfaffman (Jay Pfaffman)]

是的。我会将其恢复到一个新服务器，这样如果出现问题，您仍然有一个可用的服务器。

[ericomeehan (Eric Meehan)]

Enable hidden setting to include S3 uploads in the backups

Ok. I didn’t know. But, what’s the diff between ‘Local Uploads’ and “Files Stored in S3 ‘Uploads’” folder? Also, though I haven’t changed the setting in web_only or in rails console as told above YET, but only chose the one setting told here. But still when I ran the backup, just 10 min ago, it showed as if it downloaded thousands of files, (which I presume could only be on my aws s3 storage bucket, and ‘Download’ word itself means it is downloading from remote S3 bucket folder). Also it is …

从这个帖子来看，这个设置似乎会从 S3 下载单个文件，但我担心帖子引用 S3 URL 而不是本地文件可能会出现问题。我不确定数据库是如何处理这种情况的。我想知道你迁移到另一个提供商是否成功了，@Rhod。

[Rhod (Rodolfo)]

我顺利地完成了迁移。我强调，我没有遇到任何重大问题，因为图片或备份的数量很少。该论坛完全是用于测试目的，尚未进入“生产”阶段。最终，经过几次反复试验，我在新服务器（Hetzner）上重新安装了论坛，并为备份和图片配置了 S3（Cloudflare R2），这台服务器最终让我满意。

很抱歉我帮不上太多忙。我看看将来是否可以在另一台服务器上进行测试。

[RoldanLT]

@Rhod

已配置 Cloudflare S3 R2

使用 CF R2，您的聊天缩略图是否正常工作？

[Rhod (Rodolfo)]

我没有注意到那个错误；事实上，我在我的网站上禁用了聊天功能。

我想我不会使用它，但我仍然遇到你提到的那个错误，在你 帖子中。

不过，我会关注你的 帖子 ，看看他们是否能帮你找到解决方案。

[nickrsan (Nick S)]

谢谢你，沃尔特！我来这里发帖，却发现你已经发过了。

@Discourse 能否将沃尔特帖子的内容整合到初始帖子中？

[mingyu_zhao]

我可以上传视频到S3上面吗？目前我上传视频 只有视频的封面图会传到S3上面 视频则传不上去