I want to stop using Amazon s3 for uploads.
I’ve not done a rake task like this so want to check things before I run on the live environment.
My plan is to use a clone of my live server to test the rake task uploads:migrate_from_s3.
Will this move the files from s3?
Or copy them to the local storage and leave the s3 files there until I manually delete them?
If it deletes them from s3 that will break the live forum and that would be bad!
thanks
I believe @vinothkannans can point you in the right direction here.
Yes, it will remove the file from S3.
You should clone your S3 bucket too. And change the bucket name to newly cloned one in the site settings before testing the migration process.
Also I will recommend you to take a backup of S3 bucket before running the migration in live environment.
当我进入应用并运行 rake -AT 时，没有出现 uploads:migrate_from_s3 这个 rake 任务。
是否有新方法可以将上传文件从 S3 迁移回本地？
运行 rake task uploads:migrate_from_s3 时你会得到什么输出？
rake 执行失败！
不知道如何构建任务 ‘uploads:migrate_from_s3’（使用 rake --tasks 查看可用任务列表）
您的意思是？uploads:migrate_to_s3
该任务已在社区提交的拉取请求中被移除。您可以在以下链接了解具体原因：
谢谢。我似乎找不到关于如何手动完成此操作的说明或指导。
您可以使用 awscli 手动将所有文件从 S3 迁移到您的服务器：
cd /var/discourse
./launcher enter app
sudo apt install awscli
aws configure
使用您的 S3 ID 和密码配置 aws，然后执行：
cd public/uploads/default/
aws s3 sync s3://mybucketname/
该命令将把所有文件从 S3 下载到 public/uploads/default/。接着执行：
rake posts:rebake
exit
./launcher rebuild app
我最初尝试将数据从 S3 迁移出去，但在我用自己的迁移方案作为代码测试环境完成后，被告知我完全选错了方法。这一说法得到了佐证：我从 S3 迁移到本地文件的大型实例中出现了数据损坏。最近，我还不得不手动修复数据库中损坏的数据，因此我并不指望自己已经完成了这次尝试后的所有清理工作。
如果今天我要将数据从 S3 迁移到自己的存储，我会先在 Discourse 旁边启动一个 MinIO 容器，使用 minio-client 将数据从 S3 复制到 MinIO 容器，然后重新配置 Discourse 以指向新的“S3”位置（即本地 MinIO），最后执行 posts:remap 或重新烘焙所有帖子，使其从我的 MinIO 容器提供“S3”服务。
以下是 posts:remap 在此场景下应如何工作的示例。我隐约记得这里有更好的示例，但这个应该足以帮助入门。
所以我按照这些说明操作，成功将上传的文件复制过来了，但即使已关闭 S3 上传功能，实例中仍然处处存在指向 S3 服务器的链接。我尝试了 discourse remap，但没有效果；尝试了 rake posts:remap，也没有效果；posts:rebake 同样无效。
在 S3 上设置上传确实很简单，但想恢复原状却没那么容易。
从 S3 迁移的问题已存在很长时间，至今仍未修复。请查看 mcdanlj 的上述帖子。
运行以下命令后，你得到的输出是什么？
./launcher enter app
rails c
Upload.all.sample(20).pluck(:url)
由于目前网站规模尚小，我已逐一重新链接了上传文件。rake posts:missing_uploads 命令有助于确认是否已全部补全。
你确定所有 URL 都已正确重映射吗？不仅限于图片，头像、图标、备份等 URL 也都必须重映射。请尝试执行上述命令，查看输出中是否包含 S3 URL。
所有 URL 已恢复为相对路径。遗憾的是，对于未来阅读此帖的人，我不确定是哪些 rake 任务与应用重建的组合促成了我的成功。希望有比我对 Discourse 更精通的人能整理出一份万无一失的教程，说明应如何完成此操作。
如上文 Pravi 所述，使用 awscli 从 S3 复制文件是有效的。
我也非常困惑。
之前我在不同时间段有两个 S3 存储桶。运行 rake posts:missing_uploads 时，显示大约有 500 篇帖子缺少上传内容或受到了影响。
但后来我决定将这两个存储桶的资源都复制到本地的 Ubuntu 服务器（运行在 GCP 实例上）。现在，显示缺少上传内容的帖子数量增加到了 1000 篇左右。
如果我执行 Upload.all.sample(1000).pluck(:url)，会得到不同的上传路径：有些在本地，有些在 bucket1，极少数在 bucket2。
现在我该怎么办？（目前我已将上传配置在本地服务器上）。
我的 网站 规模很小，两年内只有大约 1000 篇帖子。
你需要重新映射 S3 URL。你能贴出 Upload.all.sample(1000).pluck(:url) 的输出吗？
谢谢，ji。
如果这成功的几率较大，我可以试试。
编辑：我试了，但结果是我帖子中的图片完全消失了。将鼠标悬停在图片占位符上时，根本没有显示任何 URL 链接。之前不仅显示一个小图片图标，还会显示图片的 URL。
我这儿大概有 45 行输出。不知道你是否需要我把全部 1000 行输出都贴在这里？
=> ["//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/1X/7277b38e1d614b3d700f6266fa5b841a15e7c6ba.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/1X/0c1f27b110ff476c636265b46d63f83c5bf575e9.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/a/a18dd8e63379c70e2a4014b0f6d85cf89ab12ed7.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/5/51798c75050206a784d9ae186cffb5c4558b1625.jpeg",
"/uploads/default/original/2X/1/159ca83317895ddaefec42ee75c8748201f34d1f.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/1/1593a5f91ac7642180c9651544806d89342a4e24.jpeg",
"/uploads/default/original/2X/f/fa7c701e58c198984707161364c81e9c59b27572.jpeg",
"/uploads/default/original/2X/b/b6b12c4df19b234f6d1551c26d92a14969f19f12.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/f/fcd4201295e6e63a7496e70e8b067225774834f3.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/f/f42150c03cce15a1bdc34c6af746fe51b63db36c.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/1X/e66cfb3a0bdc558a02fd999d1bf5a185521456cf.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/1X/1885bc0fe4cf6ee25ab8d6e250807c9836168cdc.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/c/c1f240c94d45e0be0cfcd7a3cc714e389bb49f23.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/2/293518b6119a17d64222cf5d691aee271fa572cf.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/1X/617f0bb525f05934ae455582deacb96e7fd097e9.jpeg",
"//bhdisco.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/3/331a610d2237294bdb126eecf1f087173c5087df.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/d/d0b3efa53ceab849e17e781e63f2b969f94dbde0.jpeg",
"/images/default-favicon.ico",
"/uploads/default/original/2X/b/b399c60eb0b83232531ef2ead5a060fbd9c2a64e.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/9/987178dc86175225b999151e112deb3bf26b13af.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/c/c99c8a9dd3e59cecc6fb5010f362b1dc3f49ca66.jpeg",
"//bhdisco.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/5/5c145297f26cd91f6f157d507324724ba15cb618.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/e/ef5c223cd604e3574c300671da61693dc5399e62.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/1X/9c99b85e976c42bc3c692f75b8828abbbc2bc4ea.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/6/6f88f27ee155e69bb5b701b86eba9dc3e369bc09.png",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/3/3b454434e58b05db14f51c3fcdfbd36d0d96d4eb.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/1X/b7e92457674b8c28f6ecfbc9340816fc3583d20e.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/7/731c0ee200660fb1888cd058c79282bf7f6dab95.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/1X/a42f92c28be8bd5ac86bb55f36fc9ca0bec37c72.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/1X/ac7dc4c531d428b152de3eebceb44bfe779c349d.jpeg",
"//bhdisco.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/f/f361a9ad5194ef13c2c6b24e2db98455e7cc48d9.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/1X/43ec21eafc00f8cd05976e0fbec41ef686e0e3b3.png",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/1X/8b41f21b02df80a9e5c4ea14a5dffa5776811f3d.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/6/6eee5c5a5a2b274e7d8ccc911cd4f594cc4930de.png",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/1X/c1736a8c2923f0bd341f7112db74cd62392f7e14.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/c/c3e2fbb84c55dbf2f1fdccf835964eb4319e2371.png",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/0/0b41e87ab7728dbead217e237c4ad8940bdfe776.png",
"//bhdisco.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/a/a3ac121566d741d5b396ba49625f3206ff0c498f.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/b/bbbb6973cb0900389675691c5c129c907cf4611f.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/2/28f5d5bf46b30a33bde8e67c184cb099cc71511a.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/f/ff34c3fc207781d5881d4583cceb26aa54619c0e.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/f/f85db75b4721b8d24abf78ee3d25f2ffbef26da9.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/1X/35c6402fbf4a8fca05f30924800fa0c90cf891e7.png",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/9/9d8f29892278f164e8ce27a6b58cc8af0760802c.png",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/1/1a533eb9f4ec5186427a541a7ebbf06817627f97.png",
我看到您的网站上有一些头像损坏。我推测这可能是由于您删除了 S3 存储桶，或者在从 S3 复制所有文件后没有重新生成。为什么输出中会有两个存储桶？
bhdisco.s3.dualstack.ap-south-1.amazonaws.com
bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com
您是否曾将网站配置为使用一个存储桶，之后又更改为另一个？
抱歉，我不知怎么错过了你回复的通知。非常感谢。
我并没有删除 S3 存储桶，但我确实之前切换过存储桶（当时我并未意识到其中的复杂性）。因此，我所有的资源都分布在两个 S3 存储桶中，不过现在我已经手动将它们复制到本地服务器上了（/var/discourse/shared/web_only/uploads/default）。
我该如何着手清理这个混乱的局面？
注意：我熟悉容器的基本命令、Rails 控制台以及 AWS S3。