Restore process cancelled at migrating uploads to S3 step

[RGJ (Richard - Communiteq)]

If you need any help with drilling down to the root cause of issues like this, please let me know. I’m happy to help.

[ljpp]

So, tried to restore my 2.2GB backup and got this - the restore failed:

EXCEPTION: 44 posts are not remapped to new S3 upload URL. S3 migration failed for db ‘default’.

I am on the 2.6b6, so a fairly recent version. The site is fairly old, from 2016.

From time to time we run into backups were the automatic remapping and migration to S3 fails for various reasons.

This is an old topic, so any recommendation for corrective action? Having a backup I can’t actually use to restore makes me a little nervous.

Edit: I just realized I may be in the wrong thread, as this seems like the same issue

Restore failed at "EXCEPTION: x of y uploads are not migrated to S3. S3 migration failed for db 'default'."

This has been sitting as a draft on meta.discourse.org as I worked through the issue on my Discourse server. I’ve been able to get around the problem described below, but that I couldn’t do what was effectively a disaster restore (a restore without the benefit of being able to get into the old, broken server) is concerning. (The root cause was a failed Postgres 10-to-12 upgrade, but I was able to postpone the update in the Discourse app.yml file long enough to disable the enable_s3_uploads sit…

[ljpp]

@gerhard

Gave this another shot and I am still failing on the restore to the 44 posts are not remapped -error.

Temporary disabling of S3 uploads prior to the backup (proposed by @RGJ does not solve the issue, and currently I don’t have a working backup scheme, which is pretty damn serious.

Restore Failed: "Some uploads could not be migrated to the new scheme"

Have you tried turning off s3 uploads prior to creating the backup? Just turning off S3 and not rebaking should not do any harm. Then restore and switch S3 back on.

Any suggestions on what to try?

[ljpp]

@ljpp

Any suggestions on what to try?

Okay, this is the weirdest case I’ve seen so far. I stumpled upon this post.

https://meta.discourse.org/t/using-object-storage-for-uploads-s3-clones/148916/160?u=ljpp

After wasting hours on this already, I defined the DISCOURSE_CDN_URL just for kicks, a feature my site does not use. The backup was restored successfully.

What is the significance of this parameter, in this scenario? @Falco @pfaffman

EDIT:

I take that back. After success with the test, I took a fresh backup from production and once again started to migrate to a new server. It failed again, but the error message changed.

EXCEPTION: rake posts:missing_uploads identified 3 issues. S3 migration failed for db ‘default’.

EDIT2:

So I kept drilling the database and found these 3 posts.

• 2 were very old images, that were first published but then moderator replaced the images with a modified version. These were from 2016 and 2018.
• But the last 1 was odd. It was a very recent post just a couple of weeks ago by myself and it contained a oneboxed https:// link to a development server, that no longer exists. So a broken link breaks the restore process.

I simply deleted these posts manually.

Now, again, a test run shows that a backup might actually succeed. We’ll see.

[BryanV (Bryan Van de ven)]

@ljpp I seem to be in a similar situation with a restore failing with

EXCEPTION: 1 posts are not remapped to new S3 upload URL. S3 migration failed for db ‘default’.

Can you elaborate on how you found and deleted the offending posts? What commands are used?

[pfaffman (Jay Pfaffman)]

He deleted the posts in question.

But you wiped your install so you can’t do that, right?

[Saiano]

@BryanV

@ljpp 我似乎遇到了类似的情况，恢复失败并出现以下错误：

EXCEPTION: 1 个帖子未重新映射到新的 S3 上传 URL。数据库 ‘default’ 的 S3 迁移失败。

您能详细说明一下您是如何找到并删除有问题的帖子的吗？使用了哪些命令？

我恢复数据库时遇到了这个问题。您是如何找到有问题的帖子的？

[pfaffman (Jay Pfaffman)]

@ljpp

未映射到新的 S3

我认为执行检查的代码在这里：

lib/file_store/to_s3_migration.rb

所以我想是这样：

prefix = @migrate_to_multisite ? "uploads/#{@current_db}/original/" : "original/"
base_url = File.join(SiteSetting.Upload.s3_base_url, prefix)
bad = Upload.by_users.where("url NOT LIKE '#{base_url}%'")

为了确保万无一失

good = Upload.by_users.where("url LIKE '#{base_url}%'")

请告诉我这是否对您有效，然后我将着手创建一个主题。

[Saiano]

discourse(prod)> prefix = @migrate_to_multisite ? “uploads/#{RailsMultisite::ConnectionManagement.current_db}/original/” : “original/”
discourse(prod)> base_url = File.join(SiteSetting.Upload.s3_base_url, prefix)
discourse(prod)> bad_uploads = Upload.by_users.where(“url NOT LIKE ‘#{base_url}%’”)
discourse(prod)> bad_uploads.count
=> 0

此检查未发现任何错误

[pfaffman (Jay Pfaffman)]

您是在原始服务器上执行的吗？还是需要在数据库恢复后，在执行该检查之前使用 --pause 开关来执行？

[Saiano]

我在旧服务器上执行了此操作。
恢复时我遇到了相同的错误

[2026-01-16 13:45:52] 异常: 3 个帖子未重新映射到新的 S3 上传 URL。数据库 ‘default’ 的 S3 迁移失败。
[2026-01-16 13:45:52] /var/www/discourse/lib/file_store/to_s3_migration.rb:132:in `raise_or_log’

[pfaffman (Jay Pfaffman)]

@Saiano

此检查给我 0 个错误

好吧。真糟糕。不确定还能告诉你什么。

good 找到了你的上传文件吗？

我可能会只使用 --pause 选项，并在恢复执行该检查之前停止它。

我会照理查德说的做。

[RGJ (Richard - Communiteq)]

我不确定您的具体情况，但您可能只想关闭 s3 上传，进行备份，恢复，然后再重新开启。这帮了我很多次。

[Saiano]

所以，我必须关闭这些选项，对吗？

[RGJ (Richard - Communiteq)]

正确。

• 取消勾选“启用 S3 上传”
• 备份/下载
• 上传/恢复
• 勾选“启用 S3 上传”

[Saiano]

@RGJ

正确。

• 取消选中“启用 S3 上传”
• 备份/下载
• 上传/恢复
• 选中“启用 S3 上传”

在选中“启用 S3 上传”时我遇到了一个错误

[RGJ (Richard - Communiteq)]

我不会说意大利语，除非是关于食物的……也许你可以帮大家一个忙，把它复制/粘贴到谷歌翻译里

[Saiano]

@RGJ

我不会说意大利语，除非是关于食物的……也许你可以帮我们所有人一个忙，把它复制/粘贴到谷歌翻译中

抱歉

启用 S3 上传

将上传内容存储在 Amazon S3 磁盘空间上。重要提示：需要提供有效的 S3 凭据（包括访问密钥 ID 和访问密钥密钥）。

您无法启用 S3 上传，因为它们已在全局启用，在站点级别启用 S3 上传可能会导致上传出现严重问题。

[RGJ (Richard - Communiteq)]

我假设你已经在 app.yml 中完成了。
（新的）上传功能工作正常吗？如果正常，那就没有问题了。

[Saiano]

@RGJ

我假设你已经在 app.yml 中完成了。
(新的) 上传工作正常吗？如果是，那就没有问题。

在 app.yml 中我配置了 s3，没错。

那么，我需要怎么做才能安全地备份和恢复而无需检查上传？

简单地取消选中“启用 s3 上传”不起作用