False positives on "posts are not remapped to new S3 upload URL"

Contribute Bug

AI-generated summary

Users are reporting false positives during S3 upload URL remapping when restoring Discourse backups. barreeeiroo encountered an error where 257 posts were flagged because their “cooked” content contained the string “/uploads/default/original” — not as actual asset links, but as plain text (e.g., in performance reports). This caused the S3 migration to fail, even though the posts didn’t reference real uploads. They resolved it by manually replacing the text. pfaffman suggested skipping cooked posts during migration or doing a database-only restore. WolfpackFan faced a similar issue, initially due to leftover local uploads in /var/discourse/shared/standalone/uploads/default — even though S3 was enabled and “include uploads” was disabled in backup settings. Deleting that directory allowed a successful database-only backup and restore. The issue highlights that the current migration logic may be overly broad by scanning cooked content for upload paths, potentially flagging false positives. Users recommend either excluding cooked posts from the check or ensuring no local upload files remain when using S3. The bug appears to affect S3-enabled instances during backup restores, especially when legacy local uploads exist or when text strings resemble upload paths.

1

I was migrating our Discourse instance from a server to another, and came across an interesting issue…

We use S3 to store uploads from the forum. We have enabled them several years ago, hence it’s not something we introduced in this migration.
After fixing a couple of other issues, I was able to get the backup to be imported. But, it failed on a S3-related step with the following:

Updating the URLs in the database...
Removing old optimized images...
Flagging all posts containing lightboxes for rebake...
72038 posts were flagged for a rebake
EXCEPTION: 257 posts are not remapped to new S3 upload URL. S3 migration failed for db 'default'.

After digging a bit, I was able to trace down the issue to this line:

Then, I went to the rails console, and I was able to replicate the queries with the following:

discourse(prod)> SiteSetting.cdn_path("/uploads/#{@current_db}/original").sub(/https?:/, "")
=> "/uploads//original"
discourse(prod)> RailsMultisite::ConnectionManagement.current_db
=> "default"
discourse(prod)> cdn_path = SiteSetting.cdn_path("/uploads/default/original").sub(/https?:/, "")
=> "/uploads/default/original"
discourse(prod)> Post.where("cooked LIKE '%#{cdn_path}%'")
=> ...

Then, I went to those particular posts, and they were part of the Performance Reports (screenshot is from after I run a find-and-replace script):

The image shows HTTP GET requests targeting various endpoints on a Keroiod.io educational technology community website. (Captioned by AI)
The image shows HTTP GET requests targeting various endpoints on a Keroiod.io educational technology community website. (Captioned by AI)866×685 36.9 KB

Apparently, that check is retrieving any post containing /uploads/default/original in the cooked field, despite it might not be a legitimate asset. In this case, /uploads/default/original was used as “plain text”, hence it was not missed during the migration job.

Not sure if this is expected?
Thank you!

Migrate to s3 failed - but only check fails
2

I’ve seen similar issues with cooked posts, I think.

Maybe you can do a database only restore and then it skips those checks.

That’s what I would try next.

3

I was able to actually restore it fully by just replacing the text in those posts to not match the filter. It was not an issue for me, but just bringing it up in case anyone faces the same issue, as maybe it’s worth to fix it in Discourse.

2 Likes
4

Yeah. Maybe the code just shouldn’t check cooked posts.

1 Like
5

I’m pretty sure I encountered this as well when attempting to restore a backup and I created the backup with “include uploads” as DISABLED. Is there something else I’m missing with the database only restore?

I’m new to discourse so will try to figure out how to work around, but this seems like higher priority if the backup/restore is not working correctly for users that have S3 buckets as upload storage.

Here is part of the log files of the attempted restore that failed.

[2025-06-22 16:02:24] /var/www/discourse/lib/file_store/to_s3_migration.rb:132:in `raise_or_log'
/var/www/discourse/lib/file_store/to_s3_migration.rb:81:in `migration_successful?'
/var/www/discourse/lib/file_store/to_s3_migration.rb:385:in `migrate_to_s3'
/var/www/discourse/lib/file_store/to_s3_migration.rb:59:in `migrate'
/var/www/discourse/lib/file_store/s3_store.rb:352:in `copy_from'
/var/www/discourse/lib/backup_restore/uploads_restorer.rb:69:in `restore_uploads'
/var/www/discourse/lib/backup_restore/uploads_restorer.rb:49:in `restore'
/var/www/discourse/lib/backup_restore/restorer.rb:167:in `restore_uploads'
/var/www/discourse/lib/backup_restore/restorer.rb:71:in `run'
/var/www/discourse/script/spawn_backup_restore.rb:20:in `restore'
/var/www/discourse/script/spawn_backup_restore.rb:33:in `block in <main>'
/var/www/discourse/script/spawn_backup_restore.rb:4:in `fork'
/var/www/discourse/script/spawn_backup_restore.rb:4:in `<main>'
[2025-06-22 16:02:24] Trying to rollback...
6

Update on my specific situation… I still had files in my /var/discourse/shared/standalone/uploads even though I had switched over to S3 for storage. Once I deleted the ‘default’ directory in that uploads dir I re-created a backup it successfully created one that was database only (…sql.gz).

For some reason (in my case) if there are files in that directory it ignores the setting to NOT include uploads in the backup and creates them anyway.

Let me know if any more information or clarification is needed for my situation. Seems like it is maybe slightly different than the OP’s situation.

Either way I was able to get around the problem and can successfully restore now.