False positives on "posts are not remapped to new S3 upload URL"

barreeeiroo · June 15, 2025, 8:51pm

I was migrating our Discourse instance from a server to another, and came across an interesting issue…

We use S3 to store uploads from the forum. We have enabled them several years ago, hence it’s not something we introduced in this migration.
After fixing a couple of other issues, I was able to get the backup to be imported. But, it failed on a S3-related step with the following:

Updating the URLs in the database...
Removing old optimized images...
Flagging all posts containing lightboxes for rebake...
72038 posts were flagged for a rebake
EXCEPTION: 257 posts are not remapped to new S3 upload URL. S3 migration failed for db 'default'.

After digging a bit, I was able to trace down the issue to this line:

github.com/discourse/discourse

lib/file_store/to_s3_migration.rb

main


      
          
          base_url = File.join(SiteSetting.Upload.s3_base_url, prefix)
          count = Upload.by_users.where("url NOT LIKE '#{base_url}%'").count
          if count > 0
            error_message =
              "#{count} of #{Upload.count} uploads are not migrated to S3. #{failure_message}"
            raise_or_log(error_message, should_raise)
            success = false
          end
          
          cdn_path = SiteSetting.cdn_path("/uploads/#{@current_db}/original").sub(/https?:/, "")
          count = Post.where("cooked LIKE '%#{cdn_path}%'").count
          if count > 0
            error_message = "#{count} posts are not remapped to new S3 upload URL. #{failure_message}"
            raise_or_log(error_message, should_raise)
            success = false
          end
          
          unless Rake::Task.task_defined?(MISSING_UPLOADS_RAKE_TASK_NAME)
            Discourse::Application.load_tasks
          end

Then, I went to the rails console, and I was able to replicate the queries with the following:

discourse(prod)> SiteSetting.cdn_path("/uploads/#{@current_db}/original").sub(/https?:/, "")
=> "/uploads//original"
discourse(prod)> RailsMultisite::ConnectionManagement.current_db
=> "default"
discourse(prod)> cdn_path = SiteSetting.cdn_path("/uploads/default/original").sub(/https?:/, "")
=> "/uploads/default/original"
discourse(prod)> Post.where("cooked LIKE '%#{cdn_path}%'")
=> ...

Then, I went to those particular posts, and they were part of the Performance Reports (screenshot is from after I run a find-and-replace script):

Apparently, that check is retrieving any post containing /uploads/default/original in the cooked field, despite it might not be a legitimate asset. In this case, /uploads/default/original was used as “plain text”, hence it was not missed during the migration job.

Not sure if this is expected?
Thank you!

pfaffman · June 15, 2025, 10:13pm

I’ve seen similar issues with cooked posts, I think.

Maybe you can do a database only restore and then it skips those checks.

That’s what I would try next.

barreeeiroo · June 15, 2025, 10:35pm

I was able to actually restore it fully by just replacing the text in those posts to not match the filter. It was not an issue for me, but just bringing it up in case anyone faces the same issue, as maybe it’s worth to fix it in Discourse.

pfaffman · June 16, 2025, 10:12am

Yeah. Maybe the code just shouldn’t check cooked posts.

WolfpackFan · June 22, 2025, 4:41pm

I’m pretty sure I encountered this as well when attempting to restore a backup and I created the backup with “include uploads” as DISABLED. Is there something else I’m missing with the database only restore?

I’m new to discourse so will try to figure out how to work around, but this seems like higher priority if the backup/restore is not working correctly for users that have S3 buckets as upload storage.

Here is part of the log files of the attempted restore that failed.

[2025-06-22 16:02:24] /var/www/discourse/lib/file_store/to_s3_migration.rb:132:in `raise_or_log'
/var/www/discourse/lib/file_store/to_s3_migration.rb:81:in `migration_successful?'
/var/www/discourse/lib/file_store/to_s3_migration.rb:385:in `migrate_to_s3'
/var/www/discourse/lib/file_store/to_s3_migration.rb:59:in `migrate'
/var/www/discourse/lib/file_store/s3_store.rb:352:in `copy_from'
/var/www/discourse/lib/backup_restore/uploads_restorer.rb:69:in `restore_uploads'
/var/www/discourse/lib/backup_restore/uploads_restorer.rb:49:in `restore'
/var/www/discourse/lib/backup_restore/restorer.rb:167:in `restore_uploads'
/var/www/discourse/lib/backup_restore/restorer.rb:71:in `run'
/var/www/discourse/script/spawn_backup_restore.rb:20:in `restore'
/var/www/discourse/script/spawn_backup_restore.rb:33:in `block in <main>'
/var/www/discourse/script/spawn_backup_restore.rb:4:in `fork'
/var/www/discourse/script/spawn_backup_restore.rb:4:in `<main>'
[2025-06-22 16:02:24] Trying to rollback...

WolfpackFan · June 22, 2025, 5:37pm

Update on my specific situation… I still had files in my /var/discourse/shared/standalone/uploads even though I had switched over to S3 for storage. Once I deleted the ‘default’ directory in that uploads dir I re-created a backup it successfully created one that was database only (…sql.gz).

For some reason (in my case) if there are files in that directory it ignores the setting to NOT include uploads in the backup and creates them anyway.

Let me know if any more information or clarification is needed for my situation. Seems like it is maybe slightly different than the OP’s situation.

Either way I was able to get around the problem and can successfully restore now.

Topic		Replies	Views
Migrate_to_S3 Fails on Rebake Installation	17	2346	September 10, 2022
Restore process cancelled at migrating uploads to S3 step Support	25	2501	April 24, 2021
Upload remapping is a bit too enthousiastic Bug	13	1475	November 22, 2019
Unable to migrate to S3, therefore unable to restore from backup Installation	3	1157	April 8, 2022
Bizarre Problems with migrate_from_s3 Dev	4	1091	December 26, 2018

False positives on "posts are not remapped to new S3 upload URL"

Related topics