False positives on "posts are not remapped to new S3 upload URL"

barreeeiroo · June 15, 2025, 8:51pm

I was migrating our Discourse instance from a server to another, and came across an interesting issue…

We use S3 to store uploads from the forum. We have enabled them several years ago, hence it’s not something we introduced in this migration.
After fixing a couple of other issues, I was able to get the backup to be imported. But, it failed on a S3-related step with the following:

Updating the URLs in the database...
Removing old optimized images...
Flagging all posts containing lightboxes for rebake...
72038 posts were flagged for a rebake
EXCEPTION: 257 posts are not remapped to new S3 upload URL. S3 migration failed for db 'default'.

After digging a bit, I was able to trace down the issue to this line:

github.com/discourse/discourse

lib/file_store/to_s3_migration.rb

main


      
          
          base_url = File.join(SiteSetting.Upload.s3_base_url, prefix)
          count = Upload.by_users.where("url NOT LIKE '#{base_url}%'").count
          if count > 0
            error_message =
              "#{count} of #{Upload.count} uploads are not migrated to S3. #{failure_message}"
            raise_or_log(error_message, should_raise)
            success = false
          end
          
          cdn_path = SiteSetting.cdn_path("/uploads/#{@current_db}/original").sub(/https?:/, "")
          count = Post.where("cooked LIKE '%#{cdn_path}%'").count
          if count > 0
            error_message = "#{count} posts are not remapped to new S3 upload URL. #{failure_message}"
            raise_or_log(error_message, should_raise)
            success = false
          end
          
          unless Rake::Task.task_defined?(MISSING_UPLOADS_RAKE_TASK_NAME)
            Discourse::Application.load_tasks
          end

Then, I went to the rails console, and I was able to replicate the queries with the following:

discourse(prod)> SiteSetting.cdn_path("/uploads/#{@current_db}/original").sub(/https?:/, "")
=> "/uploads//original"
discourse(prod)> RailsMultisite::ConnectionManagement.current_db
=> "default"
discourse(prod)> cdn_path = SiteSetting.cdn_path("/uploads/default/original").sub(/https?:/, "")
=> "/uploads/default/original"
discourse(prod)> Post.where("cooked LIKE '%#{cdn_path}%'")
=> ...

Then, I went to those particular posts, and they were part of the Performance Reports (screenshot is from after I run a find-and-replace script):

Apparently, that check is retrieving any post containing /uploads/default/original in the cooked field, despite it might not be a legitimate asset. In this case, /uploads/default/original was used as “plain text”, hence it was not missed during the migration job.

Not sure if this is expected?
Thank you!

pfaffman · June 15, 2025, 10:13pm

I’ve seen similar issues with cooked posts, I think.

Maybe you can do a database only restore and then it skips those checks.

That’s what I would try next.

barreeeiroo · June 15, 2025, 10:35pm

I was able to actually restore it fully by just replacing the text in those posts to not match the filter. It was not an issue for me, but just bringing it up in case anyone faces the same issue, as maybe it’s worth to fix it in Discourse.

Topic		Replies	Views
Unable to migrate to S3, therefore unable to restore from backup Installation	3	1151	April 8, 2022
Migrate_to_s3 fails Installation s3	26	1841	February 22, 2022
S3 migration error with oneboxed posts Support	0	396	October 19, 2020
Rake migrate_to_s3 errors Installation s3	3	698	February 4, 2020
Restore failed at "EXCEPTION: x of y uploads are not migrated to S3. S3 migration failed for db 'default'." Installation	4	1009	June 10, 2022

False positives on "posts are not remapped to new S3 upload URL"

Related topics