Migrate_to_s3 fails

itsbhanusharma · November 18, 2021, 6:54am

I’m trying to move uploads to s3 on a relatively busy site.
Everything goes as planned except when I run rake uploads:migrate_to_s3, I am presented with the following error:

Updating the URLs in the database...
Removing old optimized images...
Flagging all posts containing lightboxes for rebake...
2023 posts were flagged for a rebake
rake aborted!
FileStore::ToS3MigrationError: 1 of 9629 uploads are not migrated to S3. S3 migration failed for db 'default'.
/var/www/discourse/lib/file_store/to_s3_migration.rb:132:in `raise_or_log'
/var/www/discourse/lib/file_store/to_s3_migration.rb:79:in `migration_successful?'
/var/www/discourse/lib/file_store/to_s3_migration.rb:373:in `migrate_to_s3'
/var/www/discourse/lib/file_store/to_s3_migration.rb:66:in `migrate'
/var/www/discourse/lib/tasks/uploads.rake:123:in `migrate_to_s3'
/var/www/discourse/lib/tasks/uploads.rake:102:in `block in migrate_to_s3_all_sites'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/rails_multisite-4.0.0/lib/rails_multisite/connection_management.rb:80:in `with_connection'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/rails_multisite-4.0.0/lib/rails_multisite/connection_management.rb:90:in `each_connection'
/var/www/discourse/lib/tasks/uploads.rake:100:in `migrate_to_s3_all_sites'
/var/www/discourse/lib/tasks/uploads.rake:96:in `block in <main>'
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'
Tasks: TOP => uploads:migrate_to_s3
(See full trace by running task with --trace)

This line in particular caught my attention FileStore::ToS3MigrationError: 1 of 9629 uploads are not migrated to S3. S3 migration failed for db 'default'.

I’m assuming it means there is one problematic file that is causing the entire take task to fail? is there a way to figure out which file is causing the problem and maybe just remove it so the rest of the task completes peacefully?

itsbhanusharma · November 18, 2021, 12:25pm

Some Update:
Site seems to be loading normally and from what I can see, uploaded images are being served from the CDN.
The local copy of all uploads still exists on the server. I assume this is because the migration is marked as failed.
here is an output of rake uploads:s3_migration_status if it is helpful in any way?

# rake uploads:s3_migration_status
1 of 9630 uploads are not migrated to S3. S3 migration failed for db 'default'.
19 posts are not remapped to new S3 upload URL. S3 migration failed for db 'default'.
No posts require rebaking
Site is not ready for migration

Simon_Manning · November 18, 2021, 12:50pm

You might find a reference to which file failed via Admin > Logs > Error Logs

If that doesn’t help, I think searching the database for posts with cooked that contain yourdomain/uploads/ will get you a list of those 19 posts mentioned in the migration status. Worst case, you could then manually look at those and compare any uploads with what’s in S3.

Something like:

rails c
Post.where("cooked like '%discourse.example.com/uploads/%'")

itsbhanusharma · November 18, 2021, 1:41pm

Hi @Simon_Manning

I tried your suggestions, Sadly, no luck finding anything relevant in error logs. I do see an entry like
Failed to optimize image: unknown reason but it doesn’t give me any more info than what it says.

I tried your suggested query as well, I do get a bunch of posts as a result but actually visiting those posts, they contain a bunch of thumbnails from from onebox previews. when I try to open image in new tab on those thumbnails, those appear to be loading from the CDN.

Simon_Manning · November 18, 2021, 2:15pm

I think I would expect to see something like that for 18 of those 19 posts, maybe even all 19 if the one failed upload happens to be in a post with multiple uploads.

What might be easier is to pull out all the filenames from the query result (e.g. f8a2d9381889b8693db2777acac566bd7b134fa5.png) and search for them in S3. In theory, exactly one should be missing. Ignore the derivatives, just look for the ones with /originals/ in the path.

A bit tedious but shouldn’t take too long if that query did only select those 19 posts.

itsbhanusharma · November 18, 2021, 4:47pm

So I found the culprit, How do I proceed?

What seems logical to me might not be the best way out
Here are the options that I see:

Remove reference of file from the offending post
Delete file from the server
remap the file link to something generic

or maybe I’m thinking totally wrong?

Simon_Manning · November 18, 2021, 9:24pm

If there’s not an obvious solution to get it to upload successfully, either of options 1 or 3 seem like good ways to go. 2 I would probably leave to Discourse, specifically the job to clean up orphaned uploads, the interval for which is controlled by the clean orphan uploads grace period hours setting.

It might be a good idea to grab the file before it’s deleted though. If it doesn’t contain any sensitive data, it could be useful for testing why it can’t be uploaded during the migration.

Simon_Manning · November 20, 2021, 10:38pm

Something else you could try after having removed the file and completed the migration (and having downloaded the failing file) is to edit the post again to put the file back in. It’d be interesting to know if it fails to upload to S3 entirely or only during the migration task.

Saiano · February 17, 2022, 10:02am

I got the same error, on 64 post. Impossible to find them on 80K files…

What would happen if I delete local files (most of them are alredy migrated)?

Saiano · February 17, 2022, 2:24pm

I confirm that files are migrated, but I got the same error.

Saiano · February 18, 2022, 6:32am

Any suggestions? Uploads are on s3 and work fine, but the task: migrate to s3 fails.

Falco · February 18, 2022, 12:38pm

If you are okay with losing those 64 files, what I did when this same rake task kept failing on me is to edit it, and make it error inly if there are 100 errors instead of 1.

Saiano · February 18, 2022, 2:30pm

It would be fine for me.
What Shoud I do to edit the task?

Falco · February 18, 2022, 3:04pm

You can first try to run it with: SKIP_FAILED=1 rake uploads:migrate_to_s3.

If that doesn’t work edit this line here and try again

github.com

discourse/discourse/blob/main/lib/file_store/to_s3_migration.rb#L77

      
        
            end
            
            
def migration_successful?(should_raise: false)
              success = true
            
            
  failure_message = "S3 migration failed for db '#{@current_db}'."
              prefix = @migrate_to_multisite ? "uploads/#{@current_db}/original/" : "original/"
            
            
  base_url = File.join(SiteSetting.Upload.s3_base_url, prefix)
              count = Upload.by_users.where("url NOT LIKE '#{base_url}%'").count
              if count > 0
                error_message = "#{count} of #{Upload.count} uploads are not migrated to S3. #{failure_message}"
                raise_or_log(error_message, should_raise)
                success = false
              end
            
            
  cdn_path = SiteSetting.cdn_path("/uploads/#{@current_db}/original").sub(/https?:/, "")
              count = Post.where("cooked LIKE '%#{cdn_path}%'").count
              if count > 0
                error_message = "#{count} posts are not remapped to new S3 upload URL. #{failure_message}"
                raise_or_log(error_message, should_raise)

Saiano · February 18, 2022, 4:04pm

Tried with SKIP_FAILED=100 rake uploads:migrate_to_s3

=> 0 files

Listing S3 files
… … => 81070 files
Syncing files to S3

Updating the URLs in the database…
Removing old optimized images…
Flagging all posts containing lightboxes for rebake…
54453 posts were flagged for a rebake
53 posts are not remapped to new S3 upload URL. S3 migration failed for db ‘default’.

I didn’t understand how to put the modified file on my discourse script

JammyDodger · February 18, 2022, 4:28pm

I may have misunderstood, but I think you need to run SKIP_FAILED=1 to turn the SKIP_FAILED on, rather than assign an amount.

Saiano · February 18, 2022, 4:44pm

Tried with this command. … same
Result

JammyDodger · February 19, 2022, 6:25pm

Ah, that’s unfortunate. There’s @Falco’s other suggestion of editing the rake task - but unfortunately I don’t know how to do that, and I can’t find any instructions when I search.

Hopefully someone else can help?

Saiano · February 20, 2022, 3:06pm

I managed to create myscript.rb on my /shared folder but I can’t run it.

Saiano · February 21, 2022, 12:05pm

Any other suggestions? @vulkanino I noticed that you had a similar problem

Topic		Replies	Views
1 of 16551 uploads are not migrated to S3. S3 migration failed for db 'default' Installation cdn , s3	22	1606	August 10, 2023
Uploads migration to S3 fails Support	26	2949	May 7, 2021
Migrating Site to S3 Installation s3	2	708	May 25, 2020
Rake uploads:migrate_from_s3 fails Installation s3	31	2787	April 24, 2021
Rake uploads:migrate_to_s3 error Installation s3	8	755	January 6, 2020

Migrate_to_s3 fails

Related topics