Migrate_to_s3 fails

Something else you could try after having removed the file and completed the migration (and having downloaded the failing file) is to edit the post again to put the file back in. It’d be interesting to know if it fails to upload to S3 entirely or only during the migration task.

I got the same error, on 64 post. Impossible to find them on 80K files…

What would happen if I delete local files (most of them are alredy migrated)?

I confirm that files are migrated, but I got the same error.

Any suggestions? Uploads are on s3 and work fine, but the task: migrate to s3 fails.

If you are okay with losing those 64 files, what I did when this same rake task kept failing on me is to edit it, and make it error inly if there are 100 errors instead of 1.

It would be fine for me.
What Shoud I do to edit the task?

You can first try to run it with: SKIP_FAILED=1 rake uploads:migrate_to_s3.

If that doesn’t work edit this line here and try again

Tried with SKIP_FAILED=100 rake uploads:migrate_to_s3

=> 0 files

  • Listing S3 files
    … … => 81070 files
  • Syncing files to S3

Updating the URLs in the database…
Removing old optimized images…
Flagging all posts containing lightboxes for rebake…
54453 posts were flagged for a rebake
53 posts are not remapped to new S3 upload URL. S3 migration failed for db ‘default’.

I didn’t understand how to put the modified file on my discourse script

I may have misunderstood, but I think you need to run SKIP_FAILED=1 to turn the SKIP_FAILED on, rather than assign an amount.

1 Like

Tried with this command. … same
Result

Ah, that’s unfortunate. There’s @Falco’s other suggestion of editing the rake task - but unfortunately I don’t know how to do that, and I can’t find any instructions when I search. :slightly_smiling_face:

Hopefully someone else can help? :crossed_fingers:

1 Like

I managed to create myscript.rb on my /shared folder but I can’t run it.

Any other suggestions? @vulkanino I noticed that you had a similar problem

Migrating uploads to S3 for ‘default’…
Uploading files to S3…

  • Listing local files
    find: ‘uploads/default/original’: No such file or directory
    => 0 files
  • Listing S3 files
    … => 81070 files
  • Syncing files to S3

Updating the URLs in the database…
Removing old optimized images…
Flagging all posts containing lightboxes for rebake…
54453 posts were flagged for a rebake
rake aborted!
FileStore::ToS3MigrationError: 53 posts are not remapped to new S3 upload URL. S3 migration failed for db ‘default’.
/var/www/discourse/lib/file_store/to_s3_migration.rb:132:in raise_or_log' /var/www/discourse/lib/file_store/to_s3_migration.rb:87:in migration_successful?’
/var/www/discourse/lib/file_store/to_s3_migration.rb:373:in migrate_to_s3' /var/www/discourse/lib/file_store/to_s3_migration.rb:66:in migrate’
/var/www/discourse/lib/tasks/uploads.rake:123:in migrate_to_s3' /var/www/discourse/lib/tasks/uploads.rake:102:in block in migrate_to_s3_all_sites’
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/rails_multisite-4.0.0/lib/rails_multisite/connection_management.rb:80:in with_connection' /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/rails_multisite-4.0.0/lib/rails_multisite/connection_management.rb:90:in each_connection’
/var/www/discourse/lib/tasks/uploads.rake:100:in migrate_to_s3_all_sites' /var/www/discourse/lib/tasks/uploads.rake:96:in block in ’
/usr/local/bin/bundle:25:in load' /usr/local/bin/bundle:25:in
Tasks: TOP => uploads:migrate_to_s3
(See full trace by running task with --trace)

Yes, you may have already seen the discussion here:

Yes I just read this topic. I don’t use volumes, so I proceed to put discourse cdn url = “” , but I’m still getting the error.
I can’t find how to modify to_s3_migration.rb as @Falco suggested

Looking at the error message from your previous post, it’s the count on line 85 which is of interest. @Falco’s suggestion for editing the script is to increase that count to ignore some number of failures.

The simplest way to achieve that is by entering the container and using sed to make an in-place substitution on that specific line.

cd /var/discourse
./launcher enter app
sed -i '85s/0/100/' lib/file_store/to_s3_migration.rb

Run the rake task again after that and it will ignore up to 100 “post not remapped to new S3 upload URL” errors. The change will disappear the next time you rebuild.

For the error message in the original post, line 77 is the one of interest as @Falco referenced. Just replace the line number in the sed command:

sed -i '77s/0/100/' lib/file_store/to_s3_migration.rb
7 Likes

You can always reach out to vim by installing it with apt install vim inside the image.

4 Likes

DONE! WONDERFUL Thank you!!

3 Likes