Migrate_from_s3 problems

When migrating from S3 to local storage, we see a number of issues.

The main issue is that the migrate_from_s3 rake task is not taking the Uploads table as a starting point, but the posts. This causes it to skip a lot of uploads which are being left on S3.

  • uploads used as logo’s because they are not referenced in a post
  • uploads for avatars because they are not referenced in a post
  • uploads that are (for some reason) referenced by their CDN URL in raw because they do not match the regex that is being used to identify the uploads
  • uploads that are on non-AWS S3 storage because they do not match the regex because it requires amazonaws in the URL
  • uploads that for some reason do not match the second, different regex (we’re seeing this with non-image uploads like mp3 files, not sure why this is happening)

We’re working on improving the way we associate uploads to posts (by using the upload:// scheme) that will make these storage migrations much more bulletproof. There’s little point to fixing these rake tasks before that’s done.

4 Likes