Generating missing optimized images with S3

Support
1

After our migration (back) to Amazon S3, we appear to have quite a few images where the optimized versions are missing, but the full-size ones are there (example).

There is a Rake task for regenerating optimized images, but it only appears to work with local storage. How would I do this with remote images?

2 Likes
2

A question for @zogstrip I think!

3

Still waiting for an update on this.

We have some quite noticeable missing post images, and had at least one broken scaled avatar…

1 Like
4

I’m afraid you’ll have to write your own rake task for that. Here’s some pseudo code on how I’d start

FOR EACH oi IN optimized_images
   IF file_exists_on_s3?(oi.upload.url) AND NOT file_exists_on_s3?(oi.url) THEN
       OptimizedImage.create_for(oi.upload, oi.width, oi.height)
   END IF 
END FOR
1 Like
5

@uppfinnarn Did you manage to solve this?

I just migrated to S3, and indeed the /optimized/ folder is not transferred in the migration. I’d like to get rid of the /optimized/ folder, as it adds significant weight to my daily backup.

6

I’m not sure if the following is complete, but this rails console command removes all optimized images then regenerates the ones used for avatars:

OptimizedImage.destroy_all
UserAvatar.all.each {|ua| uaid = ua.gravatar_upload_id || ua.custom_upload_id; Jobs::CreateAvatarThumbnails.perform_async(upload_id: uaid) if uaid }
4 Likes
8

As a followup to the above command, doing a rake posts:rebake will regenerate all(?) of the other missing optimized images.

4 Likes
9

Team, I got stucked in the situation related with the Profile Picture and it wasted around 6 hours of me :frowning: but I found the perfect solution, so sharing here.

STORAGE: Using AWS S3 with CDN in front.

Problem.1: I was able to upload the profile pic from the change profile pic area but it was not reflecting in the profile. I updated my code to the latest Discourse and it started reflecting the profile picture on the UI :slight_smile:

OLD CODE - Was not showing the profile pic


module FileStore

  class S3Store < BaseStore
     ...
    def path_for(upload)
      url = upload&.url
      if url && url[/^\/[^\/]/]
        FileStore::LocalStore.new.path_for(upload)
      else
        url
      end
    end
     ...
  end
end

SOLUTION:LATEST CODE - Started showing the profile pic for new uploads

module FileStore

  class S3Store < BaseStore
     ...
    def path_for(upload)
      url = upload&.url
      FileStore::LocalStore.new.path_for(upload) if url && url[/^\/[^\/]/]
    end
     ...
  end
end

Note: Please guys keep updating your own code with the Discourse upstream on regular basis and it will keep our code updated with security fixes and also, bug free.

Problem.2: Me and my team tried changing profile pictures before fixing the above bug mentioned in the Problem.1. And, it created lots of entries in the uploads table with unknown as the value in the extension column. The code was not correctly processing the profile picture in the image optimization process (MODEL: OptimizedImage) due to wrong s3 file path.

So, I also found a workaround to fix these type of entries those happenend before my code fix.

Solution

Multi Site
We can run the below code directly on the production rails console - or using a rake task.

RailsMultisite::ConnectionManagement.each_connection do |db|
  Upload.where(extension: "unknown").each do |upload|
    upload.update(extension: File.extname(upload.url).gsub(".",""))
  end
  OptimizedImage.destroy_all
  UserAvatar.all.each {|ua| uaid = ua.gravatar_upload_id || ua.custom_upload_id; 
  Jobs::CreateAvatarThumbnails.perform_async(upload_id: uaid) if uaid }
end

Normal - Single Site
We can run the below code directly on the production rails console - or using a rake task.

Upload.where(extension: "unknown").each do |upload|
  upload.update(extension: File.extname(upload.url).gsub(".",""))
end
OptimizedImage.destroy_all
UserAvatar.all.each {|ua| uaid = ua.gravatar_upload_id || ua.custom_upload_id; 
Jobs::CreateAvatarThumbnails.perform_async(upload_id: uaid) if uaid }

I think, it will help lots of people :slight_smile:

Thank You

2 Likes
10

@supermathie Thanks for your advice. If it seems like all the optimised images for avatars are missing, would just running the second line likely be sufficient?

UserAvatar.all.each {|ua| uaid = ua.gravatar_upload_id || ua.custom_upload_id; Jobs::CreateAvatarThumbnails.perform_async(upload_id: uaid) if uaid }

I did a post rebake already, which took quite a long time. So would ideally want to avoid wiping the existing optimised files for post images if possible.

2 Likes
11

probably, but it’s very possible that code has changed; I haven’t looked at that in a while.

I’ll ask around…

1 Like
12

Thanks @supermathie I appreciate it! :slight_smile:

So I went ahead and gave it a shot (just the second line), but doesn’t seem to be working. It returned a lot of records like:

id: 1234,
user_id: 1234,
custom_upload_id: nil,
gravatar_upload_id: nil,
last_gravatar_download_attempt: Thu, 07 May 2015 09:40:35 UTC +00:00,
created_at: Thu, 07 May 2015 09:40:35 UTC +00:00,
updated_at: Thu, 07 May 2015 09:40:36 UTC +00:00>,

I found one that had custom_upload_id set to an id and found the user’s profile using: User.find_by(id: 123456)

When I view their profile though, the avatar is still empty. Also, I can see that no new uploads have been made to s3 recently (none in the last 6hrs). So I don’t think optimized images are being generated as a result.

The good news, is when I search for an avatar via rails command:

User.find_by_username('username').uploaded_avatar

source

The records seem correct and the urls for all the S3 urls I’ve checked so far work. The records seem consistent with another instance that has used S3 from the beginning.

Then when I use:
OptimizedImage.where(upload_id: upload_id).where(version: 2)

On the previous instance, I see a bunch of optimized images. But on the new instance, there are no records. Which seems correct. For that user with the custom upload id set, there seem to be optimized images for them on s3, but those records are from days ago.

So seems that I just need to find a way to trigger generating the optimized avatar images.

The new instance is in read only mode currently, not sure if maybe that is a factor. Rebaking posts seemed to work fine in read only mode.

Edit: I see a lot of jobs in sidekiq for creating avatar thumbnails :slight_smile:

So maybe just need to exit read only mode to allow them to get processed. Will update.

13

So I see Jobs::CreateAvatarThumbnails being run in Sidekiq. But nothing has been uploaded to the s3 bucket in 9 hours. So not sure what to make of that.