# Generating missing optimized images with S3
**URL:** https://meta.discourse.org/t/generating-missing-optimized-images-with-s3/35847
**Category:** Support
**Created:** [November 20, 2015, 5:24pm UTC](https://meta.discourse.org/t/generating-missing-optimized-images-with-s3/35847 "2015-11-20T17:24:35Z")
**Posts on this page:** 12
**Page:** 1
### Author:  [@uppfinnarn](https://meta.discourse.org/u/uppfinnarn)
#### Post date: [November 20, 2015, 5:24pm UTC](https://meta.discourse.org/t/generating-missing-optimized-images-with-s3/35847/1 "2015-11-20T17:24:35Z")
After our migration (back) to Amazon S3, we appear to have quite a few images where the optimized versions are missing, but the full-size ones are there ([example](https://forum.kazamatsuri.org/t/computers/929/6?u=uppfinnarn)).
There is a Rake task for regenerating optimized images, but it only appears to work with local storage. How would I do this with remote images?
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [November 20, 2015, 10:20pm UTC](https://meta.discourse.org/t/generating-missing-optimized-images-with-s3/35847/2 "2015-11-20T22:20:28Z")
A question for @zogstrip I think!
---
### Author:  [@uppfinnarn](https://meta.discourse.org/u/uppfinnarn)
#### Post date: [December 3, 2015, 5:12pm UTC](https://meta.discourse.org/t/generating-missing-optimized-images-with-s3/35847/3 "2015-12-03T17:12:23Z")
Still waiting for an update on this.
We have some quite noticeable missing post images, and had at least one broken scaled avatar…
---
### Author:  [@zogstrip](https://meta.discourse.org/u/zogstrip)
#### Post date: [December 3, 2015, 5:50pm UTC](https://meta.discourse.org/t/generating-missing-optimized-images-with-s3/35847/4 "2015-12-03T17:50:08Z")
I’m afraid you’ll have to write your own rake task for that. Here’s some pseudo code on how I’d start
```plaintext
FOR EACH oi IN optimized_images
IF file_exists_on_s3?(oi.upload.url) AND NOT file_exists_on_s3?(oi.url) THEN
OptimizedImage.create_for(oi.upload, oi.width, oi.height)
END IF
END FOR
```
---
### Author:  [@ljpp](https://meta.discourse.org/u/ljpp)
#### Post date: [March 23, 2016, 11:38am UTC](https://meta.discourse.org/t/generating-missing-optimized-images-with-s3/35847/5 "2016-03-23T11:38:45Z")
@uppfinnarn Did you manage to solve this?
I just [migrated](https://meta.discourse.org/t/migration-of-system-stored-images-to-s3-after-configuration-change/19605/17) to S3, and indeed the /optimized/ folder is not transferred in the migration. I’d like to get rid of the /optimized/ folder, as it adds significant weight to my daily backup.
---
### Author:  [@supermathie](https://meta.discourse.org/u/supermathie)
#### Post date: [August 27, 2018, 6:16pm UTC](https://meta.discourse.org/t/generating-missing-optimized-images-with-s3/35847/6 "2018-08-27T18:16:15Z")
I’m not sure if the following is complete, but this rails console command removes all optimized images then regenerates the ones used for avatars:
```ruby
OptimizedImage.destroy_all
UserAvatar.all.each {|ua| uaid = ua.gravatar_upload_id || ua.custom_upload_id; Jobs::CreateAvatarThumbnails.perform_async(upload_id: uaid) if uaid }
```
---
### Author:  [@supermathie](https://meta.discourse.org/u/supermathie)
#### Post date: [January 22, 2020, 8:25pm UTC](https://meta.discourse.org/t/generating-missing-optimized-images-with-s3/35847/8 "2020-01-22T20:25:10Z")
As a followup to the above command, doing a `rake posts:rebake` will regenerate all(?) of the other missing optimized images.
---
### Author:  [@ashishprajapati](https://meta.discourse.org/u/ashishprajapati)
#### Post date: [May 9, 2020, 10:26pm UTC](https://meta.discourse.org/t/generating-missing-optimized-images-with-s3/35847/9 "2020-05-09T22:26:53Z")
> [@supermathie](#):
>
> I’m not sure if the following is complete, but this rails console command removes all optimized images then regenerates the ones used for avatars:
>
> ```plaintext
> OptimizedImage.destroy_all
> UserAvatar.all.each {|ua| uaid = ua.gravatar_upload_id || ua.custom_upload_id; Jobs::CreateAvatarThumbnails.perform_async(upload_id: uaid) if uaid }
>
> ```
Team, I got stucked in the situation related with the **Profile Picture** and it wasted around 6 hours of me ☹ but I found the perfect solution, so sharing here.
**STORAGE:** Using **AWS S3** with **CDN** in front.
**Problem.1:** I was able to upload the profile pic from the change profile pic area but it was not reflecting in the profile. I updated my code to the latest **Discourse** and it started reflecting the profile picture on the UI 🙂
**OLD CODE - Was not showing the profile pic**
```plaintext
module FileStore
class S3Store < BaseStore
...
def path_for(upload)
url = upload&.url
if url && url[/^\/[^\/]/]
FileStore::LocalStore.new.path_for(upload)
else
url
end
end
...
end
end
```
**SOLUTION:LATEST CODE - Started showing the profile pic for new uploads**
```plaintext
module FileStore
class S3Store < BaseStore
...
def path_for(upload)
url = upload&.url
FileStore::LocalStore.new.path_for(upload) if url && url[/^\/[^\/]/]
end
...
end
end
```
**Note:** Please guys keep updating your own code with the **Discourse upstream** on regular basis and it will keep our code updated with security fixes and also, bug free.
**Problem.2:** Me and my team tried changing profile pictures before fixing the above bug mentioned in the **Problem.1**. And, it created lots of entries in the **uploads table** with **unknown** as the value in the **extension column**. The **code** was not correctly processing the **profile picture** in the image optimization process **(MODEL: OptimizedImage)** due to wrong s3 file path.
**So, I also found a workaround to fix these type of entries those happenend before my code fix.**
### Solution
**Multi Site**
We can run the below code directly on the production rails console - or using a rake task.
```plaintext
RailsMultisite::ConnectionManagement.each_connection do |db|
Upload.where(extension: "unknown").each do |upload|
upload.update(extension: File.extname(upload.url).gsub(".",""))
end
OptimizedImage.destroy_all
UserAvatar.all.each {|ua| uaid = ua.gravatar_upload_id || ua.custom_upload_id;
Jobs::CreateAvatarThumbnails.perform_async(upload_id: uaid) if uaid }
end
```
**Normal - Single Site**
We can run the below code directly on the production rails console - or using a rake task.
```plaintext
Upload.where(extension: "unknown").each do |upload|
upload.update(extension: File.extname(upload.url).gsub(".",""))
end
OptimizedImage.destroy_all
UserAvatar.all.each {|ua| uaid = ua.gravatar_upload_id || ua.custom_upload_id;
Jobs::CreateAvatarThumbnails.perform_async(upload_id: uaid) if uaid }
```
I think, it will help lots of people 🙂
Thank You
---
### Author:  [@markersocial](https://meta.discourse.org/u/markersocial)
#### Post date: [January 29, 2021, 6:44pm UTC](https://meta.discourse.org/t/generating-missing-optimized-images-with-s3/35847/10 "2021-01-29T18:44:52Z")
@supermathie Thanks for your advice. If it seems like all the optimised images for avatars are missing, would just running the second line likely be sufficient?
```plaintext
UserAvatar.all.each {|ua| uaid = ua.gravatar_upload_id || ua.custom_upload_id; Jobs::CreateAvatarThumbnails.perform_async(upload_id: uaid) if uaid }
```
I did a post rebake already, which took quite a long time. So would ideally want to avoid wiping the existing optimised files for post images if possible.
---
### Author:  [@supermathie](https://meta.discourse.org/u/supermathie)
#### Post date: [January 29, 2021, 9:12pm UTC](https://meta.discourse.org/t/generating-missing-optimized-images-with-s3/35847/11 "2021-01-29T21:12:53Z")
_probably_, but it’s very possible that code has changed; I haven’t looked at that in a while.
I’ll ask around…
---
### Author:  [@markersocial](https://meta.discourse.org/u/markersocial)
#### Post date: [January 30, 2021, 8:22am UTC](https://meta.discourse.org/t/generating-missing-optimized-images-with-s3/35847/12 "2021-01-30T08:22:03Z")
Thanks @supermathie I appreciate it! 🙂
So I went ahead and gave it a shot (just the second line), but doesn’t seem to be working. It returned a lot of records like:
```
id: 1234,
user_id: 1234,
custom_upload_id: nil,
gravatar_upload_id: nil,
last_gravatar_download_attempt: Thu, 07 May 2015 09:40:35 UTC +00:00,
created_at: Thu, 07 May 2015 09:40:35 UTC +00:00,
updated_at: Thu, 07 May 2015 09:40:36 UTC +00:00>,
```
I found one that had custom\_upload\_id set to an id and found the user’s profile using: `User.find_by(id: 123456)`
When I view their profile though, the avatar is still empty. Also, I can see that no new uploads have been made to s3 recently (none in the last 6hrs). So I don’t think optimized images are being generated as a result.
* * *
The good news, is when I search for an avatar via rails command:
```
User.find_by_username('username').uploaded_avatar
```
[source](https://meta.discourse.org/t/avatars-lost-after-restore-how-to-get-them-back/148223/47)
The records seem correct and the urls for all the S3 urls I’ve checked so far work. The records seem consistent with another instance that has used S3 from the beginning.
Then when I use:
`OptimizedImage.where(upload_id: upload_id).where(version: 2)`
On the previous instance, I see a bunch of optimized images. But on the new instance, there are no records. Which seems correct. For that user with the custom upload id set, there seem to be optimized images for them on s3, but those records are from days ago.
So seems that I just need to find a way to trigger generating the optimized avatar images.
The new instance is in read only mode currently, not sure if maybe that is a factor. Rebaking posts seemed to work fine in read only mode.
* * *
Edit: I see a lot of jobs in sidekiq for creating avatar thumbnails 🙂
So maybe just need to exit read only mode to allow them to get processed. Will update.
---
### Author:  [@markersocial](https://meta.discourse.org/u/markersocial)
#### Post date: [January 30, 2021, 10:22am UTC](https://meta.discourse.org/t/generating-missing-optimized-images-with-s3/35847/13 "2021-01-30T10:22:33Z")
So I see Jobs::CreateAvatarThumbnails being run in Sidekiq. But nothing has been uploaded to the s3 bucket in 9 hours. So not sure what to make of that.