Avatar problem after migrating to s3

migrated all my uploads to s3, and after cleaning/rebuilding/rebaking, im getting this in my error log and tons of uploaded avatars are not working.

Could not find file in the store located at url: https://roon-community-uploads.s3.amazonaws.com/original/3X/5/9/59407c3c6e6078c478a255c862445bf9065e63bc.JPG

the stack is:

/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/logster-1.2.11/lib/logster/logger.rb:94:in `add_with_opts'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/logster-1.2.11/lib/logster/logger.rb:51:in `add'
/usr/local/lib/ruby/2.5.0/logger.rb:545:in `error'
/var/www/discourse/app/models/optimized_image.rb:62:in `block in create_for'
/var/www/discourse/app/models/optimized_image.rb:18:in `block (2 levels) in lock'
/var/www/discourse/lib/distributed_mutex.rb:21:in `synchronize'
/var/www/discourse/lib/distributed_mutex.rb:5:in `synchronize'
/var/www/discourse/app/models/optimized_image.rb:17:in `block in lock'
/var/www/discourse/lib/distributed_mutex.rb:21:in `synchronize'
/var/www/discourse/lib/distributed_mutex.rb:5:in `synchronize'
/var/www/discourse/app/models/optimized_image.rb:16:in `lock'
/var/www/discourse/app/models/optimized_image.rb:41:in `create_for'
/var/www/discourse/app/models/upload.rb:64:in `get_optimized_image'
/var/www/discourse/app/controllers/user_avatars_controller.rb:188:in `get_optimized_image'
/var/www/discourse/app/controllers/user_avatars_controller.rb:122:in `show_in_site'
/var/www/discourse/app/controllers/user_avatars_controller.rb:87:in `block (2 levels) in show'
/var/www/discourse/lib/hijack.rb:56:in `instance_eval'
/var/www/discourse/lib/hijack.rb:56:in `block in hijack'
/var/www/discourse/lib/scheduler/defer.rb:70:in `block in do_work'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rails_multisite-2.0.4/lib/rails_multisite/connection_management.rb:63:in `with_connection'
/var/www/discourse/lib/scheduler/defer.rb:68:in `do_work'
/var/www/discourse/lib/scheduler/defer.rb:59:in `block (2 levels) in start_thread

and the request url is:


any ideas on how to fix this up?

Looks like this is caused by missing optimized images on s3. You can try to run this in your console:


done, but I have about 500k jobs in sidekiq… not making any progress at getting smaller.

I have 3 unicorn workers… bumping up to 6 now and rebuilding now

bumped up sidekiq processes too, but now, after rebuild, the site is returning:

The software powering this discussion forum encountered an unexpected problem. We apologize for the inconvenience.

Detailed information about the error was logged, and an automatic notification generated. We'll take a look at it.

No further action is necessary. However, if the error condition persists, you can provide additional detail, including steps to reproduce the error, by posting a discussion topic in the site's feedback category.

ok, fixed by “oops” by reducing sidekiq processes… I set it too high and was accidentally flooding postgres.

bumped up my machine to a much faster digital ocean instance for a day, to run through the queue… its going strong now… should be done in the next hour or two

ok, queue has been emptied… but i still have no avatars and still getting the error in the original post

unsure what the last part was, but it took a while and everything kinda just started to return…

i still cant find out where my profile avatar is in the db, because my user id’s user_avatar’s table has no custom avatar id… but all seems to be working.