Custom Avatars "gone" after updating to 3.3.0beta1

Hi,

I recently updated our testforums to 3.3.0beta1. The testforum basically is a bit of an outdated copy (content wise) of our live forums. We use it to test updates and new features. It uses the same Amazon S3 connection for uploads as the live forums.

After updating the host OS to latest Ubuntu 24.04. LTS and updating the forums to 3.3.0beta1 all custom avatars are gone and the white/gray head displays instead.

After looking at the logs I find messages like:

Could not find file in the store located at url: //ourbucket.s3.dualstack.eu-central-1.amazonaws.com/original/3X/6/9/69ca9110f27d91561axyz52a9cd9485a970fe9.jpeg

Even though in fact the image exists at that URL and works quite well. I tried wgetting the file from the server to see if thereā€™s any DNS issues or other stuff that could hinder the local avatar proxy to get it - but thatā€™s not the case as well.

Any idea what might be going on here?

Looks like this is not only happening on the test forums but also the live forums show disappearing avatars piece by piece with the same errors in the log.

I wonder if the test forum is cleaning up things it doesnā€™t expect in the s3 bucket, but you say that the stuff is in the bucket, so thatā€™s not it. Maybe restore a database to testing and look at what is in the uploads model.

The things is - the error message in the log points out that it canā€™t load the certain image but that exact image is infact available. So thereā€™s something hindering the forum/avatar proxy from pulling the image I suppose.

1 Like

I just uploaded a new avatar image on the testserver (running 3.3.0beta1). It uploaded, showed up in the preview but then again failed to load.

/var/www/discourse/app/models/optimized_image.rb:81:in `block in create_for' 
/var/www/discourse/app/models/optimized_image.rb:19:in `block (2 levels) in lock' 
/var/www/discourse/lib/distributed_mutex.rb:53:in `block in synchronize' 
/var/www/discourse/lib/distributed_mutex.rb:49:in `synchronize' 
/var/www/discourse/lib/distributed_mutex.rb:49:in `synchronize' 
/var/www/discourse/lib/distributed_mutex.rb:34:in `synchronize' 
/var/www/discourse/app/models/optimized_image.rb:19:in `block in lock' 
/var/www/discourse/lib/distributed_mutex.rb:53:in `block in synchronize' 
/var/www/discourse/lib/distributed_mutex.rb:49:in `synchronize' 
/var/www/discourse/lib/distributed_mutex.rb:49:in `synchronize' 
/var/www/discourse/lib/distributed_mutex.rb:34:in `synchronize' 
/var/www/discourse/app/models/optimized_image.rb:18:in `lock' 
/var/www/discourse/app/models/optimized_image.rb:73:in `create_for' 
/var/www/discourse/app/models/upload.rb:130:in `get_optimized_image' 
/var/www/discourse/app/controllers/user_avatars_controller.rb:218:in `get_optimized_image' 
/var/www/discourse/app/controllers/user_avatars_controller.rb:136:in `show_in_site' 
/var/www/discourse/app/controllers/user_avatars_controller.rb:89:in `block (2 levels) in show' 
/var/www/discourse/lib/hijack.rb:64:in `instance_eval' 
/var/www/discourse/lib/hijack.rb:64:in `block in hijack' 
concurrent-ruby-1.2.3/lib/concurrent-ruby/concurrent/promises.rb:911:in `callback_on_resolution' 
concurrent-ruby-1.2.3/lib/concurrent-ruby/concurrent/promises.rb:797:in `call_callback' 
concurrent-ruby-1.2.3/lib/concurrent-ruby/concurrent/promises.rb:803:in `call_callbacks' 
concurrent-ruby-1.2.3/lib/concurrent-ruby/concurrent/promises.rb:692:in `resolve_with' 
concurrent-ruby-1.2.3/lib/concurrent-ruby/concurrent/promises.rb:1325:in `resolve' 
/var/www/discourse/lib/scheduler/defer.rb:115:in `block in do_work' 
rails_multisite-5.0.1/lib/rails_multisite/connection_management/null_instance.rb:49:in `with_connection'
rails_multisite-5.0.1/lib/rails_multisite/connection_management.rb:21:in `with_connection'
/var/www/discourse/lib/scheduler/defer.rb:109:in `do_work' 
/var/www/discourse/lib/scheduler/defer.rb:97:in `block (2 levels) in start_thread' 

This is from the log.

The log message for that is:

Could not find file in the store located at url: //ourbucket.s3.dualstack.eu-central-1.amazonaws.com/original/3X/0/2/02832e36e27bbad791fda46e2290df31e5ee2dda.jpeg

(modified the URL so that wonā€™t work though the real one works)

ā€˜block in hijackā€™ is what gets me.

We are running behind Cloudflare on the live system. The test system is ā€œDNS onlyā€ so obiously not filtered through cloudflare. We did have some issues lately with bots on the main site which we played around with Cloudflare filters quite a bit.

But then why would fetching from an amazonaws.com url be a problem for that - and again the test site isnā€™t run through cloudflare at all.

Iā€™m a bit confused right now to be honest.

Iā€™m really stuck on that one. It seems to affect both the live and the test forums - live is still on an older version and the problem doesnā€™t even seem to have started with the update but appeared a few days ago without any notable changes happening.

So what happens is that Discourse uploads new custom avatars correctly to S3, puts ACLs on them as far as I can see correctly and the files are also publicly accessible through the URL that discourse tries to use to pull them into the avatar proxy - still it fails to do so (see stack trace above).

Does anyone with good knowledge about how the avatar proxy works any idea or can pull anything from that stack trace?

I did do a wget of the image URL from the servers and they all work fine - so nothing there that should block the access to that URL.

And you can get them from your browser?

Ah yes, sorry if that was unclear. I can access them from the browser (so i assumed public availability) and also from the server (to exclude itā€™s some kind of trouble the server has getting them. Itā€™s really just discourse not getting them for whatever reason (see stack trace above).

1 Like

Did you found a solution to your problem? Weā€™re having exactly the same issue on version 3.2.1. The bucket has all public access blocked, however it was still working through presigned URLs. Now Iā€™m having exactly the same error:

Could not find file in the store located at url: