I recently updated our testforums to 3.3.0beta1. The testforum basically is a bit of an outdated copy (content wise) of our live forums. We use it to test updates and new features. It uses the same Amazon S3 connection for uploads as the live forums.
After updating the host OS to latest Ubuntu 24.04. LTS and updating the forums to 3.3.0beta1 all custom avatars are gone and the white/gray head displays instead.
After looking at the logs I find messages like:
Could not find file in the store located at url: //ourbucket.s3.dualstack.eu-central-1.amazonaws.com/original/3X/6/9/69ca9110f27d91561axyz52a9cd9485a970fe9.jpeg
Even though in fact the image exists at that URL and works quite well. I tried wgetting the file from the server to see if thereās any DNS issues or other stuff that could hinder the local avatar proxy to get it - but thatās not the case as well.
Looks like this is not only happening on the test forums but also the live forums show disappearing avatars piece by piece with the same errors in the log.
I wonder if the test forum is cleaning up things it doesnāt expect in the s3 bucket, but you say that the stuff is in the bucket, so thatās not it. Maybe restore a database to testing and look at what is in the uploads model.
The things is - the error message in the log points out that it canāt load the certain image but that exact image is infact available. So thereās something hindering the forum/avatar proxy from pulling the image I suppose.
I just uploaded a new avatar image on the testserver (running 3.3.0beta1). It uploaded, showed up in the preview but then again failed to load.
/var/www/discourse/app/models/optimized_image.rb:81:in `block in create_for'
/var/www/discourse/app/models/optimized_image.rb:19:in `block (2 levels) in lock'
/var/www/discourse/lib/distributed_mutex.rb:53:in `block in synchronize'
/var/www/discourse/lib/distributed_mutex.rb:49:in `synchronize'
/var/www/discourse/lib/distributed_mutex.rb:49:in `synchronize'
/var/www/discourse/lib/distributed_mutex.rb:34:in `synchronize'
/var/www/discourse/app/models/optimized_image.rb:19:in `block in lock'
/var/www/discourse/lib/distributed_mutex.rb:53:in `block in synchronize'
/var/www/discourse/lib/distributed_mutex.rb:49:in `synchronize'
/var/www/discourse/lib/distributed_mutex.rb:49:in `synchronize'
/var/www/discourse/lib/distributed_mutex.rb:34:in `synchronize'
/var/www/discourse/app/models/optimized_image.rb:18:in `lock'
/var/www/discourse/app/models/optimized_image.rb:73:in `create_for'
/var/www/discourse/app/models/upload.rb:130:in `get_optimized_image'
/var/www/discourse/app/controllers/user_avatars_controller.rb:218:in `get_optimized_image'
/var/www/discourse/app/controllers/user_avatars_controller.rb:136:in `show_in_site'
/var/www/discourse/app/controllers/user_avatars_controller.rb:89:in `block (2 levels) in show'
/var/www/discourse/lib/hijack.rb:64:in `instance_eval'
/var/www/discourse/lib/hijack.rb:64:in `block in hijack'
concurrent-ruby-1.2.3/lib/concurrent-ruby/concurrent/promises.rb:911:in `callback_on_resolution'
concurrent-ruby-1.2.3/lib/concurrent-ruby/concurrent/promises.rb:797:in `call_callback'
concurrent-ruby-1.2.3/lib/concurrent-ruby/concurrent/promises.rb:803:in `call_callbacks'
concurrent-ruby-1.2.3/lib/concurrent-ruby/concurrent/promises.rb:692:in `resolve_with'
concurrent-ruby-1.2.3/lib/concurrent-ruby/concurrent/promises.rb:1325:in `resolve'
/var/www/discourse/lib/scheduler/defer.rb:115:in `block in do_work'
rails_multisite-5.0.1/lib/rails_multisite/connection_management/null_instance.rb:49:in `with_connection'
rails_multisite-5.0.1/lib/rails_multisite/connection_management.rb:21:in `with_connection'
/var/www/discourse/lib/scheduler/defer.rb:109:in `do_work'
/var/www/discourse/lib/scheduler/defer.rb:97:in `block (2 levels) in start_thread'
This is from the log.
The log message for that is:
Could not find file in the store located at url: //ourbucket.s3.dualstack.eu-central-1.amazonaws.com/original/3X/0/2/02832e36e27bbad791fda46e2290df31e5ee2dda.jpeg
(modified the URL so that wonāt work though the real one works)
āblock in hijackā is what gets me.
We are running behind Cloudflare on the live system. The test system is āDNS onlyā so obiously not filtered through cloudflare. We did have some issues lately with bots on the main site which we played around with Cloudflare filters quite a bit.
But then why would fetching from an amazonaws.com url be a problem for that - and again the test site isnāt run through cloudflare at all.
Iām really stuck on that one. It seems to affect both the live and the test forums - live is still on an older version and the problem doesnāt even seem to have started with the update but appeared a few days ago without any notable changes happening.
So what happens is that Discourse uploads new custom avatars correctly to S3, puts ACLs on them as far as I can see correctly and the files are also publicly accessible through the URL that discourse tries to use to pull them into the avatar proxy - still it fails to do so (see stack trace above).
Does anyone with good knowledge about how the avatar proxy works any idea or can pull anything from that stack trace?
I did do a wget of the image URL from the servers and they all work fine - so nothing there that should block the access to that URL.
Ah yes, sorry if that was unclear. I can access them from the browser (so i assumed public availability) and also from the server (to exclude itās some kind of trouble the server has getting them. Itās really just discourse not getting them for whatever reason (see stack trace above).
Did you found a solution to your problem? Weāre having exactly the same issue on version 3.2.1. The bucket has all public access blocked, however it was still working through presigned URLs. Now Iām having exactly the same error: