Letter images blank behind proxy server

support
1

I had a similar issue a while ago - Avatar images broken behind proxy

Letter avatar images are generating a 500 error when loaded directly. From within the Docker container, I can curl https://avatars.discourse.org/ through our proxy server as well as https://avatars.discourse-cdn.com/.

I can’t find any errors in the logs that would indicate why it’s failing, either in nginx or the discourse production.log.

Any idea where I should be looking for errors, or if there’s something obvious that happened in the last ~week that I don’t know about?

2

I am still having this problem. I dug into the code a bit, and as I expected, https://avatars.discourse-cdn.com/#{params[:version]}/letter/#{params[:letter]}/#{params[:color]}/#{params[:size]}.png" is what the code is actually requesting. Within the Discourse container, I’m able to wget the file directly and get a valid PNG. For example: https://avatars.discourse-cdn.com/v4/letter/b/bcef8e/48.png → https://forum.pkp.sfu.ca/48.png

In production.log, it shows the image being successfully downloaded, allegedly:


Started GET "/presence/get?channels%5B%5D=%2Fdiscourse-presence%2Freply%2F88689&channels%5B%5D=%2Fdiscourse-presence%2Fwhisper%2F88689" for 172.17.1.79 at 2024-07-09 19:59:01 +0000
Processing by PresenceController#get as JSON
Parameters: {"channels"=>["/discourse-presence/reply/88689", "/discourse-presence/whisper/88689"]}
Processing by UserAvatarsController#show_proxy_letter as PNG
Parameters: {"version"=>"v4", "letter"=>"b", "color"=>"bcef8e", "size"=>"48"}
Completed 200 OK in 14ms (Views: 0.5ms | ActiveRecord: 0.0ms | Allocations: 3095)
Completed 418  in 9ms (Views: 0.4ms | ActiveRecord: 0.0ms | Allocations: 1680)

So for whatever reason, I’m still getting a blank image. Maybe it’s not using HTTP_PROXY or HTTPS_PROXY or something.

In site_settings.yml, it says that the default letter_avatar_proxy url is: "/letter_avatar_proxy/v4/letter/{first_letter}/{color}/{size}.png" and when I rightclick on one of the images to get the URL, it is (correctly?) https://forum.pkp.sfu.ca/letter_avatar_proxy/v4/letter/b/bcef8e/48.png

3

Doing a curl -I https://forum.pkp.sfu.ca/letter_avatar_proxy/v4/letter/r/4491bb/48.png returns a 500 Internal Server Error.

Weirdly, I seem to get two GET requests. I wonder if our loadbalancer is misconfigured.

edit: Okay, it’s not the LB, the two GET request thing was a red herring.

4

This is pretty frustrating. I can’t figure out why letter avatar images are returning 500.

5

Ahhah, I found out that Discourse has its OWN LOG SECTION ONLY ACCESSIBLE VIA THE WEB USER INTERFACE. /facepalm.

This mentions show_proxy_letter so it’s probably relevant.

Message (70085 copies reported)

Failed to process hijacked response correctly : EOFError : end of file reached

Backtrace

/usr/local/lib/ruby/3.3.0/net/protocol.rb:237:in `rbuf_fill'
/usr/local/lib/ruby/3.3.0/net/protocol.rb:199:in `readuntil'
/usr/local/lib/ruby/3.3.0/net/protocol.rb:209:in `readline'
/usr/local/lib/ruby/3.3.0/net/http/response.rb:158:in `read_status_line'
/usr/local/lib/ruby/3.3.0/net/http/response.rb:147:in `read_new'
/usr/local/lib/ruby/3.3.0/net/http.rb:1625:in `connect'
/var/www/discourse/lib/final_destination/http.rb:27:in `block in connect'
/var/www/discourse/lib/final_destination/http.rb:17:in `each'
/var/www/discourse/lib/final_destination/http.rb:17:in `each_with_index'
/var/www/discourse/lib/final_destination/http.rb:17:in `connect'
/usr/local/lib/ruby/3.3.0/net/http.rb:1580:in `do_start'
/usr/local/lib/ruby/3.3.0/net/http.rb:1569:in `start'
/usr/local/lib/ruby/3.3.0/net/http.rb:1029:in `start'
/var/www/discourse/lib/final_destination.rb:556:in `safe_session'
/var/www/discourse/lib/final_destination.rb:491:in `safe_get'
/var/www/discourse/lib/final_destination.rb:170:in `get'
/var/www/discourse/lib/file_helper.rb:73:in `download'
/var/www/discourse/app/controllers/user_avatars_controller.rb:176:in `proxy_avatar'
/var/www/discourse/app/controllers/user_avatars_controller.rb:54:in `block in show_proxy_letter'
/var/www/discourse/lib/hijack.rb:64:in `instance_eval'
/var/www/discourse/lib/hijack.rb:64:in `block in hijack'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/concurrent-ruby-1.3.3/lib/concurrent-ruby/concurrent/promises.rb:911:in `callback_on_resolution'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/concurrent-ruby-1.3.3/lib/concurrent-ruby/concurrent/promises.rb:797:in `call_callback'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/concurrent-ruby-1.3.3/lib/concurrent-ruby/concurrent/promises.rb:803:in `call_callbacks'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/concurrent-ruby-1.3.3/lib/concurrent-ruby/concurrent/promises.rb:692:in `resolve_with'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/concurrent-ruby-1.3.3/lib/concurrent-ruby/concurrent/promises.rb:1325:in `resolve'
/var/www/discourse/lib/scheduler/defer.rb:115:in `block in do_work'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-6.0.0/lib/rails_multisite/connection_management/null_instance.rb:49:in `with_connection'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-6.0.0/lib/rails_multisite/connection_management.rb:21:in `with_connection'
/var/www/discourse/lib/scheduler/defer.rb:109:in `do_work'
/var/www/discourse/lib/scheduler/defer.rb:97:in `block (2 levels) in start_thread'
6

I turned off external system avatars entirely.