Letter images blank behind proxy server

Support

AI-generated summary

The discussion revolves around an issue with letter avatar images appearing blank behind a proxy server. rbos initially shares a similar issue from a previous thread and explains that the images are generating a 500 error when loaded directly. Despite being able to access the images through a proxy server within the Docker container, the issue persists.

rbos digs deeper into the code and discovers that the request for the image is correct, but the image is still not loading. They find a relevant error message in the production log, which suggests that the image is being successfully downloaded, but not displayed.

After further investigation, rbos finds a log section in the web user interface that mentions a show_proxy_letter error and an EOFError. They suspect that this might be related to the issue.

rbos eventually resolves the issue by turning off external system avatars entirely.

1

I had a similar issue a while ago - Avatar images broken behind proxy

Letter avatar images are generating a 500 error when loaded directly. From within the Docker container, I can curl https://avatars.discourse.org/ through our proxy server as well as https://avatars.discourse-cdn.com/.

I can’t find any errors in the logs that would indicate why it’s failing, either in nginx or the discourse production.log.

Any idea where I should be looking for errors, or if there’s something obvious that happened in the last ~week that I don’t know about?

2

I am still having this problem. I dug into the code a bit, and as I expected, https://avatars.discourse-cdn.com/#{params[:version]}/letter/#{params[:letter]}/#{params[:color]}/#{params[:size]}.png" is what the code is actually requesting. Within the Discourse container, I’m able to wget the file directly and get a valid PNG. For example: https://avatars.discourse-cdn.com/v4/letter/b/bcef8e/48.png → https://forum.pkp.sfu.ca/48.png

In production.log, it shows the image being successfully downloaded, allegedly:


Started GET "/presence/get?channels%5B%5D=%2Fdiscourse-presence%2Freply%2F88689&channels%5B%5D=%2Fdiscourse-presence%2Fwhisper%2F88689" for 172.17.1.79 at 2024-07-09 19:59:01 +0000
Processing by PresenceController#get as JSON
Parameters: {"channels"=>["/discourse-presence/reply/88689", "/discourse-presence/whisper/88689"]}
Processing by UserAvatarsController#show_proxy_letter as PNG
Parameters: {"version"=>"v4", "letter"=>"b", "color"=>"bcef8e", "size"=>"48"}
Completed 200 OK in 14ms (Views: 0.5ms | ActiveRecord: 0.0ms | Allocations: 3095)
Completed 418  in 9ms (Views: 0.4ms | ActiveRecord: 0.0ms | Allocations: 1680)

So for whatever reason, I’m still getting a blank image. Maybe it’s not using HTTP_PROXY or HTTPS_PROXY or something.

In site_settings.yml, it says that the default letter_avatar_proxy url is: "/letter_avatar_proxy/v4/letter/{first_letter}/{color}/{size}.png" and when I rightclick on one of the images to get the URL, it is (correctly?) https://forum.pkp.sfu.ca/letter_avatar_proxy/v4/letter/b/bcef8e/48.png

3

Doing a curl -I https://forum.pkp.sfu.ca/letter_avatar_proxy/v4/letter/r/4491bb/48.png returns a 500 Internal Server Error.

Weirdly, I seem to get two GET requests. I wonder if our loadbalancer is misconfigured.

edit: Okay, it’s not the LB, the two GET request thing was a red herring.

4

This is pretty frustrating. I can’t figure out why letter avatar images are returning 500.

5

Ahhah, I found out that Discourse has its OWN LOG SECTION ONLY ACCESSIBLE VIA THE WEB USER INTERFACE. /facepalm.

This mentions show_proxy_letter so it’s probably relevant.

Message (70085 copies reported)

Failed to process hijacked response correctly : EOFError : end of file reached

Backtrace

/usr/local/lib/ruby/3.3.0/net/protocol.rb:237:in `rbuf_fill'
/usr/local/lib/ruby/3.3.0/net/protocol.rb:199:in `readuntil'
/usr/local/lib/ruby/3.3.0/net/protocol.rb:209:in `readline'
/usr/local/lib/ruby/3.3.0/net/http/response.rb:158:in `read_status_line'
/usr/local/lib/ruby/3.3.0/net/http/response.rb:147:in `read_new'
/usr/local/lib/ruby/3.3.0/net/http.rb:1625:in `connect'
/var/www/discourse/lib/final_destination/http.rb:27:in `block in connect'
/var/www/discourse/lib/final_destination/http.rb:17:in `each'
/var/www/discourse/lib/final_destination/http.rb:17:in `each_with_index'
/var/www/discourse/lib/final_destination/http.rb:17:in `connect'
/usr/local/lib/ruby/3.3.0/net/http.rb:1580:in `do_start'
/usr/local/lib/ruby/3.3.0/net/http.rb:1569:in `start'
/usr/local/lib/ruby/3.3.0/net/http.rb:1029:in `start'
/var/www/discourse/lib/final_destination.rb:556:in `safe_session'
/var/www/discourse/lib/final_destination.rb:491:in `safe_get'
/var/www/discourse/lib/final_destination.rb:170:in `get'
/var/www/discourse/lib/file_helper.rb:73:in `download'
/var/www/discourse/app/controllers/user_avatars_controller.rb:176:in `proxy_avatar'
/var/www/discourse/app/controllers/user_avatars_controller.rb:54:in `block in show_proxy_letter'
/var/www/discourse/lib/hijack.rb:64:in `instance_eval'
/var/www/discourse/lib/hijack.rb:64:in `block in hijack'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/concurrent-ruby-1.3.3/lib/concurrent-ruby/concurrent/promises.rb:911:in `callback_on_resolution'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/concurrent-ruby-1.3.3/lib/concurrent-ruby/concurrent/promises.rb:797:in `call_callback'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/concurrent-ruby-1.3.3/lib/concurrent-ruby/concurrent/promises.rb:803:in `call_callbacks'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/concurrent-ruby-1.3.3/lib/concurrent-ruby/concurrent/promises.rb:692:in `resolve_with'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/concurrent-ruby-1.3.3/lib/concurrent-ruby/concurrent/promises.rb:1325:in `resolve'
/var/www/discourse/lib/scheduler/defer.rb:115:in `block in do_work'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-6.0.0/lib/rails_multisite/connection_management/null_instance.rb:49:in `with_connection'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-6.0.0/lib/rails_multisite/connection_management.rb:21:in `with_connection'
/var/www/discourse/lib/scheduler/defer.rb:109:in `do_work'
/var/www/discourse/lib/scheduler/defer.rb:97:in `block (2 levels) in start_thread'
6

I turned off external system avatars entirely.