We have been pushing our new members to go through the discobot tutorial and quite a few are hitting a problem we had already experienced in a previous (staging) Discourse instance: the certificate in the last post takes a long time to generate and to actually appear, with no indication that it is being loaded/generated. By long time I mean a 10-20s period — enough for someone to give up on it. I tried on meta and it seems to be instant here.
I noticed that the certificate can be generated on demand using the /discobot/certificate.svg?date=Oct+28+2020&user_id=123
URL, so I did some tests. At first, it was instant. But as soon as I used user ids that didn’t do the tutorial recently, I had to wait the 10-20s period (it’s fast once generated the first time, so I assume there’s cache somewhere). I also got some 500 errors during the waiting period which I assume was a timeout of some sort because it worked on the next refresh.
On the /logs
, I’m seeing a warning which I don’t know if it is related:
Failed to process hijacked response correctly : Net::ReadTimeout
traceback
/usr/local/lib/ruby/2.6.0/net/protocol.rb:217:in `rbuf_fill'
/usr/local/lib/ruby/2.6.0/net/protocol.rb:191:in `readuntil'
/usr/local/lib/ruby/2.6.0/net/protocol.rb:201:in `readline'
/usr/local/lib/ruby/2.6.0/net/http/response.rb:40:in `read_status_line'
/usr/local/lib/ruby/2.6.0/net/http/response.rb:29:in `read_new'
/usr/local/lib/ruby/2.6.0/net/http.rb:1509:in `block in transport_request'
/usr/local/lib/ruby/2.6.0/net/http.rb:1506:in `catch'
/usr/local/lib/ruby/2.6.0/net/http.rb:1506:in `transport_request'
/usr/local/lib/ruby/2.6.0/net/http.rb:1479:in `request'
rack-mini-profiler-2.0.2/lib/patches/net_patches.rb:9:in `block in request_with_mini_profiler'
rack-mini-profiler-2.0.2/lib/mini_profiler/profiling_methods.rb:39:in `step'
rack-mini-profiler-2.0.2/lib/patches/net_patches.rb:8:in `request_with_mini_profiler'
/var/www/discourse/lib/final_destination.rb:370:in `block in safe_get'
/var/www/discourse/lib/final_destination.rb:414:in `block in safe_session'
/usr/local/lib/ruby/2.6.0/net/http.rb:920:in `start'
/usr/local/lib/ruby/2.6.0/net/http.rb:605:in `start'
/var/www/discourse/lib/final_destination.rb:411:in `safe_session'
/var/www/discourse/lib/final_destination.rb:362:in `safe_get'
/var/www/discourse/lib/final_destination.rb:131:in `get'
/var/www/discourse/lib/final_destination.rb:152:in `get'
/var/www/discourse/lib/file_helper.rb:55:in `download'
/var/www/discourse/plugins/discourse-narrative-bot/plugin.rb:113:in `fetch_avatar'
/var/www/discourse/plugins/discourse-narrative-bot/plugin.rb:98:in `block in generate'
/var/www/discourse/lib/hijack.rb:56:in `instance_eval'
/var/www/discourse/lib/hijack.rb:56:in `block in hijack'
/var/www/discourse/lib/scheduler/defer.rb:94:in `block in do_work'
rails_multisite-2.3.0/lib/rails_multisite/connection_management.rb:68:in `with_connection'
/var/www/discourse/lib/scheduler/defer.rb:89:in `do_work'
/var/www/discourse/lib/scheduler/defer.rb:79:in `block (2 levels) in start_thread'
I take it that the certificate is generated locally, without any external network requests?
The machine is t3a.medium (4gb of memory) and the forum is not particular busy (~0.4 of load, plenty of free memory still). During the wait for the certificate to generate, the CPU didn’t seem to change at all, so that doesn’t seem to be the bottleneck.
We are not the only ones experiencing this (not sure if this is related), but I don’t see anything there that can help.
Not sure if I’m onto something, but if I refreshed the certificate URL a few times, eventually I get an error telling me to slow down. Is there some URL throttling mechanism that may be interfering here if many people are generating the certificates at that same URL path? Possibly not, but I’m at loss on why this is happening. Any hints welcome.