Recently a few team members noticed that reloading after saving ruby files was painfully slow. It seems to take about 10 seconds after each save to reload in a browser.
TL;DR: Check out the latest Discourse, add a ENV variable
CHECKOUT_TIMEOUT=0.1for a workaround.
I am quite busy right now but the pain of reloading was significant enough that I spent a few hours spelunking into Discourse and Rails to try and track it down.
The good news is I found a workaround. The bad news is I’m still not sure what the root cause is. I’m writing it up here in hopes that someone smarter than me can figure it out so we can fix it properly.
When reloading in development mode, the clear_reloadable_connections! method is called.
From its comments:
# The pool first tries to gain ownership of all connections. If unable to # do so within a timeout interval (default duration is # <tt>spec.config[:checkout_timeout] * 2</tt> seconds), then the pool forcefully # clears the cache and reloads connections without any regard for other # connection owning threads.
It seems in our application, every time it tries to do this it fails, and has to wait the full 5 * 2 = 10 seconds before continuing, when it works fine.
I tried to dig deeper and quickly got caught up in threads and locks.
My gut feeling is the problem is something in our stack has its hands on a connection, and is not handing it back properly. It always seems to work when the timeout is reached, so simply lowering the timeout to something small like
0.1 makes the request work quickly.
I’ve added support to Discourse to support changing the timeout via an ENV variable:
Things I’ve tried to fix it
Disabling our custom reload code blocks
Different web servers like Webrick instead of Puma
Commenting out “weird” bits of Discourse code
So far, I’ve come up with nothing.
If you have any ideas I’d love to hear them!
Reproducing the bug
rm -rf tmp
- bundle exec rails s
- Visit a dynamic URL such as
/admin/dashboard/problems.json(it should load quickly)
- Make a change to a file such as discourse/dashboard_controller.rb at master · discourse/discourse · GitHub - adding a comment with different text is enough
- Reload the problems.json URL.
- It should spin for 10 seconds until the timeout is reached.