Multiple users reported Sidekiq hanging on Discourse forums, often coinciding with Jobs::BotInput jobs from the Narrative Bot (not AI). The issue affected only a small subset of servers, typically those using the bot heavily. Restarting Sidekiq temporarily resolved it, but the root cause remained elusive until tgxworld analyzed backtraces and traced the problem to a redundant require line in discourse-narrative-bot code. Removing it on affected instances stopped the hangs, suggesting it was the culprit. RGJ confirmed the fix worked on their most problematic servers and requested it be backported to stable. The issue may relate to Rails 7.1’s autoloading behavior — specifically, Rails.autoloaders.main.do_not_eager_load(config.root.join("lib")) — which loic noted was retained for compatibility. No evidence suggests user interaction (e.g., refreshing pages during bot chats) causes hangs. Deleting the critical queue is unsafe; recommended fix is sv reload unicorn. markschmucker initially suspected AI but clarified they used the API as discobot, which triggered the same job. The fix is now merged via PR #31727.
In the past week we have seen three Sidekiq instances on different forums being stuck. There was nothing special going on, it was just that Sidekiq was not processing any work and showing 5 of 5 jobs being processed.
One interesting thing they all had in common was that there was one critical BotInput job among the jobs. Now this is quite a common job, but it still stands out.
After restarting Sidekiq everything works normal again. Manually queuing a job with the same parameters does not cause it to hang again. There is nothing special with the specific post it was called for.
Does anyone have any idea how we could track down what is going on here?
I’m seeing this in the past few days. Eventually all jobs stop running. Previously I rebooted, but is it safe to delete the critical queue? Is it a redis queue?
I’m up-to-date at 3.5.0.beta1-dev.
Just a wild guess, but sometimes when I’m chatting with the bot it stops responding so I refresh the page or give up. Maybe those cases leave a job hanging?
These jobs are asynchronous so they wouldn’t even know that you did that.
It’s interesting to hear that you are having this on Jobs::BotInput as well. We’re seeing this issue on only a small subset of all our servers (a few percent) and it seems to be the instances that use the narrative bot quite heavily.
No, you would lose all the other queued jobs as well.
The most easy and safe way is sv reload unicorn from within the container.
Interesting find, thank you for looking into it.
It’s hard to tell when such an intermittent problem goes away. I have removed that line on the three instances that hung the most often (one of them almost daily). I will check back in here either:
when one of those instances hangs (we then know that this did not do the trick)
on Friday if none of them hung (we can then start assuming it was the solution)
While the issues have been occuring this week, they haven’t been happening on the three instances where we removed that require line, so I think we can safely assume that this is the culprit . Thank you for spotting that @tgxworld , I would have never found that.
. Thank you for spotting that