# Webhooks/Sidekiq issue on dev instance
**URL:** https://meta.discourse.org/t/webhooks-sidekiq-issue-on-dev-instance/71129
**Category:** Development
**Created:** [October 1, 2017, 2:07pm UTC](https://meta.discourse.org/t/webhooks-sidekiq-issue-on-dev-instance/71129 "2017-10-01T14:07:29Z")
**Posts on this page:** 19
**Page:** 1
### Author:  [@jack2](https://meta.discourse.org/u/jack2)
#### Post date: [October 1, 2017, 2:07pm UTC](https://meta.discourse.org/t/webhooks-sidekiq-issue-on-dev-instance/71129/1 "2017-10-01T14:07:29Z")
I run 2 Discourse instances:
1. a standard one (Docker)
2. a dev one, behind a nginx proxy
I’ve set up the same webhook on both instances. It works well from the standard instance, but not from the dev instance:
- the Ping button never gets a response and stays gray, although the corresponding POST event seems to appear in the nginx log (EDIT: this log entry is about the click event localhost-\>Discourse, not the outgoing webhook ping message).
- I see no error in the Discourse server console or nginx logs.
What should I check?
---
### Author:  [@jack2](https://meta.discourse.org/u/jack2)
#### Post date: [October 1, 2017, 3:12pm UTC](https://meta.discourse.org/t/webhooks-sidekiq-issue-on-dev-instance/71129/2 "2017-10-01T15:12:03Z")
My nginx is set up according to @riking’s [excellent post](https://meta.discourse.org/t/running-other-websites-on-the-same-machine-as-discourse/17247).
---
### Author:  [@hellekin](https://meta.discourse.org/u/hellekin)
#### Post date: [October 1, 2017, 7:07pm UTC](https://meta.discourse.org/t/webhooks-sidekiq-issue-on-dev-instance/71129/3 "2017-10-01T19:07:48Z")
You might need to set a pointer in `/etc/hosts` to `127.0.0.1` (or your public IP, depending on where your nginx is listening) for your dev host.
---
### Author:  [@jack2](https://meta.discourse.org/u/jack2)
#### Post date: [October 2, 2017, 7:36am UTC](https://meta.discourse.org/t/webhooks-sidekiq-issue-on-dev-instance/71129/4 "2017-10-02T07:36:43Z")
Thanks @hellekin.
My nginx routes requests from `www.myhostname.org` to `127.0.0.1:3000`.
So I’ve tried to add `127.0.0.1 www.myhostname.org` to my host file, but it hasn’t solved the problem…
---
### Author:  [@hellekin](https://meta.discourse.org/u/hellekin)
#### Post date: [October 2, 2017, 8:16am UTC](https://meta.discourse.org/t/webhooks-sidekiq-issue-on-dev-instance/71129/5 "2017-10-02T08:16:37Z")
What do the logs say?
---
### Author:  [@jack2](https://meta.discourse.org/u/jack2)
#### Post date: [October 2, 2017, 12:08pm UTC](https://meta.discourse.org/t/webhooks-sidekiq-issue-on-dev-instance/71129/6 "2017-10-02T12:08:18Z")
I don’t see anything pertaining to the webhook call neither in nginx nor Discourse logs.
However, I do see the webhook call in Sidekiq “Enqueued” list. The entry stays there forever. Any idea why Sidekiq never processes the job?
P.S.: because I could curl to my webhook service from the Discourse server, I believe nginx is not involved in the issue. I’m going to change the topic’s title to reflect that.
---
### Author:  [@hellekin](https://meta.discourse.org/u/hellekin)
#### Post date: [October 2, 2017, 12:26pm UTC](https://meta.discourse.org/t/webhooks-sidekiq-issue-on-dev-instance/71129/7 "2017-10-02T12:26:18Z")
Using Firefox or Chromium, do you have network logs?
I’m sorry I can’t help you much, as I didn’t look into web hooks so far.
---
### Author:  [@jack2](https://meta.discourse.org/u/jack2)
#### Post date: [October 2, 2017, 3:50pm UTC](https://meta.discourse.org/t/webhooks-sidekiq-issue-on-dev-instance/71129/8 "2017-10-02T15:50:38Z")
It’s solved now. The problem was about Sidekiq not processing jobs. I did a lot of things (updating Discourse, flushing redis, restarting Sidekiq, changing `database.yml` then restoring it, rebooting the server) and now it works.
Thanks again @hellekin!
---
### Author:  [@jack2](https://meta.discourse.org/u/jack2)
#### Post date: [October 2, 2017, 10:07pm UTC](https://meta.discourse.org/t/webhooks-sidekiq-issue-on-dev-instance/71129/9 "2017-10-02T22:07:11Z")
It was too good to be true. Here is what I do on my dev instance:
1. Create a webhook
2. Use the Ping button: it works (I can trace it all the way up to my webhook service)
3. Create a post: this adds 4 jobs in Sidekiq (2 x “event\_name”=\>“user\_updated” and 2 x “event\_name”=\>“post\_created”). But those jobs stays in the Busy list and aren’t processed.
4. If I keep triggering events, they add up to the busy list. Somewhere along the way, even Ping events get stuck.
5. At that point, I need to flush Redis and restart Sidekiq if I want to go back to point 2.
If I do the same on my Docker instance, it works like a charm.
I also want to mention that, in my admin Dashboard (on the dev instance), I have the following warning: “A check for updates has not been performed. Ensure sidekiq is running.”
---
### Author:  [@gerhard](https://meta.discourse.org/u/gerhard)
#### Post date: [October 2, 2017, 10:19pm UTC](https://meta.discourse.org/t/webhooks-sidekiq-issue-on-dev-instance/71129/10 "2017-10-02T22:19:41Z")
Stuck jobs in development mode are a known problem with Sidekiq and Rails 5.1
It’s probably because of missing dependencies. See the following post for more information on that.
> [@When should Discourse upgrade to Rails 5?](https://meta.discourse.org/t/when-should-discourse-upgrade-to-rails-5/67123/26):
>
> Rails 5.1 is merged into Discourse. One notable change which was a source of pain in development for me is that Sidekiq supports live reloading with Rails 5 but dependencies in each file has to be required properly. Otherwise, the job will be stucked in the Sidekiq queue. I’ve fixed most that I can find but if you run into any other job, feel free to let me know.
Feel free to send a pull request if you find missing dependencies in sidekiq jobs.
Unfortunately the `ProcessPost` job can’t be fixed that way. We are aware of the problem…
As a workaround you can change `config.eager_load` to `true` in `development.rb`
---
### Author:  [@jack2](https://meta.discourse.org/u/jack2)
#### Post date: [October 3, 2017, 7:31am UTC](https://meta.discourse.org/t/webhooks-sidekiq-issue-on-dev-instance/71129/11 "2017-10-03T07:31:31Z")
Thanks a lot @gerhard, setting `config.eager_load` to `true` seemed to solve the issue.
EDIT: I could work for 2 hours without any problem, then the issue came back…
---
### Author:  [@tgxworld](https://meta.discourse.org/u/tgxworld)
#### Post date: [October 4, 2017, 5:53am UTC](https://meta.discourse.org/t/webhooks-sidekiq-issue-on-dev-instance/71129/15 "2017-10-04T05:53:13Z")
Hi @jack2,
The next time it gets stucked, can you run `kill -TTIN `? It’ll print out the backtrace of where the code is stucked at.
---
### Author:  [@jack2](https://meta.discourse.org/u/jack2)
#### Post date: [October 4, 2017, 8:52am UTC](https://meta.discourse.org/t/webhooks-sidekiq-issue-on-dev-instance/71129/16 "2017-10-04T08:52:54Z")
@tgxworld, the trace is too long to be posted here (52390 characters \> 40000 character limit). Please advise.
---
### Author:  [@zogstrip](https://meta.discourse.org/u/zogstrip)
#### Post date: [October 4, 2017, 9:11am UTC](https://meta.discourse.org/t/webhooks-sidekiq-issue-on-dev-instance/71129/17 "2017-10-04T09:11:40Z")
Post it on [https://pastebin.com/](https://pastebin.com/)
---
### Author:  [@jack2](https://meta.discourse.org/u/jack2)
#### Post date: [October 4, 2017, 9:17am UTC](https://meta.discourse.org/t/webhooks-sidekiq-issue-on-dev-instance/71129/18 "2017-10-04T09:17:53Z")
Here it is: [Sidekiq trace log - Pastebin.com](https://pastebin.com/pWuqKVKw)
---
### Author:  [@tgxworld](https://meta.discourse.org/u/tgxworld)
#### Post date: [October 9, 2017, 10:31am UTC](https://meta.discourse.org/t/webhooks-sidekiq-issue-on-dev-instance/71129/19 "2017-10-09T10:31:59Z")
Autoreloading for Sidekiq that was made available with the Rails 5 upgrade wasn’t compatible with our code which was causing the jobs to be stucked. The main problem is that the job that is being execute by sidekiq has to be execute in the same thread as the sidekiq processor. However, that wasn’t the case as we were wrapping each job in a new thread from within Sidekiq iteself. Once I figured out what the problem was, the fix is pretty straight forward.
[https://github.com/discourse/discourse/commit/59aeb0bc56634edd8a8b35f638c30c014a826004](https://github.com/discourse/discourse/commit/59aeb0bc56634edd8a8b35f638c30c014a826004)
It also seems like `ActiveSupport::Concurrency::ShareLock` code in Rails 5 doesn’t have any form of timeout and just waits forever if it can’t acquire the lock.
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [October 9, 2017, 10:33am UTC](https://meta.discourse.org/t/webhooks-sidekiq-issue-on-dev-instance/71129/20 "2017-10-09T10:33:38Z")
> [@tgxworld](#):
>
> It also seems like ActiveSupport::Concurrency::ShareLock code in Rails 5 doesn’t have any form of timeout and just waits forever if it can’t acquire the lock.
We need to add a timeout. Infinite timeout on db related stuff is a recipe for suffering, as we have seen several times now..
---
### Author:  [@tgxworld](https://meta.discourse.org/u/tgxworld)
#### Post date: [October 9, 2017, 10:40am UTC](https://meta.discourse.org/t/webhooks-sidekiq-issue-on-dev-instance/71129/21 "2017-10-09T10:40:37Z")
Yea I’ll get a reproducible script up and open an issue with the Rails team.
---
### Author:  [@tgxworld](https://meta.discourse.org/u/tgxworld)
#### Post date: [October 9, 2017, 2:46pm UTC](https://meta.discourse.org/t/webhooks-sidekiq-issue-on-dev-instance/71129/22 "2017-10-09T14:46:43Z")
It looks like there is something unique about our setup as I couldn’t reproduce it on a fresh Rails app. I’m going to leave this for now as I’ve spent way too much time on this for something that only affects development mode.