Webhook Won't Delete: CPU Rails to 100% for 30 Seconds, then 502 Error

Bug
1

I have an old Webhook that I want to delete. But when I go to delete it, things hang for 30 seconds and then a 502 Error pops up.

This dumps to console:
image

During this 30 seconds, one of my four ruby processes goes to 100% and stays there (I’m just monitoring this in top), so I think this is more of a bug or misconfiguration than an issue on my end (see note below).

Nothing is in my /logs page, but here’s my logs:

E, [2017-10-28T01:49:29.428736 #213] ERROR -- : worker=2 PID:28593 timeout (31s > 30s), killing

==> error.log <==
2017/10/28 01:49:29 [error] 209#209: *66829 upstream prematurely closed connection while reading response header from upstream, client: MY.IP.ADD.DY, server: _, request: "DELETE /admin/api/web_hooks/1 HTTP/1.1", upstream: "http://127.0.0.1:3000/admin/api/web_hooks/1", host: "forum.mydomain.com", referrer: "https://forum.mydomain.com/admin/api/web_hooks"

That’s after running tail -f /var/log/nginx/error.log /var/log/nginx/error.log.1 /var/log/nginx/error.letsencrypt.log /shared/log/rails/unicorn.stderr.log while inside of my web_only container (my data and web containers are separated).

Note: I’m indeed having some problems elsewhere in my installation, much of it related to CloudFlare (Cannot Save Theme Customizations: 403 Error on PUT). But this is happening even without CloudFlare running at all, DNS or otherwise.

Thanks for any help. Not sure why this webhook’s so painful to kill - a short workaround would be to bump up the timeout??

1 Like
2

Have you tried deleting that webhook from the rails console?

Something like

# ssh into your server
cd /var/discourse
./launcher enter app
rails c
WebHook.find(<id>).destroy

With “<id>” the id of the webhook which you can get when editing the details of the webhook.

  • URL = https://meta.discourse.org/admin/api/web_hooks/3
  • ID = 3
2 Likes
3

Thanks, that worked!

Interestingly, it seemed to take less than 30 seconds to run that. Not sure why the web-based method took so long it had to time out.

Might be nice to get a longer timeout on some of these operations though?

Either way, much appreciated.

2 Likes