Webhooks feedback

[fantasticfears (Erick Guan)]

Webhook is merged to master. See Setting up webhooks for instructions.

Webhooks is my GSoC projects which ends in Aug 23. It’s fully operating but I would keep improving it on more event types as well as payload improvement. I would like to hear comments about how you like to use it. The question like I need X in the payload and I need more information in the payload is generally welcomed.

This topic works as a progress report and feedback topic. Please leave your comments and questions.

Code

Code is merged to the core.

Demo bot: https://github.com/fantasticfears/discourse-webhook-github-issue

Highlights

If you are looking for instruction for webhooks, take a look at this #howto.

Event type and event name

For example, post_created and post_destroyed are two events for plugin authors. Webhooks also take advantage of that. Once such event happens internally, a webhook may be also triggered. Let’s assume a webhook enables post event.
So, post_created or post_destroyed happend. The webhook triggered. X-Discourse-Event-Type: post with post json is sent to you.

Do you need event name in the webhook? And how does that helps your application?

• Yes
• No

X-Discourse-Event shows the exact internal event now.

Event payload serialization

For example, the payload of post event is the exact same as Ember application gets. But the operator is our system user so that can_edit? is also sent to you which I believe there aren’t anything useful to your application.

Do you want me to omit those fields? Or what else information do you need?

• Yes
• No

Webhook event list

In our event delivery list, new event is not updated unless you refresh the page. And recent status is updated in the webhook list. Do you think it’s necessary to notice those changes in real time? How does that helps?

• Yes
• No

A blue notice shows the incoming event if available.

[Lapinot]

Full disclosure: I didn’t checkout and live-test this branch.

Awesome work, I am looking forward to use this feature. Some early feedback/questions:

• Why isn’t this a plugin (is it too deeply integrated in the core)? I am not really comfortable with testing a custom branch as it means a full manual install until it gets merged (a beta/unstable plugin would be much more lightweight).
• Some ideas for other hooks (that don’t seem to exist now after reading the docs):
  • edit-post which also triggers when a post gets edited.
  • watch-first-post which is like edit-post but for every first post in a topic.
• Filters on top of hooks which match against the payload json to restrict the trigger. Agreed, this can be implemented on the remote side of the webhook but sometimes it could save a lot of bandwith. This could be paired with an output filter which selects the wanted json fields (or produces some other structure based on the initial payload). This could be implemented with mongodb-style json querying.

I just looked at the commits and discovered the existence of DiscourseEvent. Is there any documented list of such events or am I supposed to grep the source code for DiscourseEvent.trigger?
Edit: So I grepped and there are not so much such events. I don’t really understand “post validation” but the validate_post event might be relevent for my edit-post hook proposition.

[tgxworld (Alan Tan)]

Why isn’t this a plugin (is it too deeply integrated in the core)? I am not really comfortable with testing a custom branch as it means a full manual install until it gets merged (a beta/unstable plugin would be much more lightweight).

It is going to be merged into core but we didn’t want to ship it in version 1.6 so we developed the feature on another branch first. I’ll be merging it to core this coming week.

[fantasticfears (Erick Guan)]

watch-first-post which is like edit-post but for every first post in a topic.

watch-first-post sounds like a different type of thing. What types of application do you want to build? validate_post happens just before persistent it

Is there any documented list of such events or am I supposed to grep the source code for DiscourseEvent.trigger?

Unfortunately no. Those hooks are set for plugin authors and continue to grow. So, just search or propose something you need in the exact place.

[Lapinot]

Actually I wasn’t really clear, I meant “like edit-post but for the first post of every topic”. My usecase for this one is to extract the first posts of some specially formatted topics to publish them on a static generated website (and stay in sync if it gets updated):

Wiki sync plugin

Hi everyone, I had an idea of a simple plugin that I am going to write but I may need some help/advices. The idea is just to sync up the posts in wiki mode with some external version control (a git server). This is really simple if there is a server-side discourse hook like after-edit-post or something similar but I can’t find it. My use case is that I don’t like CMSs but I need a collaboratively built website (kinda like a wiki) so I would like to have a automatically run static site generator…

[fantasticfears (Erick Guan)]

I get it. Now you would have to use Post Event and check its post_number.

As of customized payload, I am not sure it’s worth it. A filter is harder to learn than just throw others away. If there is a way to get edit-post for first post, I guess it doesn’t matter how large the payload it is as it happens less than you notice.

[Lapinot]

@fantasticfears

As of customized payload, I am not sure it’s worth it.

Yeah, I do agree, but just filtering may be really handy: if I set up the post webhook here on meta, I will maybe get a call every couple minutes (maybe more I don’t know), a lot more than say the edition on the first post of a topic starting with [news]. And this query language isn’t complicated, for this example the filter query would be something like:

{'post': {'post_number': 1, 'topic_title': {'$regex':'^\[news\]'}}}

And here is a sample implementation in python as bonus (I’ll try to do it in ruby to make a pr or something).

https://gist.github.com/Lapin0t/3862ada8bb5f0bf58e101096a6e06c81

[tgxworld (Alan Tan)]

Ok webhooks feature has been merged into core

[codinghorror (Jeff Atwood)]

Is there a webhook for “new user account created and email was just confirmed?” If not, can we make sure we add one?

[tgxworld (Alan Tan)]

Not yet. Right now webhooks are only triggered for create/destroy/edit events on topics and posts.

@fantasticfears can you have a look to see if it is easy to add one?

[tgxworld (Alan Tan)]

Some feedback

Events page for webhook

1. Is it possible to have message bus publish the events so that we get live updates instead of having to refresh the page?
2. 
   We should probably add headers to explain what each column mean.
3. The ping button needs to be disabled until the previous ping event completes. Otherwise, I can spam pings at the hook.

[tgxworld (Alan Tan)]

Another thing I noticed is that there is no differentiation between creation, revision and deletion in the web hook header.

[fantasticfears (Erick Guan)]

Sure, I will start working on them.

[Semaphorism]

I have a question, the webhooks say you can’t have localhost events, I am trying it on http://127.0.0.1:1245, and it says this message

[quote]It seems you are trying to set up the webhook to a local url. Event
delivered to a local address may cause side-effect or unexpected
behaviours. Continue?[/quote]

Is it possible to allow that? I am trying to setup a webhook to a local nodejs server for a project, but the requests don’t appear to go through to localhost, is there a way to do so? I don’t want to have to write a plugin for this.

[riking (Kane York)]

Yep, that’s what the warning was talking about.

You need to give your server’s external IP, or the docker link’s host IP (172.[1,2,3]*.*.*) as the webhook destination - I’d recommend the external IP.

Of course, then you need to watch for other people sending garbage to the server…

[Semaphorism]

Ahh thanks
I can always setup a IP tables rule to deny outside access to it anyway, or just whitelist the IP on the node server itself. That does work using the external IP of the server as well!

[DanielMarquard]

Great work on this! I’ve set one up to talk to a PHP script I’m working on, but I’m having trouble differentiating the event data. I have the webhook configured to trigger on post and topic events, but the payload doesn’t contain the event names you mentioned in your topic on setting up webhooks.

Also, I noticed that the post event trigger says in Discourse that it fires when posts are edited and deleted, but this wasn’t the case on my website. This is not important for my use case, just thought I’d let you know.

[DanielMarquard]

Oh! I just realized that the event name is sent in the X-Discourse-Event header. Perfect.

[Bala_V]

As of customized payload, I am not sure it’s worth it.

I don’t agree with this. We are integrating discourse with a healthcare mobile app. One of the main features is to notify all the contributors of a specific topic whenever there is an “event” on it. With more than 50K users in the platform, the payload size is really huge. Specifically, this is due to the “actions_summary” field in the payload. If there is a way to filter it out, the payload size reduces drastically and becomes manageable.

[Falco (Rafael dos Santos Silva)]

But are you sending this hook to 50k endpoints? Or to another system that parses and then send the notifications? Couldn’t this system filter the fields?