I’m building AI-powered automations using independent AI triage scripts (e.g., spam check, tag determination). These currently run concurrently, which is inefficient. I need to “chain” them so, for example, the tag script only runs if the spam script doesn’t flag the content.
How can I govern and orchestrate these scripts for a more logical workflow? Specifically, how can I chain these scripts conditionally?
Can you describe the full flow you are after? Would this be something you run on a separate server or would you like this to run in Discourse?
AI workflows is something we are thinking about a lot, the ability to define chains is key with workflows. I agree completely with that.
From what I can see, most of my use cases can be done in Discourse with AI Triage and Automation, if one can trigger then next.
Here is a hypothetical flow where each step would send the post contents and a prompt to the LLM:
This is great and a very interesting use case.
Custom tools already have support now for scripting so we have a great vehicle for this kind of change.
I am thinking Persona + forced tool use - then from the tool we can run the flow given we already have all the infra to do so. Just need to give custom tools the ability to trigger other llm calls, something that is reasonably simple to add.
Interestingly given custom tools have support for rest calls they can run the entire flow (and just use the discourse rest api to wire this up)
Let me have a think about this over the weekend, and will reply again next week with how I think we can swing this.
Automation chaining is also a very interesting approach here, @j.jaffeux have you thought about this problem?
Hi @Cloud_spanner I am trying to get to the bottom of this and would like to flesh out the actual flow here a bit more, can you help with some answer to questions along the way 5. ?
Which posts should be scanned?
Intent
If intent can be manual, should manually tagged stuff also be scanned for “hot button”?
hot an invisible tag or visible tag?Particularly, what I am thinking about here is:
Every new topic should kick off an AI triage workflow. Edits can be ignored.
To be clear, I am using intent to illustrate the workflow so it shouldn’t be considered a hard-coded flow. The point I am trying to make is that there is no reason to kick off ‘intent’ workflow if the first triage workflow deems it unnecessary. +1 to the IFTTT workflow concept.
Intent and ‘hot’ would be invisible tags in this example visible to mods and admins only.
There should be one intent tag per post.
Yes.
What if a private tag was used to show that the flow has run for that topic? And then it can be ignored for any future runs.
Another thought I have is that with the increased ability for LLM “reasoning” + large context windows would it be better to allow for structured outputs in the Discourse Automation window. IFTTT logic could then just be applied to a single automation instead of chaining multiple automations together. Imagine if there were the ability to have one automation, but many “Search for text” actions.
I have been thinking about how to solve this within our current system and one very appealing option is allow for a new automation type:
triage_using_custom_tool
We already have the custom tool system:
We can then allow it a few more functions such as llm.generate and topic.close, topic.tag and so on which could be used by the tool to perform these types of workflows.
Another advantage this has is that you could even test it which would make it easier to tune it.
That sounds like a great idea. I’m still newish to the Discourse ecosystem, so I’ll dig into Custom Tools and also how feature requests make their way into production.
I do have some good news, your workflow is all workable now using custom tools!
The idea is that you would define a single custom tool with all the params
is_spam, intent, hot, requires_invite
Then you would like on triage using persona to call the tool and the tool would perform all the actions (at the moment via Discourse API but we can expose more built-ins as we go)
A good primer on how this stuff can all be glued together is:
I also encountered the problem of not being able to orchestrate the execution of an Automation scripts with the spam detector.
The great news is that Discourse has now created a “Workflows” core plugin that facilitates the same types of applications as the “Automation” plugin, but in a far more powerful and flexible manner:
The orchestration requested here can be achieved if you use the the Workflows plugin to implement the systems that would have previously been implemented via the Automation plugin.
The purpose of my Automation was to raise flags on posts with certain characteristics that might indicate spam, but not definitively. The problem I encountered is that these same characteristics are often present in the spam posts flagged by the Discourse AI Spam detector system. This resulted in the Automation frequently raising redundant flags, creating pointless additional work for the moderators.
The “mash everything into a single Automation” workaround presented above was not applicable to my system because I intentionally separated the two systems:
Since the Discourse AI Spam detector has a special implementation in the Discourse framework (as opposed to purely operating on the Discourse AI framework provided to the forum admin), I did not have any interest in trying to replace it with an Automation.
Furthermore, it was not appropriate to merge my supplemental system into the spam detector (by adding instructions to the prompt). The spam system’s behavior of immediately hiding the post and silencing the author is appropriate as long as it is configured to operate with a high degree of accuracy. Conversely, my supplemental system is inherently prone to false positives, and thus its flags must not affect the subject user prior to human review.
The solution I was able to achieve after replacing the Automation with a Workflow:
