I have been thinking about how to solve this within our current system and one very appealing option is allow for a new automation type:
triage_using_custom_tool
We already have the custom tool system:
We can then allow it a few more functions such as llm.generate and topic.close, topic.tag and so on which could be used by the tool to perform these types of workflows.
Another advantage this has is that you could even test it which would make it easier to tune it.