I think discourse-ai API needs a regression

Dev
1

Discourse AI now is using structured output for content, but structured output has not been supported by a lot of API providers, almost all of the AI providers in my country like DeepSeek or Qwen do not do that. The use of structured output leads to very limited API choices ranging only a few main AI providers.

I think, if Discourse AI has to break down the current limitation in provider choices, giving up structured output is essential, and nothing will be actually broken without it. For a solution, we can use some simple unique separators or just asking AI to generate JSON with giving JSON Schema or examples, but do not using structured output, why do not?

1 Like
2

Can you try Moonshot AI Kimi K2 via their API endpoints for China?

Looks like the base API URL is https://api.moonshot.cn/v1.

We embarked the structured outputs requirement because I had endured thousands of complaints of about issues such as, models adding “Here is your summary” or failures in parsing when getting a list of suggested titles for AI helper, etc. Now with translation, it became an even worse issue when you have to be able to trust millions of LLM invocations to not add anything extra to your output.

Given the need to reliability, and the general availability of structured outputs we made this call on the providers where it is supported, but I’d be open for a PR that adds a checkbox for disabling Structured Outputs from the API request on the OpenAI provider.

1 Like
3

That would be nice, I am not and can not demand Discourse team, so just giving some suggestions.

Using old “separators” solution for robuster parsing, a XML-like raw content like this:

<SUMMARY_START>
Summary contnet.
<SUMMARY_END>

<SUGGESTIONS_START>
<SUGGESTION_START>
Suggestion 1
<SUGGESTION_END>
<SUGGESTION_START>
Suggestion 2
<SUGGESTION_END>
...
<SUGGESTIONS_END>

Then not caring about anything outside of <START> and <END>, I believe that the most foolish productional AI model can process labels correctly.

And asking AI not to add extra discription; then warning admins possible problems if opening that checkbox.

I do not know what Discourse AI generated request body looks like. But if not, just putting system prompts in first message said by role system, vector database query result in the next system role message with a prefix anything like Past important content:\n, and putting post content or other actual data in user role’s messages.