Discourse AI: structured output response was not valid JSON

Support
1

I updated yesterday[1] and now there are a lot of errors in the logs:

Discourse AI: structured output response was not valid JSON, falling back to best-effort parsing (311 bytes)

I inspected the HTTP response and it had no issue. Then I tried the text with rails console:

...
discourse(prod)> p = JsonCompleter.new
=>
#<JsonCompleter:0x00007f48763b0270
...
discourse(prod)> p.parse text
=>
{"spam" => false,
 "reason" =>
  "..."}
discourse(prod)> p.parse DiscourseAi::Utils::BestEffortJsonParser.escape_control_characters(text)
(discourse):14:in '<main>': unexpected token "\\" (JsonCompleter::ParseError)

it turns out that Discourse escaped line feed characters before parsing[2], causing the error.

  1. ↩︎

  2. code, introduced by pr#41716 ↩︎

2

That is a warning, not an error.

That said, I’m moving that to INFO log level, as people keep getting worried about these :face_exhaling:

3

Well, technically true, but it indicates a bug in the code. The fallback works so no error is raised, but it is better to fix the source instead of moving it to INFO level to cover up, isn’t it?

JSON allows line feed charaters as whitespace but doesn’t allow escape sequences outside strings; the escape function however escapes it. To fix it, you can change to escape only ASCII characters 0x0 to 0x1f inclusive according to the JSON spec. (I don’t know how to write it in Ruby so no PR.)

4

To illustrate, the following is valid JSON:

{
  "key": "value"
}

The following is an escaped one, which is invalid and causing the issue:

{\u000a  "key": "value"\u000a}