Translation silently truncated when JSON stream parsing breaks (no error raised)

Falco · July 14, 2026, 11:39pm

Started working on a possible fix at

github.com/discourse/discourse

FIX: Silent truncation and corruption of AI structured output streams (#41716)

main ← fix-structured-output-stream-corruption

merged 03:00PM - 15 Jul 26 UTC

xfalcox

+161 -855

Reported at https://meta.discourse.org/t/translation-silently-truncated-when-jso…n-stream-parsing-breaks-no-error-raised/407251 ## The bug Some providers stream structured output whose string values were unescaped by an outer JSON parse, so real newlines appear inside string values. When that happened, `JsonStreamingTracker` had two failure modes: - It marked the stream broken and `StructuredOutput` fell back to `BestEffortJsonParser`, whose extraction regex (`[^"]+`) cut the value at the first escaped quote and left `\n` sequences as literal text. A 2,000-char translation could come back as ~50 chars, cut right before the first quoted word — exactly what the report shows. - Its escape-and-resume hack (`String#dump` + buffer-growth offset) miscomputed the resume index whenever the chunk contained non-ASCII, quotes, or backslashes, silently duplicating or corrupting content **without ever marking the stream broken**. Testing a realistic corrupted payload across chunk sizes 1–60: 31 produced the truncated fallback, 27 produced silent corruption, 1 raised, and only 1 came out correct. Either way the result was persisted as a successful translation with nothing in the logs. ## The fix Replace the hand-rolled parsing with two gems and keep only glue: - **json_completer** (pure Ruby): `JsonStreamingTracker` now feeds the cumulative buffer — with control characters re-escaped — to an incremental, truncation-tolerant parser and notifies consumers of changed keys. The corrupted payloads above stream correctly at every chunk size, so the broken-stream path is only reached for responses that aren't JSON at all. - **smarter_json**: `BestEffortJsonParser` becomes a three-attempt chain (strict-with-completion → control-chars re-escaped → lenient) covering the quirk shapes the old regexes handled: single quotes, unquoted keys, markdown fences, prose-wrapped JSON. This deletes the vendored 668-line SAX parser, the resume hack, and all manual regex extraction (net −694 lines), and adds a log warning whenever a response falls back to best-effort parsing. ## Behavior changes - Scalars now stream progressively: mid-stream `read_buffered_property(:number)` returns the digits buffered so far instead of `nil`. Consumers act on final values, so this only affects mid-stream reads. - Arrays of objects stream partial objects mid-stream instead of returning `nil` until finish. - A trailing comma in an array reads as a `nil` placeholder slot until the next element arrives. - Partial tool calls surface a few more progressive updates (the openai endpoint spec count moved 128 → 134); values still only ever grow. ## Tests - Regression specs for the report: unescaped control characters with escaped quotes/emoji streamed across chunk boundaries, fenced + unescaped responses, truncated JSON, numeric casting. - 916 examples green across `completions/`, `translation/`, `modules/ai_helper/`, and `utils/`.

Topic		Replies	Views
Discourse AI: a Gemini error event inside a streamed reply is ignored, and the partial text is stored as a complete translation Bug ai	1	115	October 5, 2026
Discourse AI: structured output response was not valid JSON Product Support ai	9	314	August 5, 2026
AI translations errors Bug ai	10	476	May 10, 2026
Error when translate post by ai （Job exception: undefined method `strip' for an instance of DiscourseAi::Completions::StructuredOutput Product Support ai , dynaloc	5	304	November 30, 2025
Discourse AI: Gemini streaming decoder drops the rest of a reply when a network chunk ends inside the blank line between two events Bug ai	1	83	October 6, 2026

Translation silently truncated when JSON stream parsing breaks (no error raised)

Related topics