Translation silently truncated when JSON stream parsing breaks (no error raised)

Contribute Bug
,
1

Summary
DiscourseAi::Translation::PostLocalizer (content localization) occasionally
produces truncated translations with no error logged anywhere (Rails log,
Sidekiq log, Sidekiq dead/retry queues). The truncation happens mid-sentence,
silently, and the partial result is saved as if it were a complete, successful
translation.

Observed on: Discourse core + discourse-ai 46bea6613 (2026-07-09), LLM:
Gemini 2.5 Flash via Google provider.

Reproduction evidence
Two real cases on our instance:

  • A ~2090-char German post translated to Spanish produced only 50 chars,
    cutting off immediately before a quoted word (“aufwendiger”) in the source
    text.
  • A ~1830-char German post translated to English (83 chars) and Slovak
    (97 chars) — both cut off at the exact same point in the sentence,
    immediately before an emoji shortcode (:collision:) in the source text.
    All other locales (fr, it, pt, ru, he) for the same post translated
    completely (1600–1950 chars).

Root cause (as far as I can trace it)
In lib/completions/structured_output.rb, StructuredOutput#read_buffered_property
falls back to DiscourseAi::Utils::BestEffortJsonParser.extract_key whenever
@partial_json_tracker.broken? is true. This fallback returns whatever was
parsed before the JSON stream broke — with no exception, no warning, no
indication anywhere that the output is incomplete. In
lib/translation/base_translator.rb#get_translation, this value is used as-is
as the final translation and persisted via PostLocalization#save!.

Both truncation points in our data land immediately before a “special”
character in the source text (a straight quote, an emoji shortcode),
suggesting the JSON streaming tracker becomes confused by certain
characters/sequences in the model’s streamed output and marks the stream
broken? prematurely.

Impact
Content is published with visibly incomplete, sometimes nonsensical
translations, and there’s no operational signal (log, dead job, admin
notification) to catch it. We only discovered it because a community member
spotted mid-sentence garbage in the UI.

Suggested fixes

  • Surface a warning/error (Rails.logger + Discourse-error-log) whenever
    StructuredOutput#broken? is true and the best-effort fallback is used, so
    admins can detect this instead of silently shipping partial content.
  • Consider validating translated length against a sanity threshold relative
    to source length before saving a PostLocalization.

Happy to provide full raw/translated text pairs privately if useful for
debugging the JSON tracker.

2 Likes
2

Adding a data point to this — we’re seeing what looks like the same underlying
mechanism, but manifesting differently.

Observed on: Discourse core + discourse-ai, LLM: GPT-5.1 via OpenAI (custom API connection, not a seeded model).

Reproduction evidence

Same source post (Russian, ~2000 chars, headers/bold/bullet lists/links),
translated into multiple locales via native AI translation:

  • Polish (pl): the embedded image markdown lost its structure — went from
    ![alt text|690x460](upload://...) to a malformed link missing the !
    prefix and the | separator, with the alt text and dimensions run
    together as plain text. Rendered as a clickable link instead of an
    embedded image.
  • Ukrainian (uk): literal \n\n sequences appear as plain text throughout
    the entire post, in place of paragraph breaks. The image markdown in this
    same translation was intact — so the corruption isn’t tied to one fixed
    symptom, it varies by run.

Unlike the truncation described above, our case shows no missing content —
the full text is present, but with corrupted escape sequences / markdown
syntax rather than a shortened output. Different provider (OpenAI vs Google
here), different symptom (corruption vs truncation), same suspected root:
StructuredOutput#read_buffered_property falling back to
BestEffortJsonParser#extract_key, which doesn’t unescape JSON string
sequences (\n stays literal) and appears to also mishandle special
characters adjacent to markdown syntax when the fallback triggers mid-parse.

No errors in Sidekiq or Rails logs on our side either — same silent-failure
behavior.