Interesting, can you run this data explorer query?
-- [params]
-- integer :topic_id = 66
SELECT
a.id,
a.created_at,
a.response_tokens,
(REGEXP_MATCH(a.raw_response_payload, '"text": "([^"\\]+)"'))[1] AS llm_detected_locale,
a.raw_response_payload
FROM ai_api_audit_logs a
JOIN posts p ON p.id = a.post_id AND p.deleted_at IS NULL
LEFT JOIN topics t ON t.id = a.topic_id AND t.deleted_at IS NULL
WHERE t.id = :topic_id
AND p.post_number = 1
AND a.feature_name = 'translation'
AND a.response_tokens < 5
This should tell us what the LLM is actually returning.
If we’re seeing an “en” in the llm_detected_locale column, I suspect your site’s prompts for locale detection may need a change, or you can use a more suitable LLM (maybe qwen?)