The usage problem after using AI translation

[co_choa]

I followed the official tutorial to configure Discourse AI for translation and set it to translate all past posts over a span of days. Indeed, this resulted in a substantial amount of input and output tokens. However, after two days, I encountered a situation where only input tokens were being processed without any output. I am unsure of the cause—could it be that all the previous posts have already been translated? If so, what measures can I take to reduce token input and thereby conserve costs?

[nat (Natalie T)]

Hey there, have you followed these recommendations?

@nat

To cover some important settings and recommendations:

• AI translation model - We highly recommend setting a quota to the LLM you choose for the translation model.
• AI translation backfill hourly rate - this setting is hidden in the UI and defaults to 0. Automatic translation will not begin if this value is 0. Assuming the rate is 50, your site will translate 50 posts, 50 topics, and 50 categories per hour, to the locales you have set in Content localization supported locales. Keep this to a low number when starting out.
• AI translation backfill max age days - defaults to 5. This means topics and posts older than 5 days will not be translated. You may increase this to a large number to translate all topics and posts.

The usage graph definitely looks concerning. Can you try out this data explorer query:

SELECT 
  a.id,
  a.language_model,
  LENGTH(p.raw) as raw_length,
  a.response_tokens,
  a.raw_request_payload,
  a.raw_response_payload,
  a.topic_id,
  a.post_id
FROM ai_api_audit_logs a
LEFT JOIN posts p ON p.id = a.post_id AND p.deleted_at IS NULL
LEFT JOIN topics t ON t.id = a.topic_id AND t.deleted_at IS NULL
WHERE a.created_at > CURRENT_DATE - INTERVAL '1 days'
AND p.deleted_at IS NULL
AND t.deleted_at IS NULL
AND p.user_deleted = false
AND a.feature_name = 'translation'
AND LENGTH(p.raw) < 1000
AND a.response_tokens > 10000
ORDER BY a.created_at DESC
LIMIT 100

The query should show you the number of response tokens used based on the post’s raw length. Ideally you should see a similar number, not more than 1.5x tokens. The AiApiAuditLog will help with determining what is going on.

Additionally please share,

• What model are you using?
• What’s your backfill hourly rate? I suggest to keep it to a low value, like 50 for starters.
• How many languages are you supporting? Does your selected model support them?

[co_choa]

I have configured GPT-4.1 Nano as the translation model. The hourly backfill rate was previously set at 1,000, but today I adjusted it to 100. I have enabled support for both Japanese and English, and this model indeed supports these languages.

[nat (Natalie T)]

Yeah, 1000 is probably not a good idea and I should add a site setting limit here.

I am not sure how the OpenAI API deals with getting hit about 3000x per hour. Basically for your setup, per post we are doing locale detection, translate to Japanese, translate to English. If you check /logs I suspect you’ll see your site hitting rate limits.

I suggest lowering it still to 50 and see how it goes.

We’ll be implementing a way to view translation progress of the entire site as well in the near future.

[co_choa]

Furthermore, when I execute the query command you provided, the database returns no results. Is there perhaps a need for some customization or modification?

[nat (Natalie T)]

Hmm this query should work. Do you have the discourse-data-explorer plugin?

[co_choa]

Very well, I shall give it a try first. Thank you.

[co_choa]

I will install it afterward and then give it another try. Currently, rebuilding the forum is not feasible because the users are still actively using it.