Content Localization - Manual and Automatic with Discourse AI

In this topic, we will walk you through the Content Localization features and how to enable them. The features are split into two parts: What is available by default in Discourse; and Discourse AI for automatic translations.

:warning: For quick access to the relevant sections, use the wiki headings :backhand_index_pointing_right:t2:

Localizing Your Community’s Content

Discourse gives you access to several localization features available for configuration at:

  • <your-site-url>/admin/site_settings/category/content_localization
Content Localization in Site Settings 📸

Preparation

Firstly, it is good to get some information on your community via data explorer.

Locale via user preferences

The following query can give you an idea of how many users may have set their locale in /my/preferences/interface.

SELECT locale, count(*) as count
FROM users
WHERE (locale IS NOT null AND locale <> '')
GROUP BY locale
ORDER BY count DESC
Sample results from Data Explorer

Where your users are coming from

A better solution, if you have Browser pageview events enabled (site setting), is this query that tells you which country your user is from:

SELECT
  e.country_code AS country_code,
  COUNT(*) AS pageviews
FROM browser_pageview_events e
WHERE e.country_code IS NOT NULL
  AND e.country_code <> ''
GROUP BY e.country_code
ORDER BY pageviews DESC
Sample results

Setup

With the information above, you are now more informed about which locales your community should support.

In <your-site-url>/admin/site_settings/category/content_localization, start setting this up.

  • Content localization enabled - turns on the feature that replaces original written user content with localized content. Read on for auto (AI) and manual (human) modes of localizing.
  • Content localization supported locales - the list of languages your site supports
  • Content localization crawler param - covered in the crawlers section below
  • Content localization language switcher - covered just below
List of locales in Site Settings 📸

Language switcher

Enabling the subsequent setting Content localization language switcher also allows you to make your community more accessible to non-logged-in users by showing the list of languages you’ve chosen in the list of supported locales:

Language switcher at the top right of the page

Viewing localized content

For viewers of localized content (all site visitors), they may cursor over the indicator next to the post’s date to view the original language of the post. This indicator only shows up if the post is not in their language. They may simply tap on the indicator to see the original post as well.

For our multilingual logged-in viewers, they may indicate that they understand multiple languages, so content in those languages won’t be translated.

Automatic translations with Discourse AI :sparkles:

Discourse AI is highly essential for the localization feature, and takes away the need to do manual translations.

As an admin, you’d want to head to our dedicated area in Discourse AI for Translations. This dedicated area covers:

  • translation progress
  • AI settings
  • app language settings

To cover some important settings and recommendations:

  • AI translation backfill hourly rate - this setting defaults to 50 - your site will translate 50 posts, 50 topics, 50 categories, 50 tags per hour, to the locales you have set in Content localization supported locales. Keep this to a low number when starting out.
  • AI translation category scope - defaults to Public categories. You may change this to all categories, or a specific selection.
  • AI translation personal messages and AI translation include bot content are self-explanatory.
  • AI translation backfill start date - this value is empty by default. If you enable the AI translation feature “today”, the date is automatically set to “today - 5 days”
  • AI translation post raw translator agent (and other agents) - In more formal communities, admins may choose to create their own agent. This allows you to set a prompt that is more fine-tuned to the language or vocabulary you prefer.

You can refer to AI bot - Agents on how to configure suitable agents and fine-tune prompts for each function.

Manual localization

As localization is a core feature in Discourse, we provide the ability for you to fill in and edit localizations manually in the event automatic translations with Discourse AI is not available.

By default, admins and moderators are set up to edit localizations.

Currently, we have post content, topic title, category name, category description, tag name, tag description, and sidebar localizations. Subsequent sections below will show you how they work.

Category localization

Localized categories are visible in the following areas, with both category name and description localized:

Places where categories are localized 📸
  1. Homepage, sidebar, and category dropdown
  1. Categories page
  1. A specific category with subcategories

As an admin, you should be able to access category settings as usual, and find the new “Localizations” nav item on the left.

Editing category localizations in Category Settings 📸

Topic and Post localization

From the screenshots above in Category localization, you may have noticed topic titles and excerpts being localized.

There are some pre-requisite settings

  • Ensure your user is in content localization allowed groups
  • The addTranslation button is automatically added to post menu and post menu hidden items when Content localization enabled is turned on. This allows the :globe_with_meridians: to show up in the post menu for users in content localization allowed groups.
  • Content localization allow author localization is enabled by default, and allows post authors to localize their own content using the same post menu as above.
post menu setting

Editing a localized post

In the event the user might be viewing a localized post, and wants to edit the post, a dialog will appear to ask which version they would prefer to edit:

The appropriate composer will appear after deciding.

Custom sidebar localization

Sidebar sections that are “visible to everyone” also show additional translations controls:

Crawlers

You can allow crawlers to see your site in the various languages you’ve configured in Content localization supported locales. The site setting resides in “Content localization” section as Content localization crawler param:

The result would be that crawlers can see the following, where each supported language has a corresponding hreflang in each topic’s head:

For extra proof / verification

We’ve tested this feature extensively, and can confirm that we are getting indexed properly after introducing X numbers of supported languages

FAQ

I’ve set things up, but automatic translation is still not working for me
Confirm if you’ve these set up

  • Content localization supported locales has at least one language
  • Content localization enabled is :check_mark:
    • Allow user locale is :check_mark: (enabled by default)
    • Set locale from cookie is :check_mark:
  • Ai translation enabled is :check_mark:
  • You must have a working LLM set for each translation agent

If all else fails, you can enable SiteSetting.ai_translation_verbose_logs.

Are the automatic translations saved, or is it being sent to the LLM each time someone views a topic?
The translations are saved, each post is only sent once per language and the translations are reused.

If my forum supports English and Japanese (via Content localization supported locales), and someone writes in Spanish, will their post be translated?
Yes. All topics and posts will be translated to English and Japanese, regardless of the written language.

If the original post is edited, is it re-translated?
Yes – with a maximum of 2 times per day. When a post is edited, it gets sent to re-translation after a delay of the greater of 5 minutes or the SiteSetting.editing_grace_period to account for ninja edits. Authorized users in Content localization allowed groups have the option to send a post to re-translate immediately.

Will translations be deleted if I change the Agent or LLM?
No, translations will typically persist across settings changes unless explicitly deleted using the post menu item or the translation composer.


25 Likes
Content Localization and Automatic Translations for Your Community
I can't find discourse AI as a provider in the discourse translator plugin
2 language site
Machine-translate messages so users can read without knowing sender's language
Übersetzung wie hier auf Meta
Anonymous user language selector for Discourse
Update or Replace Header Locale Selector for New Discourse Header (widgets-end-of-life) – Is Built-in Support Available?
Translate Discourse automatically (without a button)
Help which plugin is in the screenshot
About auto translation
Auto Country Translation
How to Add Automatic Multilingual Translation Interface in Discourse?
3.6.0.beta2: Built-in palette editing, live AI translation progress, and better wiki tracking
The usage problem after using AI translation
AI translation backfill not working after all settings configured
AI translation backfill not working after all settings configured
Single post/header only partially translated -issue
Language Switcher Problem, Not Translating Posts
配置好 ai 翻译后,帖子已经被自动翻译,但是在帖子顶部和右侧没有显示语言切换按钮
Feature Request: Allow Users to Localize Only Their Own Topics
How to configure ai translate of `Ai translation backfill hourly rate`?
Seeking experience: Supporting a multilingual Discourse community
Ongoing translation in threads
I need help to configure LLM DeepSeek R1 0528 - free for Automatic translations with Discourse AI
I need help to configure LLM DeepSeek R1 0528 - free for Automatic translations with Discourse AI
Not sure how to juggle languages while setting up my francophone community
Structuring a multilingual community
Why is Discourse AI translation not working
How to set up Discourse AI for internal-only usage
AI translation of all PMs
Content localization language switcher doesn't work
This Meta Branded theme is special. Can I use it on my forum too?
Feature Request: Translation of FAQ/Terms of Service/Privacy
Single post/header only partially translated -issue
Header Locale Selector
为啥的布署的discourse不支持修改语言
Topic list is shown in a language (DE) but has one (DE) topic not translated
If topic info is too long, tags are cut off and create a blank line
Content Localization and Automatic Translations for Your Community
Images break when the page is auto-translated by a browser translation extension (S3/R2 + CDN is configured correctly)
Bearbeitung von Übersetzen Beiträgen verbessern
Why Is the Discourse AI Translation Icon Not Appearing?
Why are posts in Swedish un-translated to Finnish, default locale?
Discourse AI Translator, Planet Icon Missing
Language switcher and language content
Multilingual user feedback on Automatic Translations
Sharing approaches to using automation to send messages to members

Are there any recommendations for doing this in bulk for existing categories? Worst case, perhaps via API?

2 Likes

Hmm great question. I’ll see to it that API docs get updated for the category update endpoint. :memo:

4 Likes

Will there be a support for per-language moderator (translator)? I am thinking about meta – where I might be volunteering for checking the posts in a specific language and manually update them. Especially documentation which may welcome some human touches. But you say only moderators can do it which I probably never be.

3 Likes

Hmm good suggestion. I think that can be done but we’ll need to think about the details on how it can be set up.

We currently have the following, but let me check if it can be extended to a group called “localization moderators”.

3 Likes

How to access it? Could you provide a command please?

Does Sidekiq have any job linked? Is it possible to trigger it manually?

1 Like
2 Likes

To add on to Moin’s post above, it’s just SiteSetting. ai_translation_backfill_hourly_rate once you get to the console. The job runs every five minutes and rate limit accordingly.

3 Likes

I see localization is now available in the docs. Thanks @nat!

3 Likes

That’s wonderful, kudos to the team! I’m testing it right now and I will share my thoughts and overall experience.

*Moving from Discourse Translator plugin we miss Esperanto.

Can this 'simply‘ be added, or does it need to be built into discourse-languages first?*

Wow you’re on the ball – I was just about to report here. :laughing:

Yes, kind of. We want a full localized experience where the controls (buttons, labels, etc) are translated properly and sufficiently (70% would be really good) via Crowdin (see Translations - Discourse Meta), and with that we can provide support to the language.

2 Likes

Does the content localization work with Documentation Categories? It seems to me that sidebar content is not getting translated even though I localize the index topic.

Also I noticed a strange behavior. When I see a localized topic in the original language and I refresh, it switches to the localized version. I have to switch again to the original version manually.

3 Likes

Oh fantastic catch, yeah it is not working yet, but something @nat will track!

I wonder if this is a catalyst to coming up with a better abstraction / data model for the sidebar doc links.

2 Likes

Yes that’s right – there are many places in Discourse that will need explicit translation, so I’m logging them as and when. Most recently, we localized notifications for topic titles as well. This is an example of a feature topic I created - Show translated user bios.

I’ll create a new topic and @ you so we make sure we cover all the bases in sidebar.

EDIT: @tvavrda covered here - Translate sidebar documentation links. Please have a look and see if it makes sense.

What do you mean by “switch again”?

Do you mind sharing a video recording (include the address bar) next time it happens? :folded_hands:t2: Feel free to DM me for this case if the content is not suitable in public. Also, were you logged in? Technically speaking these things are tracked by cookies so it’s a bit puzzling for me.

1 Like

Sent you a video.

Another observation – I cannot see diffs of the translated content, right? That might be useful if there are updates to it. Not super important but it would make sense, I guess.

And another – the backlinks under the topic don’t show a localized topic name.

And a question: What’s the point of localizing category descriptions in the category settings? The category description should come from the localized version of the “About” topic, shouldn’t it? The localized version does not support markdown, so I cannot use a link, which I would like to.

Other than that it works beautifully! :heart: :heart: :heart:

1 Like

Well… the old GitHub - discourse/discourse-docs-sidebar · GitHub component actually respects the localization :slight_smile: I temporarily switched to that one.

Yeah, this is currently not supported as well and would be quite an endeavour.

We have a little special-coloured indicator (similar to post edits indicator next to it) when a translation may be outdated as the post version has changed.

1 Like

I also see an untranslated content on pinned topic’s abstracts. So I see a topic list in the translated language but the abstract of the pined topic shows original.

1 Like

We can manually do or fix translations, but can we manually trigger building of translations? Kind of on-demand job.

What I am thinking is I’ve allowed translation for a year old topics. But if that year is counted from current date, that limit will move all the time toward already translated content. But the biggest demand is older valuable content that I would like to reach fast and without quite slow bulk actions.

I am curious, does anyone have any cost figures after turning on the translations? Our site has been around for a while, and whilst I’d like to translate the whole site if possible, cost is definitely a concern. so if someone has some rough idea of cost from your experience, e.g. 1000 posts resulting in a $1 cost, that’d help tremendously to get a ballpark cost idea.

Is the content localization done once then stored somewhere, i.e. not on demand? if so, is there anything that prevents me from firing up Ollama and some open source LLM on my desktop like Llama 3 or Deepseek 3 and just let the job run until its done?

Edit: I guess it may work to lower the initial translation cost, but it won’t work for newer posts unless one decides to keep the local LLM running perpetually.