What LLM to use for Discourse AI?

Documentation Site Management
,
1

It’s important to understand the needs of you as the community admin and your members when choosing a Large Language Model (LLM) to power Discourse AI features.

Several factors may influence your decisions.

  1. Performance for use-case. Are you looking for the best-performing model? Performance can vary depending on task, the assessment is based on the model’s ability to generate correct and relevant responses.

  2. Context length. The context window is the amount of text a model can “see” and consider at one time when processing information or generating responses. Bigger context windows allow for processing more inputs and maintaining coherence over longer conversations.

  3. Compatibility. Is the model supported out of the box? Will it require additional configuration to get it working?

  4. Language support. What if your community speaks in Bengali? While most LLMs support English out of the box, certain models may have better language support.

  5. Multimodal models. While some models may be limited to text, certain models can also support images, audio, video, and other inputs.

  6. Speed. Bigger and better-performing models may be slower and not appropriate for certain tasks where lighter-weight models can excel.

  7. Cost. Budget can be a primary factor in choosing the right LLM. Better-performing LLMs generally tend to be more expensive as they consume more tokens. Though LLMs are getting better and cheaper over time.

  8. Privacy concerns. Different providers will have varying answers to how they handle your data and whether they train on it.

  9. Open vs closed source. Considerations around performance, cost, and privacy would be leading factors in choosing a specific model.

As of writing this guide, here are the current recommendations for choosing which LLMs to power Discourse AI features

:warning: These are general guidelines that can be customized. LLMs get updated continually so it’s always best to check directly with the LLM provider for the latest changes.

An alternative option is hosted customers can use the pre-configured open-weight LLMs hosted by Discourse. They can be enabled via the admin What's New experiments.

Last edited by @sam 2025-01-24T03:29:07Z

Check documentPerform check on document:
5 Likes