Discourse AI - Large Language Model (LLM) settings page

Documentation Site Management
,
1

:bookmark: This guide covers the LLM settings page which is part of the Discourse AI plugin.

:person_raising_hand: Required user level: Administrator

The dedicated settings page is designed to have everything related to Large Language Models (LLMs) used for Discourse AI features in one place.

:raised_hand_with_fingers_splayed: Depending on the Discourse AI feature enabled, an LLM might be needed. Please check each Discourse AI feature to know if an LLM is a pre-requisite

CleanShot 2024-08-02 at 15.06.50@2x
CleanShot 2024-08-02 at 15.06.50@2x2532×1686 284 KB

CleanShot 2024-08-02 at 15.07.09@2x
CleanShot 2024-08-02 at 15.07.09@2x1550×1592 176 KB

Features

  • Add new models, with prepopulated information
  • Add custom models not mentioned
  • Configure LLM settings
  • Allow specific LLM use for AI Bot
    • See the AI Bot username
  • Enable vision support (model dependent)
  • Test connections
  • Save settings
  • Delete connections

Availability

:discourse: Hosted by us? Currently, this settings page is available for anyone on our hosting as long as you have the Discourse AI plugin installed

:information_source: Self-hosted users can install the plugin anytime by following Install Plugins in Discourse - sysadmin - Discourse Meta

Adding LLM connections

  1. Go to Admin → Plugins → AI

  2. Go to the LLMs tab

  3. Add a new connection, pick your model

  4. Add in the API key (depending on the model, you might have more fields to input manually) and save

  5. (Optional) Test your connection to make sure it’s working

Supported LLMs

:person_tipping_hand: You can always add a custom option if you don’t see your model listed

  • Claude 3.5 Sonnet
  • Claude 3 Opus
  • Claude 3 Sonnet
  • Claude 3 Haiku
  • Gemini 1.5 Pro
  • Gemini 1.5 Flash
  • GPT-4 Omni
  • GPT-4 Omni Mini
  • GPT-4 Turbo

Configurations fields

:information_source: You will only see the fields relevant to your selected LLM provider

  • Name to display
  • Model name
  • Service hosting the model
  • URL of the service hosting the model
  • API Key of the service hosting the model
  • AWS Bedrock Access key ID
  • AWS Bedrock Region
  • Optional OpenAI Organization ID
  • Tokenizer
  • Number of tokens for the prompt

Technical FAQ

  • What is tokenizer?
    • The tokenizer translates strings into tokens, which is what a model uses to understand the input.
  • What number should I use for Number of tokens for the prompt ?
    • A good rule of thumb is 50% of the model context window, which is the sum of how many tokens you send and how many tokens they generate. If the prompt gets too big, the request will fail. That number is used to trim the prompt and prevent that from happening

Caveats

  • Sometimes you may not see the model you wanted to use listed. While you can add them manually, we will support popular models as they come out.
  • In the future, we plan to revamp the page so its more user-friendly and intuitive

Last edited by @hugh 2024-08-14T05:45:58Z

Check documentPerform check on document:
5 Likes