This guide covers the LLM settings page which is part of the Discourse AI plugin.
Required user level: Administrator
The dedicated settings page is designed to have everything related to Large Language Models (LLMs) used for Discourse AI features in one place.
Depending on the Discourse AI feature enabled, an LLM might be needed. Please check each Discourse AI feature to know if an LLM is a pre-requisite.
Features
- Add new models, with prepopulated information
- Add custom models not mentioned
- Configure LLM settings
- Allow specific LLM use for AI Bot
- See the AI Bot username
- Enable vision support (model dependent)
- Test connections
- Save settings
- Delete connections
Adding LLM connections
- Go to
Admin
βPlugins
βAI
- Go to the
LLMs
tab - Add a new connection, pick your model
- Add in the API key (depending on the model, you might have more fields to input manually) and save
- (Optional) Test your connection to make sure itβs working
Supported LLMs
You can always add a custom option if you donβt see your model listed. Supported models are continually added.
- GPT-4o
- GPT-4o mini
- OpenAI o1 Preview
- OpenAI o1 mini Preview
- Claude Sonnet 3.5
- Claude Haiku 3.5
- Gemini Pro 1.5
- Gemini Flash 1.5
- Llama 3.1
- Qwen 2.5 Coder
Additionally, customers can use the following pre-configured LLMs in the settings page. These are open-weights LLMs hosted by Discourse, ready for use to power AI features.
- CDCK Hosted Large LLM
- CDCK Hosted Small LLM
- CDCK Hosted Vision LLM
Configurations fields
You will only see the fields relevant to your selected LLM provider
Name to display
Model name
Service hosting the model
URL of the service hosting the model
API Key of the service hosting the model
AWS Bedrock Access key ID
AWS Bedrock Region
Optional OpenAI Organization ID
Tokenizer
Number of tokens for the prompt
Technical FAQ
What is tokenizer?
- The tokenizer translates strings into tokens, which is what a model uses to understand the input.
What number should I use for Number of tokens for the prompt
?
- A good rule of thumb is 50% of the model context window, which is the sum of how many tokens you send and how many tokens they generate. If the prompt gets too big, the request will fail. That number is used to trim the prompt and prevent that from happening
Caveats
- Sometimes you may not see the model you wanted to use listed. While you can add them manually, we will support popular models as they come out.
Last edited by @Saif 2024-12-05T00:08:31Z
Check document
Perform check on document: