Discourse Chatbot 🤖

[merefield (Robert)]

	Summary	A cloud chatbot adaptor for Discourse, currently supporting OpenAI
	Repository Link	GitHub - merefield/discourse-chatbot: An AI bot with RAG capability for Topics and Chat in Discourse, currently powered by OpenAI
	Install Guide	How to install plugins in Discourse

Enjoying this plugin? Please it on GitHub !

What is it?

• The original Discourse AI Chatbot!
• You can use this bot for some Customer Support information tasks (see this guide Building a technical support chatbot)
• Converse with the bot in any Topic or Chat Channel, one to one or with others!
• Customise the character of your bot to suit your forum!
  • want it to sound like William Shakespeare, or Winston Churchill? can do!
• The “RAG Mode” bot can now:
  • Search your forum** for answers so the bot can be an expert on the subject of your forum.
    • Rerank the searches to favour particular groups of authors or Topic tags
    • not just be aware of the information on the current Topic or Channel.
  • Render DALL-E AI pictures
  • Search Wikipedia
  • Search current news*
  • Search Google*
  • Crawl remotes sites*
  • Return current End Of Day market data for stocks.*
  • Do “complex” maths accurately (with no made up or “hallucinated” answers!)
• Vision support - the bot can see your pictures and answer questions on them!
• Automatically respond to new Topics in specific Categories
• Uses cutting edge Open AI API and functions capability of their excellent, industry leading Large Language Models.
• Includes a special quota system to manage access to the bot: more trusted and/or paying members can have greater access to the bot!
• Also supports Azure and proxy server connections.
  • Use third party proxy processes to translate the calls to support alternative LLMs like Gemini e.g. this one

^{*sign-up for external (not affiliated) API services required. Links in settings.}

RAG mode is very smart and knows facts posted on your forum:

Basic bot mode can sometimes make mistakes, but is cheaper to run because it makes fewer calls to the Large Language Model:

(Sorry China! )

**Bot’s “vision” - what it can see (potentially share) and privacy

This bot can be used in public spaces on your forum. To make the bot especially useful there is RAG mode (one setting per bot trust level). This is not set by default.

In this mode the bot is, by default, privy to all content a Trust Level 1 user would see, working from this setting:

Thus, if interacted with in a public facing Topic, there is a possibility the bot could “leak” information if you tend to gate content at the Trust Level 0 or 1 level via Category permissions. This level was chosen because through experience most sites usually do not gate sensitive content at low trust levels but it depends on your specific needs.

For this mode, make sure you have at least one user with Trust Level 1 and no additional group membership beyond the automated groups. (bear in mind the bot will then know everything a TL1 level user would know and can share it). You can choose to lower chatbot embeddings benchmark user trust level if you have a Trust Level 0 user with no additional group membership beyond automated groups.

Alternatively:

• Switch chatbot embeddings strategy to category and populate chatbot embeddings categories with Categories you wish the bot to know about. (Be aware that if you add any private Categories, it should know about those and anything the bot says in public, anywhere might leak to less privileged users so just be a bit careful on what you add).
• only use the bot in normal mode (but the bot then won’t see any posts)
• mitigate with moderation

In addition, note anything it can “see” gets shared with Open AI.

You can see that this setup is a compromise. In order to make the bot useful it needs to be knowledgeable about the content on your site. Currently it is not possible for the bot to selectively read members only content and share that only with members which some admins might find limiting but there is no way to easily solve the that whilst the bot is able to talk in public. Contact me if you have special needs and would like to sponsor some work in this space. Bot permissioning with semantic search is a non-trivial problem. The system is currently optimised for speed. NB Private Messages are never read by the bot.

FYI’s

• May not work on mulit-site installs (not explicitly tested), but PR welcome to improve support
• Open AI API response can be slow at times on more advanced models due to high demand. However Chatbot supports GPT 3.5 too which is fast and responsive and perfectly capable.
• Is extensible and supporting other cloud bots is intended (hence the generic name for the plugin), but currently ‘only’ supports interaction with Open AI Large Language Models (LLM) such as GPT-4 natively. Please contact me if you wish to add additional bot types or want to support me to add more. PR welcome. Can already use proxy servers to access other services without code changes though!
• Is extensible to support the searching of other content beyond just the current set provided.

Setup

Prerequisites

Aside from the normal changes to app.yml you need to be aware of the following:

To build with Chatbot or AI Topic Summary you need at least version 0.5.1 of the pgvector postgres extension.

Most people will have at least this version already. However, occasionally some installs have an older version installed. This will prevent you building with an error similar to: PG::UndefinedObject: ERROR: access method "hnsw" does not exist

First make sure your container is running:

./launcher restart app

then enter your container

./launcher enter app

then go into the database and update the version of pgvector:

:/var/www/discourse# su postgres -c 'psql discourse'
\dx
ALTER EXTENSION vector UPDATE;
\dx
exit

now leave the container with exit

You should now be able to rebuild.

Creating the Embeddings

If you wish Chatbot to know about the content on your site, turn this setting ON:

chatbot_embeddings_enabled

Only necessary if you want to use the RAG type bot and ensure it is aware of the content on your forum, not just the current Topic.

Initially, we need to create the embeddings for all in-scope posts, so the bot can find forum information. This now happens in the background once this setting is enabled and you do not need to do anything.

This seeding job can take a period of days for very big sites.

Embeddings Scope

This is determined by several settings:

• chatbot_embeddings_strategy which can be either “benchmark_user” or “category”
• chatbot_embeddings_benchmark_user_trust_level sets the relevant trust level for the former
• chatbot_embeddings_categories if category strategy is set, gives the bot access to consider all posts in specified Category.

If you change these settings, over time, the population of Embeddings will morph.

To speed population up

Enter the container:

./launcher enter app

and run the following rake command:

rake chatbot:refresh_embeddings[1]

which at present will run twice due to unknown reason (sorry! feel free to PR) but the [1] ensures the second time it will only add missing embeddings (ie none immediately after first run) so is somewhat moot.

In the unlikely event you get rate limited by OpenAI (unlikely!) you can complete the embeddings by doing this:

rake chatbot:refresh_embeddings[1,1]

which will fill in the missing ones (so nothing lost from the error) but will continue more cautiously putting a 1 second delay between each call to Open AI.

Compared to bot interactions, embeddings are not expensive to create, but do watch your usage on your Open AI dashboard in any case.

NB Embeddings are only created for Posts and only those Posts for which a Trust Level One user would have access. This seemed like a reasonable compromise. It will not create embeddings for posts from Trust Level 2+ only accessible content.

Useful Data Explorer query to monitor embeddings population

@37Rb writes: “Here’s a SQL query I’m using with the Data Explorer plugin to monitor & verify embeddings… in case it helps anyone else.”

SELECT e.id, e.post_id AS post, p.topic_id AS topic, p.post_number,
       p.topic_id, e.created_at, e.updated_at, p.deleted_at AS post_deleted
FROM chatbot_post_embeddings e LEFT JOIN posts p ON e.post_id = p.id

Error when you are trying to get an embedding for too many characters.

You might get an error like this:

OpenAI HTTP Error (spotted in ruby-openai 6.3.1): {"error"=>{"message"=>"This model's maximum context length is 8192 tokens, however you requested 8528 tokens (8528 in your prompt; 0 for the completion). Please reduce your prompt; or completion length.", "type"=>"invalid_request_error", "param"=>nil, "code"=>nil}}

This is how you resolve it …

As per your error message, the embedding model has a limit of:

8192 tokens

however you requested 8528

You need to drop the current value of this setting:

chatbot_open_ai_embeddings_char_limit:

by about 4 x the diff and see if it works (a token is roughly 4 characters).

So, in this example, 4 x (8528 - 8192) = 1344

So drop chatbot_open_ai_embeddings_char_limit current value by 1500 to be safe. However, the default value was set according to a lot of testing for English Posts, but for other languages it may need lowering.

This will then cut off more text and request tokens and hopefully the embedding will go through. If not you will need to confirm the difference and reduce it further accordingly. Eventually it will be low enough so you don’t need to look at it again.

How To Switch Embeddings model

You don’t need to do anything but change the setting: the background job will take care of things, if gradually.

If you really want to speed the process up, do:

• Change the setting chatbot_open_ai_embeddings_model to your new preferred model
• It’s best to first delete all your current embeddings:
  • go into the container ./launcher enter app
  • enter the rails console rails c
  • run ::DiscourseChatbot::PostEmbedding.delete_all
  • exit (to return to root within container)
• run rake chatbot:refresh_embeddings[1]
• if for any Open AI side reason that fails part way through, run it again until you get to 100%
• the new model is known to be more accurate, so you might have to drop chatbot_forum_search_function_similarity_threshold or you might get no results :). I dropped my default value from 0.8 to 0.6, but your mileage may vary.

Bot Type

Take a moment to read through the entire set of Plugin settings. The chatbot bot type setting is key, and there is one for each chatbot “Trust Level”:

RAG mode is superior but will make more calls to the API, potentially increasing cost. That said, the reduction in its propensity to ultimately output ‘hallucinations’ may facilitate you being able to drop down from GPT-4 to GPT-3.5 and you may end up spending less despite the significant increase in usefulness and reliability of the output. GPT 3.5 is also a better fit for the Agent type based on response times. A potential win-win! Experiment!

For Chatbot to work in Chat you must have Chat enabled.

Bot’s speed of response

This is governed mostly by a setting: ‎chatbot_reply_job_time_delay‎ over which you have discretion.

The intention of having this setting is to:

• protect you from reaching rate limits of Open AI
• protect your site from users that would like to spam the bot and cost you money.

It is now default ‘1’ second and can now be reduced to zero , but be aware of the above risks.

Setting this zero and the bot, even in ‘agent’ mode, becomes a lot more ‘snappy’.

Obviously this can be a bit artificial and no real person would actually type that fast … but set it to your taste and wallet size.

Ollama & llama3 support

Ollama support for (Completely AWESOME!) llama3 is not shipping:

This is for when the bot is run locally in dev or in the cloud with ollama server … and in Basic mode:

• make sure model is llama3:

• custom URL needs to be set http://localhost:11434:

If you have a big enough server, you could be serving ollama in the cloud there.

Enjoy!

NB I cannot directly control the speed of response of Open AI’s API - and the general rule is the more sophisticated the model you set the slower this response will usually be. So GPT 3.5 is much faster that GPT 4 … although this may change with the newer GPT 4 Turbo model.

OpenAI

You must get a token from https://platform.openai.com/ in order to use the current bot. A default language model is set (one of the most sophisticated), but you can try a cheaper alternative, the list is here

There is an automated part of the setup: upon addition to a Discourse, the plugin currently sets up a AI bot user with the following attributes

• Name: ‘Chatbot’
• User Id: -4
• Bio: “Hi, I’m not a real person. I’m a bot that can discuss things with you. Don’t take me too seriously. Sometimes, I’m even right about stuff!”
• Group Name: “ai_bot_group”
• Group Full Name: “AI Bots”

You can edit the name, avatar and bio (see locale string in admin → customize → text) as you wish but make it easy to mention.

It’s not free, so there’s a quota system, and you have to set this up

Initially no-one will have access to the bot, not even staff.

Calling the Open AI API is not free after an initial free allocation has expired! So, I’ve implemented a quota system to keep this under control, keep costs down and prevent abuse. The cost is not crazy with these small interactions, but it may add up if it gets popular. You can read more about OpenAI pricing on their pricing page.

In order to interact with the bot you must belong to a group that has been added to one of the three levels of trusted sets of groups, low, medium & high trust group sets. You can modify each of the number of allowed interactions per week per trusted group sets in the corresponding settings.

You must populate the groups too. That configuration is entirely up to you. They start out blank so initially no-one will have access to the bot:

In this example I’ve made staff have high trust access, whilst trust_level_0 have low trust. They get the corresponding quotas in three additional settings.

Note the user gets the quota based on the highest trusted group they are a member of.

“Prompt Engineering”

There are several locale text “settings” that influence what the bot receives and how the bot responds.

The most important one you should consider changing is the bot’s system prompt. This is sent every time you speak to the bot.

For the basic bot, you can try a system prompt like:

’You are an extreme Formula One fan, you love everything to do with motorsport and its high octane levels of excitement’ instead of the default.

(For the rag bot you must keep everything after “You are a helpful assistant.” or you may break the agent behaviour. Reset it if you run into problems. Again experiment!)

Try one that is most appropriate for the subject matter of your forum. Be creative!

Note that there are now two system prompts for each bot type. One .open is used when talking to the bot in “public”. The other .private is applied when talking to the bot in Personal Messages or Direct Message chat. This is so that you can customize private behaviour for e.g. a support bot.

Changing these locale strings can make the bot behave very differently but cannot be amended on the fly. I would recommend changing only the system prompt as the others play an important role in agent behaviour or providing information on who said what to the bot.

NB In Topics, the first Post and Topic Title are sent in addition to the window of Posts (determined by the lookback setting) to give the bot more context.

You can edit these strings in Admin → Customize → Text under chatbot.prompt., the most important of which are the system prompts which are in: chatbot.prompt.system.

merefield/discourse-chatbot/blob/7c2fd29e1c3f1091243ca07bebe306849be9de96/config/locales/server.en.yml#L56

Supports both Posts & Chat Messages!

The bot supports Chat Messages and Topic Posts, including Private Messages (if configured).

You can prompt the bot to respond by replying to it, or @ mentioning it. You can set how far the bot looks behind to get context for a response. The bigger the value the more costly will be each call.

There’s a floating quick chat button that connects you immediately to the bot. Its styling is a little experimental (modifying some z-index values of your base forum on mobile) and it may clash on some pages. This can be disabled in settings. You can choose whether to load the bot into a 1 to 1 chat or a Personal Message.

Now you can choose your preferred icon (default ) or if setting left blank, will pick up the bot user’s avatar!

avatar: OR icon:

And remember, you can also customise the text that appears when it is expanded:

… using Admin → Customize → Text

(though you may need to customise the CSS a little to accommodate colours and sizing you want).

Some debugging help

• make sure setting chatbot include inner thoughts in private messages is ON
• make sure setting chatbot enable verbose rails logging is ON
• ssh into your server
• cd /var/discourse/shared/standalone/log/rails
• in parallel ask the bot something in the PM
• go back to console immediately
• look for these kind of messages:
  • general chat = tail -n 2000 production.log | grep {\"model\":\"
  • vision calls = tail -n 2000 production.log | grep {\"type\":\"image_url\"
• check “inner thoughts” in the PM

Uninstalling the plugin - Important!

Due to recent efforts to simplify the plugin, the only steps necessary to uninstall the plugin are now to remove the clone statement.

Thanks for your interest in the plugin!

Disclaimer: I’m not responsible for what the bot responds with. Consider the plugin to be at Beta stage and things could go wrong. It will improve with feedback. But not necessarily the bots response Please understand the pro’s and con’s of a LLM and what they are and aren’t capable of and their limitations. They are very good at creating convincing text but can often be factually wrong.

Important Privacy Note: whatever you write on your forum may get forwarded to Open AI as part of the bots scan of the last few posts once it is prompted to reply (obviously this is restricted to the current Topic or Chat Channel). Whilst it almost certainly won’t be incorporated into their pre-trained models, they will use the data in their analytics and logging. Be sure to add this fact into your forum’s TOS & privacy statements. Related links: https://openai.com/policies/terms-of-use, https://openai.com/policies/privacy-policy, https://platform.openai.com/docs/data-usage-policies

Copyright: Open AI made a statement about Copyright here: https://help.openai.com/en/articles/5008634-will-openai-claim-copyright-over-what-outputs-i-generate-with-the-api

TODO/Roadmap Items

• Add front and back-end tests
• Add “bot typing” indicator and “response streaming” (@Aizada_M, @MarcP)
• forgot to mention the bot? Get bot to respond to edits that add its @ mention (@frold )
• Add a badge? You did mention @botname (@frold )
• Add setting to include Category and Pinned Posts prompt? (@Ed_S)
• Ditto Bios to each message history prompt? (@Ed_S , @codergautam). Will this even work. Let’s get evidence.
• Update Discourse Frotz with this better codebase?
• Move to use pgvector in favour of pgembedding for vector search now that former supports fast HNSW lookup.
• Add semantic search so that the bot can read your forum Posts and become an “expert”
• Add agent behaviour to reduce hallucinations and leverage reliable, factual information.
• Add extra logic to convert suspected usernames into @ mentions (@frold )
• Add GPT-4 support (when Open AI deems me worthy enough of access! )
• Add custom model name support.
• Add option to strip out quotes from Posts before passing text to API.
• Improve error transparency & handling for when Open AI returns an error state
• Add retry capability for timed out API requests
• Add support for ChatGPT
• Lint the plugin to Discourse core standards
• Add CI workflows
• Add settings to influence the nature of the bots response (e.g. how wacky it is).
• include Topic Title & first Posts to prompt
• Add setting to switch from raw Post/Message data to cooked to potentially leverage web training data better (suggestion by @MarcP). NB May cost more and limit what is returned as input tokens are counted and cooked is much bigger. think we’ve abandoned this idea

Credits:

• Thanks to @MarcP for his enthusiastic support and detailed testing feedback.
• The guys who’ve contributed to the emerging standard Ruby API for Open AI which this leverages
• The floating button design is mostly taken from Discourse’s own GitHub - discourse/material-design-stock-theme (so they are not currently compatible, doh!)
• The fixtures code is based on @vinothkannans work here
• If @P16 hadn’t hired me many moons ago to help him with a chat bot this might not exist

^{*It still uses OpenAI’s chat GPT engine, but can now leverage local functions and data from API calls to limit hallucinations.}

[merefield (Robert)]

I added Jina.ai to the list of supported API’s which can be used for web search and web crawling. One key serves both services. Existing API’s are used if you do not remove their keys.

FEATURE: add jina.ai API for web search and crawl

There were two main reasons:

• crawling performance seems really good
• they offer a pay as you go model as opposed to a monthly fee which suits me and may suit you.

Link in settings.

Jina AI - Your Search Foundation, Supercharged.

[Festinger]

Hi everyone,

The AI Chatbot created by @merefield is really wonderful. It’s doing an awesome job in our community. However, I would like to know if it’s possible for it to reply to every post, similar to what the AI Post classifier can do with the default Discourse AI setup.

Discourse AI Post Classifier - Automation rule

The Discourse AI Post Classifier. This feature is designed to enhance the management and moderation of forum posts by automating the process of classifying posts. Please note that this feature requires both the discourse-automation and discourse-ai plugins to function. Use Cases Automated Post Categorization: The post classifier can automatically categorize topics based on their content. This is particularly beneficial for large forums where manual categorization can be time-consuming. You c…

Is this possible?

Thanks!

[merefield (Robert)]

Hey thanks for your interest in Discourse Chatbot.

Two things may be relevant:

• the bot will Reply to every Post if invoked in a Topic with only a single User until more people join the Topic
• Per Category, you can get Chatbot to Reply to every new Topic and you can give it special instructions on what to do, governed by a Category specific prompt that you set up in the Category settings.

the bot will always be invoked if you @ mention it so I don’t know if you can utilise that feature too.

[Festinger]

Thanks for your prompt response @merefield.

@merefield

Two things may be relevant:

Thanks for the explanation. However, our use case is different. We want to trigger the AI chatbot based on various keywords in the assigned category with the custom prompt.

For example - we have a support category where the AI chatbot replies to every new topic. We have a second private support category where the AI chatbot should reply to ‘every’ post without being mentioned. The AI chatbot should be triggered by a keyword.

[merefield (Robert)]

@Festinger

We have a second private support category where the AI chatbot should reply to ‘every’ post without being mentioned. The AI chatbot should be triggered by a keyword.

‘every’ post? Or only those with a keyword? Are those two different requirements?

@sam

Automated Responses: The post classifier can generate replies to posts using predefined responses. This is useful for addressing frequently asked questions or common queries, sending spam to junk categories, and more.

Perhaps you could mention Chatbot in the predefined response and the bot will follow up? That would mean two Posts but might work.

So if your predefined response was “@Chatbot (or whatever you’ve renamed it to) please handle this” then Chatbot would follow up. Obviously you can tailor that response.

Note Chatbot can respond to whispers too (if you configure it so), so if the predefined response was in a whisper then normal users would only see the Chatbot response which might look neater.

I’m open to extending the capability of Chatbot if you wish to hire me to do so, so long as we can come up with a sufficiently generic solution that would be useful to others, but it makes sense to leverage what exists I think?

I wonder if the best action here is to add a Whisper capability to the AI Post Classifier? You could hire someone to PR that or raise it as a feature and hope it gets prioritised. That would undoubtedly be genericly useful?

[merefield (Robert)]

I made some code simplifications by making use of a feature that was added to the Open AI API. Let me know if you encounter any issues.

IMPROVE: use Chat Completions name attribute to simplify code

I decided it was time to call this 1.0.0

[merefield (Robert)]

Added the ability to add other functions in separate plugins, so you don’t have to maintain a fork of the Chatbot repo.

Example function plugin here:

GitHub - merefield/discourse-chatbot-function-extension-example: An example plugin to show you how to add new...

but of course you could add this to any plugin …

PR:

FEATURE: ability to extend with functions from other plugins

[merefield (Robert)]

@Festinger did you solve this?

[ozkn (Mathx)]

Post Embedding: There was a problem, but will retry til limit: undefined method `destroy!’ for nil

After the last update, I started getting the above error. I get this error especially when it comes to images. My vision mode setting is: directly. I wasn’t getting such an error before.

[merefield (Robert)]

Do you recall deleting a Post (or Topic) shortly after creating one?

This error will occur if you create a Post then delete it before it has chance to create an embedding, so when you delete it, Chatbot tries to delete a non-existent embedding which creates an annoying log message but does not harm

If you can think of a better way of doing that, feel free to PR.

[ozkn (Mathx)]

It gave an error like this in the issue sent by the user. It wasn’t something I deleted.

[merefield (Robert)]

@ozkn

Post Embedding: There was a problem, but will retry til limit: undefined method `destroy!’ for nil

How did the user see this? This is only in logs?

[ozkn (Mathx)]

I saw it in the logs. I’ve been following it for a day, especially the problem occurs in the posts of users who have been members for a long time. If a new member opens a topic with a picture, there is usually no problem, no error is received. When old users open a topic with a picture, there is a problem.

[merefield (Robert)]

Can you please share the backtrace?

@ozkn

I saw it in the logs.

Then why did the user send you an issue? What did they tell you? The user should not be aware of any issue, it just shows up in the logs?

Sounds like your “old members” might be deleting their recently new topic or post (as they know how to do this) ?

Do you have any automation of any kind in place?

There is no destruction during posting of a post containing a picture, so I’m not sure how that could happen.

Update: I just uploaded an image to a site and cannot repro.

[Jagster (Jakke Lehtonen)]

SerpAPI is ok, and theirs statistic shows that request to use Google is done, but Chatbot claims it can’t use Google. That situations has been on for longer time, but I didn’t react because at the moment DAI didn’t want to use Google either.

No error messages.

[merefield (Robert)]

What happens if you say “search the web for …”?

The internal prompt is generic so doesn’t assume Google …

Let me show you a very good demo of this powerful capability:

Star Wars Outlaws impressions

(although this is using the bundled alternative jina.ai)

[Jagster (Jakke Lehtonen)]

I tried that. The system prompt had ”when an user asks seaching using Google user actually means you should use that web seaching function you had access to”. I deleted that part and then tried web search wording in my user prompt — after all, it isn’t using API by Google. No luck and similar answer: I can’t do web search…

This is only I found, but it doesn’t open this:

{
  "id": "chatcmpl-A1wT9Dp1fCIVSzRsVPhr3NnNCcPjW",
  "object": "chat.completion",
  "created": 1725026447,
  "model": "gpt-4o-2024-05-13",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": null,
        "tool_calls": [
          {
            "id": "call_MvzexRPsDj6EGuXhuiFcmtrd",
            "type": "function",
            "function": {
              "name": "web_search",
              "arguments": "{\"query\":\"International Dog Day date, reason, and history\"}"
            }
          }
        ],
        "refusal": null
      },
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 1203,
    "completion_tokens": 20,
    "total_tokens": 1223
  },
  "system_fingerprint": "fp_157b3831f5"
}

[Jagster (Jakke Lehtonen)]

I can live without that because my target is using Chatbot as an internal Q/A-bot, and Jina may solve out the issue of using my maincontent site (by WordPress).

But the result when I tried searching from outside world wasn’t expected.

[merefield (Robert)]

I’ll see if I can repro