Discourse Chatbot 🤖

37Rb · September 8, 2025, 10:56pm

Our bot was timing out until we set reasoning to minimal. Thanks!

merefield · September 9, 2025, 6:20am

Tbh I’m finding GPT-5 generally too slow with higher reasoning kevels and not obviously worth the additional lead time in responses.

How have you found it for your support bot? Does a minimal reasoning GPT-5 perform better than say 4o or 4.1?

Jagster · September 9, 2025, 6:30am

I’ve tried gpt-5 using Chat GPT, that is really much different thing than via API, and it needs that long reasoning time to give slightly better answers what 4o would give, or o1. When it has to answer fast, it is not any better than 4.1.

I’m quite sure the situation is same`ish, or worse because of lack of tools and prompting, when using API. But I don’t know for sure, because gpt-5 is painfully slow and in forum environment it must answer close to speed of light.

37Rb · September 9, 2025, 4:05pm

In terms of content performance, anecdotally, it seems like gpt-5 is giving noticeably better technical answers that gpt-4o. I’m not sure how to quantify that but it really impressed me.

I’m getting varying results in how long it takes to respond. It does seem, from experimenting this morning, like gpt-5 is slower on average but not by too much, and there were some cases where the response came faster with gpt-5. I’m measuring anywhere from 5 seconds to 35 seconds for a reply.

We’re using RAG and I can’t tell what portion of the latency is from the RAG search vs the chat completion. It could be that sometimes it chooses not to RAG search, the search happens faster, or something is cached (in the search or the completion).

We would typically choose better answers over a faster response because giving customers bad technical advice is costly. Up to a point though, if it times out then that’s a very bad user experience.

GPT-5 recommends primarily gpt-5-mini for our use case, and escalate to gpt-5 in some circumstances. Sounds neat but complicated. Have you considered switching between models dynamically? Why doesn’t OpenAI just do that automatically? ChatGPT - Compare GPT models performance

37Rb · September 9, 2025, 8:41pm

We had to switch back to gpt-4o because apparently gpt-5-mini thinks it can do things it can’t do. It confidently offered to set up a customer’s alarm monitoring service for them and connect it to their home alarm equipment. It asked them for equipment ID numbers and hallucinated like it was a concierge setting everything up for them. Our website can do that but the chatbot can’t. It doesn’t seem to be respecting the guardrails in the system prompt like gpt-4o did. We’ll need to tighten it up before we can let people use it.

Update: It turns out that gpt-5 is much better at following instructions and respecting rules in the prompt than gpt-5-mini. If you’re going to let a bot represent your brand, I recommend gpt-5 even though it’s slower and 5x more expensive. There’s too much risk that gpt-5-mini will go off the rails.

StevePlex · November 20, 2025, 12:06am

I have had really good luck with GTP-5-mini in agentic flows via tool calling, code writing and structured data. I generally find structured data is easier for AI apps than unstructured ! .. not what I expected ! but guardrails are easier .. (code-in-loop, human-in-loop, llm-as-judge, etc)

please watch this for blow by blow walkthru of high performance , low cost gpt-5-mini and gpt-4o …

If anyone out there is interested in working structured data capabilities into Discourse as a plugin, etc. Please reach out.

An NLP extension for sql/stats/datascience to Data Explorer is an example.. But could also possibly have a tool / plugin / feature that allows natural language queries of read-only sqlLite or duckdb etc olap files loaded into the container ? just a thought..

merefield · November 20, 2025, 7:49am

Btw, I added GPT 5.1 to the plugin along with some fixes:

https://github.com/merefield/discourse-chatbot/pull/147

merefield · December 2, 2025, 6:57pm

@tom_eric you asked about the ability to play games with other forum members on another Topic.

I tried this prompt with Chatbot and it seems to work with GPT 5.1, try it out:

@Chatbot facilitate a 9x9 game of Go with @Other_Player and I using a markdown table to render the board - let me go first and only accept plays from myself and @Other_Player and only when it is our turn - render the board and ask me to kick off …

tom_eric · December 2, 2025, 7:06pm

Oh, thank you. That’s a really creative idea.Oh, thank you. That’s a really creative idea.

merefield · December 3, 2025, 9:24am

np let me know how that works out

merefield · December 20, 2025, 1:27pm

A slightly customized dev setup let me experiment with Chatbot playing Zork using this plugin and Discourse Frotz

(here with GPT 5.1 on lower reasoning)

A little time later:

Racking up quite a score! And:

(Apologies, setup is a little too complex to simply share here).

NateDhaliwal · December 20, 2025, 1:35pm

Wow! I had no idea that was possible. Guess pretty much anything is possible with good prompts

merefield · February 15, 2026, 5:00pm

Chatbot model dropdowns now have gpt 5.2 and 5.2 pro - and for 5.2 there is now an xhigh reasoning level if you love burning tokens and devastating the environment

You may also notice that users of the Maths Plugin can now easily get Chatbot to discuss maths with nicely rendered maths equations without adding anything to the system prompt …

merefield · February 16, 2026, 4:28pm

I’ve moved the launch button to a much neater location in the bottom right (except for iOS PWA/app) - let me know if that’s not working as intended.

This was always intended to simplify the css whilst accommodating those annoying control bars on iOS, but I couldn’t live with it any longer, it was offending my eyes

merefield · March 2, 2026, 9:36am

@ThisSource this was the first AI chatbot for Discourse and is still running :).

merefield · March 7, 2026, 5:37pm

Delighted to announce my first ongoing business sponsor of Chatbot, Surety, who now occupy the new Project Sponsor section of the README.

Surety is on a mission to provide professional-grade security alarm monitoring and home automation to DIYers in the most transparent and efficient way possible, bucking the trend in the home security industry.

Thank you Surety!

github.com/merefield/discourse-chatbot

README.md

main

# discourse-chatbot

## Project Sponsors

Our kind sponsors of this project:

| Sponsor | Level | Weblink | Logo |
| --- | --- | --- | --- |
| Surety | Silver Plus | [suretyhome.com](https://suretyhome.com/) | <img src="images/surety.webp" alt="Surety" width="120"> |

# What is it?

* The original Discourse AI Chatbot!
* Converse with the bot in any Post or Chat Channel, one to one or with others!
* Customise the character of your bot to suit your forum!
  * want it to sound like William Shakespeare, or Winston Churchill? can do!
* The new "RAG mode" can now:
  * Search your whole* forum for answers so the bot can be an expert on the subject of your forum.
    * not just be aware of the information on the current Topic or Channel.
  * Search Wikipedia

This file has been truncated. show original

@37Rb

If you’d like to become a sponsor of one of my projects, take a look at: Sponsor @merefield on GitHub Sponsors · GitHub

merefield · August 3, 2026, 11:12am

Recent PR summary — 1–3 August 2026

Several related PRs were merged across Discourse Chatbot and its extension plugins.

Highlights

Introduced advanced local reasoning strategies, including verify-and-revise, best-of-two, and uncertainty-guided reasoning.
Added current OpenAI models and the new max reasoning effort.
Added a semantic blocked-question gate using embedding similarity.
- significant savings as uses embeddings to block subjects you don’t wish the bot to handle without costing you LLM tokens.
Consolidated the basic and RAG implementations into a single DiscourseChatbot::Bot.
Replaced bot-mode settings with trust-level built-in tool selectors.
Renamed the extension API from “Function” to “Tool”.
Replaced the SafeRuby based calculator tool with the constrained Dentaku expression evaluator - this is a big improvement in security, as despite its name, there are vulnerabilities in SafeRuby.
Adopted Zeitwerk loading across Chatbot and its extension plugins.
Moved location-specific tools into the Locations Early Access plugin (sponsor me to regain access)
Improved calculator error recovery and support for common π/e notation.
Refreshed the README
Advanced the main plugin from version 1.8.0 to 2.4.1.