After taking a look at a few of the topics this is what I learned and recommended.
The site is for support of the Surety home security system.
The goal for the AI used with Discourse forum is (ref)
- Compliment our human support team for a technical support forum.
- It would need to learn from our historical forum data.
- Have the domain knowledge to provide support.
- It should only attempt to answer on-topic questions.
- It should provide a way for the user to tell the bot to stop answering so they can wait for a human instead if they feel like the bot isn’t helping.
And here are some of the questions from Ryan
- Do these automatically learn or fine-tune with historical forum data?
- Not duplicate work that’s already been done by smarter people.
- Restrict the topics the bot will engage in so it doesn’t engage in general, off-topic conversations?
- Is the original Discourse Chatbot intended to be developed and maintained long term?
- Or is it planned to eventually be replaced by the AI Bot in the official plugin, or folded into it?
Here is a posted question I gave to the Discourse AI bot currently enabled on the version of the site I tried and an answer from ChatGPT.
The question:
I have a IQP4 with a Powerg to hardwire. The wired sensors are working. I wanted to run a test on the bell/siren, but could not find the option on the IQP4. I can test the siren on the panel and it sounds along with the IQ Remote, but the hardwired bell does not. I have to trip the alarm to get the hardwired siren to sound.
Is there a way to test the bell on the powerg to hardwire from the IQP4?
Here is the reply using the Discourse AI bot with GPT-4
I found a relevant topic on our forum that may help you with your issue.
In the topic Test wired bell on PowerG to hardwire?, the user had a similar issue where they could not find the option to test the bell on the PowerG to hardwire from the IQP4.
Please have a look at this topic for possible solutions. If you still have questions or need further assistance, feel free to ask.
Here is the reply using the Discourse AI bot with GPT-3.5
I found a topic on the Surety Support Forum that discusses testing the wired bell on the PowerG to hardwire connection with the IQ Panel 4 (IQP4). You can find the topic here.
In the topic, the user mentions that they couldn’t find an option to test the bell/siren on the IQP4 itself. However, they were able to test the siren on the panel and it sounded along with the IQ Remote. The hardwired bell, on the other hand, did not sound unless the alarm was tripped.
You can visit the topic for more information and to see if there are any solutions or suggestions provided by other users.
Both the Discourse AI GPT-4 and GPT-3.5 bot found the same topic.
Here is the reply using ChatGPT
I don’t have specific information on the IQ Panel 4’s latest software version or features, as my training only includes knowledge up to January 2022. However, based on general principles of alarm system testing, here are some suggestions that might help you test your hardwired siren connected to the PowerG:
System Test Mode:
- Check if your alarm panel has a “System Test” or “Walk Test” mode. This mode is typically designed for testing sensors and alarms without triggering a real alarm event. If your panel has this feature, it should allow you to test the siren without setting off a full alarm.
Output Control:
- Look for an “Output Control” or “Auxiliary Output” option in your alarm panel settings. This feature may allow you to manually trigger outputs, such as the siren, for testing purposes.
Zone Bypass:
- Some systems allow you to bypass specific zones during testing. If your hardwired siren is associated with a particular zone, you might be able to bypass that zone for testing.
User Manual:
- Refer to the user manual or documentation that came with your IQ Panel 4. It should provide specific instructions on how to test and control different components of your alarm system.
Technical Support:
- If you’re unable to find a solution on your own, consider reaching out to the manufacturer’s technical support. They can provide guidance based on the specific features and capabilities of your system.
Panel Software Update:
- Check if there are any available software updates for your IQ Panel 4. Manufacturers often release updates to add new features or address issues, and an update might include a specific test mode for the siren.
Remember to exercise caution while testing to avoid accidentally triggering a real alarm event. If you’re unsure about any steps, it’s always a good idea to consult the user manual or contact technical support for assistance.
Clearly ChatGPT is giving a more detailed answer. I will go into possible reasons later in the topic.
Note: I do not have access to the Discourse AI plugin by Merefield (Card) (GitHub), it would be nice to see the response to the same question posted here for comparison. 
Here is my feedback.
The AI for these bots are based on LLMs, the knowledge is limited by the training set. The bots are also using RAG to augment their knowledge by accessing the topics in the forum.
The bots will typically use a system prompt, think persona, which is not a user prompt with the request to the LLM API.
I will let ChatGPT explain the difference.
ChatGPT
What is a system prompt? How is it different from a user prompt?
Completion
A system prompt and a user prompt are terms often used in the context of language models and conversational AI systems like GPT-3.5. Let me clarify the difference between them:
User Prompt:
- A user prompt is the input provided by a human user to initiate a conversation or request information from the AI system.
- It’s the text or query that a person types or speaks to interact with the AI model.
- For example, if you’re using a chatbot or a language model, your input asking a question or making a statement would be the user prompt.
System Prompt:
- A system prompt, on the other hand, is the initial input or instruction given to the AI model to set the context or guide its behavior for a conversation.
- It’s used to provide the AI with a starting point or context for generating responses.
- The system prompt helps shape the tone, style, or content of the AI’s responses, and it serves as a sort of instruction to guide the model’s behavior.
- While user prompts are specific to the user’s queries or statements, system prompts are more about giving a general context or instruction for the entire conversation.
In the case of GPT-3.5 or similar language models, a conversation typically starts with a system prompt followed by user prompts in an ongoing exchange. The system prompt helps influence the model’s responses throughout the conversation, but it’s the combination of system and user prompts that determines the model’s output.
The reason the bots are not able to give a much more specific answer is because their knowledge base is lacking the user manuals, installation instructions, etc. for these products. Further bots are often not able to extract images and diagrams from such documentation such as wiring diagrams and images used in the installation manuals.
As a start, RAG should be used to expand the knowledge for the AI as much as possible even if it is only the text from the documentation. While the Discourse AI bot can use RAG for all of the topics in the forum I don’t know if they are allowing external information to be added. If not, one idea would be to import all of the documents as PDFs into a category, possibly hidden, and then the Discourse code could get at them easier as they are in the forum and then use Langchain to access the PDFs (ref)
With this the knowledge in there the system prompt can be tweaked to improve the replies but this will still be far from what you seek. You will still most likely get replies like the first with references to information but no step by step instructions when needed. Also there is no way to give feedback about the AI reply that the AI would know is giving feedback. In other words if a user replied to the bot that something is wrong, the bot would read the reply but not understand that it would need to update the knowledge based on that.
To improve the quality of the replies three thoughts come to mind.
- Make use of HyDE (Hypothetical Document Embeddings)
“Precise Zero-Shot Dense Retrieval without Relevance Labels” by Luyu Gao, Xueguang Ma, Jimmy Lin and Jamie Callan (pdf )
I actually learned of HyDE from Discourse (ref) and know they are using it.
-
RLHF ( Reinforcement learning from human feedback). I don’t think either of the bots are capable of this and Discourse might be considering this; would like to hear their feedback if they care to share.
-
Synthetic data with RLHF. This is really cutting edge and not something I would expect either bot to implement.
One of the most demonstrated and successful ways to train an LLM is to have them train themselves but you need some metric so they know if the result is better or worse. For your site the only way I currently understand to know if a solution is better or worse is for a human to rate the solution from the AI. To get more possible rounds of questions and solutions, the questions are generated by an AI with another AI generating the solution and then the solution is measured and used to improve the AI generating the solution.
To improve the ability of the bot to help users it should perform a series of questions to collect information to better understand the problem similar to the way a medical doctor will for a diagnosis and prognosis.
Another option is to look at similar technology that can be accessed with the AI bot such as Microsoft Copilot.
One item that you did not request or note that is of value is references back to the source. The reason this is important is to check if the AI completion is an hallucination or not. If a reference is included, is a real reference and not a hallucination then the reference can be checked to see if it is from an authoritative source such as a manual and then be known to not be a hallucination. Adding references is easy to do with RAG and should be expected.
Now to address your specific goals.
Compliment our human support team for a technical support forum.
That is more of a statement than a question, so will not answer it as a question.
It would need to learn from our historical forum data.
This was actually part of a larger statement that I broke up. The reason I broke it up is that while you are correct that the historical data in the forum is of value, the base knowledge such as that in the manuals is also needed. The current bots use RAG with the forum post but AFAIK do not know how to access external data.
It should only attempt to answer on-topic questions.
This can be accomplished by tweaking the system prompt. This would also need some of the technology in Discourse AI Post Classifier - Automation rule
It should provide a way for the user to tell the bot to stop answering so they can wait for a human instead if they feel like the bot isn’t helping.
This can be accomplished by tweaking the system prompt.
Do these automatically learn or fine-tune with historical forum data?
Good question. While I have brushed on how to start to solve this earlier, it really is a much harder question than it seems but the large LLM researchers are making much better progress with this and their might be some interesting announcements in the next few months that can make this much easier.
Not duplicate work that’s already been done by smarter people.
I will take that to mean that the LLM should not digress on what it knows. That depends on how the model is trained on new knowledge. It is possible to make the model worse and even harder to get a model to unlearn. AFAIK neither of the Discourse AI bots have the ability to learn, meaning change the weights in the LLM, but they do have the ability to use RAG with topic post. So bad topic post, garbage in, can create garbage out.
Restrict the topics the bot will engage in so it doesn’t engage in general, off-topic conversations?
This would be similar to Discourse AI Post Classifier - Automation rule
Is the original Discourse Chatbot intended to be developed and maintained long term?
Will leave that one for the Discourse staff.
Or is it planned to eventually be replaced by the AI Bot in the official plugin, or folded into it?
Will leave that one for the Discourse staff.