Discourse Chatbot 🤖 (supporting ChatGPT)

merefield · March 1, 2023, 11:08am


	Summary	A cloud chatbot adaptor for Discourse, currently supporting OpenAI
	Repository Link	GitHub - merefield/discourse-chatbot: A cloud chatbot adaptor for Discourse, currently supporting OpenAI
	Install Guide	How to install plugins in Discourse

EARLY ACCESS: This is early days for this plugin. It’s great to have help with testing and feedback. Production use should be ok now but expect minor issues. The most egregious ones should be fixed now!

Known Issues:

Persistent notifcation on Bot DM channel that won’t go away (almost certainly a core bug Discourse team is working on … to do with length of messages not fitting in screen - workaround on desktop is to maximise the chat window :tada)
May not work in multi-site configurations.
~~Code doesn’t handle it well when you delete the bot user.~~ fixed!
~~Deleted posts and messages: the bot can’t handle these at present~~ should be fixed
~~in Direct Message channels the bot is over intrusive and will presently try to reply to everything. This is being worked on.~~ I’ve released a fix for this, so you might want to test it out to confirm.
~~Strange responses that appear to suggest the post history is being gathered from different topics~~ I’ve since released a fix.

What is it?

A plugin that uses a cloud-based chatbot to provide a fun and sometimes useful interactive AI experience.

(Sorry China!

)

Is extensible and supporting other cloud bots is intended (hence the generic name for the plugin), but currently ‘only’ supports interaction with Open AI Large Language Models (LLM) such as “ChatGPT” and GPT3. This may change in the future. Please contact me if you wish to add additional bot types or want to support me to add more. PR welcome.

Setup

There is some minor setup required before you can use the bot: take a moment to read through the entire set of Plugin settings.

For AIbot to work in Chat you must have Chat enabled.

OpenAI

You must get a token from https://platform.openai.com/ in order to use the current bot. A default language model is set (one of the most sophisticated), but you can try a cheaper alternative, the list is here

There is an automated part of the setup: upon addition to a Discourse, the plugin currently sets up a AI bot user with the following attributes

Name: ‘AIBot’
User Id: -4
Bio: “Hi, I’m not a real person. I’m a bot that can discuss things with you. Don’t take me too seriously. Sometimes, I’m even right about stuff!”
Group Name: “ai_bot_group”
Group Full Name: “AI Bots”

You can edit the name, avatar and bio (see locale string in admin → customize → text) as you wish but make it easy to mention.

It’s not free, so there’s a quota system, and you have to set this up

Initially no-one will have access to the bot, not even staff.

Calling the Open AI API is not free after an initial free allocation has expired! So, I’ve implemented a quota system to keep this under control, keep costs down and prevent abuse. The cost is not crazy with these small interactions, but it may add up if it gets popular. You can read more about OpenAI pricing on their pricing page.

Example calculations can be found in this post:

In order to interact with the bot you must belong to a group that has been added to one of the three levels of trusted sets of groups, low, medium & high trust group sets. You can modify each of the number of allowed interactions per week per trusted group sets in the corresponding settings.

You must populate the groups too. That configuration is entirely up to you. They start out blank so initially no-one will have access to the bot:

In this example I’ve made staff have high trust access, whilst trust_level_0 have low trust. They get the corresponding quotas in three additional settings.

Note the user gets the quota based on the highest trusted group they are a member of.

“Prompt Engineering”

There are several locale text “settings” that influence what the bot receives and how the bot responds.

The most important one you should consider changing is the bot’s system prompt. This is sent every time you speak to the bot.

You can try a system prompt like:

’You are an extreme Formula One fan, you love everything to do with motorsport and its high octane levels of excitement’ instead of the default.

Try one that is most appropriate for the subject matter of your forum. Be creative!

Changing these locale strings can make the bot behave very differently but cannot be amended on the fly. I would recommend changing only the system prompt as the others play an important role in providing information on who said what to the bot.

NB In Topics, the first Post and Topic Title are sent in addition to the window of Posts (determined by the lookback setting) to give the bot more context.

You can edit these strings in Admin → Customize → Text under chatbot.prompt.

github.com

merefield/discourse-chatbot/blob/87d832203cb1fcc72b5494557f9a4ed8290e8ce3/config/locales/server.en.yml#L26


      
            chatbot_quota_low_trust: "The allowed number of bot responses allowed by prompting user per week - low trust groups"
            chatbot_high_trust_groups: "Groups considered high trust for bot interaction"
            chatbot_medium_trust_groups: "Groups considered medium trust for bot interaction"
            chatbot_low_trust_groups: "Groups considered low trust for bot interaction"
            chatbot_request_temperature: "Numeric value within the range of 0 to 100 to determine the level of randomization for AI responses. Higher temp = more creative, lower temp = more conservative"
            chatbot_max_response_tokens: "Specify the maximum amount of tokens bot will use to generate a response: <a>https://openai.com/pricing<a/>"
          
          
chatbot:
            bio: "Hi, I’m not a real person. I’m a bot that can discuss things with you. Don't take me too seriously.  Sometimes, I'm even right about stuff!"
            prompt:
              system: "You are a helpful assistant."
              title: "The subject of this conversation is %{topic_title}"
              first_post: "The first thing someone said was %{username} who said %{raw}"
              post: "%{username} said %{raw}"
            errors:
              general: "Sorry, I'm not well right now. Lets talk some other time. Meanwhile, please ask the admin to check the logs, thank you!"
              forbiddeninprivatemessages: "Use in private messages has not been permitted, please contact the admin"
              forbiddenoutsidethesecategories: "Public use of Chatbot has only been permitted in these Categories: "
              forbiddenanycategory: "Public use of Chatbot has not been enabled in any Category, please contact your admin"
              overquota: "I'm afraid you've reached this week's quota for bot interactions"

Supports both Posts & Chat Messages!

The bot supports Chat Messages and Topic Posts, including Private Messages (if configured).

You can prompt the bot to respond by replying to it, or @ mentioning it. You can set how far the bot looks behind to get context for a response. The bigger the value the more costly will be each call.

There’s a floating quick chat button that connects you immediately to the bot. Its styling is a little experimental (modifying some z-index values of your base forum on mobile) and it may clash on some pages. This can be disabled in settings. PR welcome to improve how it behaves.

Disclaimer: I’m not responsible for what the bot responds with. Consider the plugin to be at Beta stage and things could go wrong. It will improve with feedback. But not necessarily the bots response Please understand the pro’s and con’s of a LLM and what they are and aren’t capable of and their limitations. They are very good at creating convincing text but can often be factually wrong.

Important Privacy Note: whatever you write on your forum may get forwarded to Open AI as part of the bots scan of the last few posts once it is prompted to reply (obviously this is restricted to the current Topic or Chat Channel). Whilst it almost certainly won’t be incorporated into their pre-trained models, they will use the data in their analytics and logging. Be sure to add this fact into your forum’s TOS & privacy statements. Related links: Terms of use, Privacy policy, OpenAI API

Copyright: Open AI made a statement about Copyright here: Will OpenAI claim copyright over what outputs I generate with the API? | OpenAI Help Center

TODO/Roadmap Items

Add front and back-end tests
Add extra logic to convert suspected usernames into @ mentions (@frold )
forgot to mention the bot? Get bot to respond to edits that add its @ mention (@frold )
Add a badge? You did mention @botname (@frold )
Add setting to include Category and Pinned Posts prompt? (@Ed_S)
Ditto Bios to each message history prompt? (@Ed_S , @codergautam). Will this even work. Let’s get evidence.
Update Discourse Frotz with this better codebase?
~~Add support for ChatGPT~~
~~Lint the plugin to Discourse core standards~~
~~Add CI workflows~~
~~Add settings to influence the nature of the bots response (e.g. how wacky it is)~~.
~~include Topic Title & first Posts to prompt~~
Add setting to switch from raw Post/Message data to cooked to potentially leverage web training data better (suggestion by @MarcP). NB May cost more and limit what is returned as input tokens are counted and cooked is much bigger. think we’ve abandoned this idea

Credits:

Thanks to @MarcP for his enthusiastic support and detailed testing feedback.
The guys who’ve contributed to the emerging standard Ruby API for Open AI which this leverages
The floating button design is mostly taken from Discourse’s own GitHub - discourse/material-design-stock-theme (so they are not currently compatible, doh!)
The fixtures code is based on @vinothkannans work here
If @P16 hadn’t hired me many moons ago to help him with a chat bot this might not exist

jord8on · March 1, 2023, 7:12pm

Wow! This is super interesting. I am still trying to wrap my head around all the use cases of GPT. I think so many people have become experts and know the best phrases and keywords and even booleans to use when searching Google. But we also know the limitations and our search queries have been within that box of what we know Google can actually spit out for us. GPT seems to take this same search functionality to a whole new level… even a tactical level where it can help you take action with the results you’re given.

Great work on this @merefield!

merefield · March 1, 2023, 8:28pm

And would you believe the timing? This just turned up in my inbox:

I will try to squeeze in some work to support this at some point.

Present model is pretty good though.

Now back to work

jimmynewtron · March 1, 2023, 10:20pm

We get some pretty strange responses at the moment:

It also sometimes responds like this, with the “CREATE_POST_RESPONSES” and also I had one with a flag notice or something along the lines of that:

Any ideas?

merefield · March 1, 2023, 10:24pm

That’s quite odd! I’ve had some odd responses but never that odd.

The responses aren’t always factually correct but rarely so bizarre. The demo in the screenshot is fairly typical.

Are you embedding anything in posts? Any images that might be throwing it off? Try it on the end of just a series of text posts?

I just tried this:

and this:

Both were requested at the beginning of a Topic.

You are better to ask it specific things, not just “help us out”: help us out with what?

If your posts are cluttered with links and images, try Chat.

jimmynewtron · March 1, 2023, 11:22pm

This is just chat and nothing else & I still get these very peculiar responses. I feel like the AI lacks it’s usual knowledge and just seems kind of “stupid”.

merefield · March 1, 2023, 11:32pm

I cannot explain your experience. It is far more capable. I’d be keen to know if others are having this issue.

Could you please try Discourse Chat?

jimmynewtron · March 1, 2023, 11:38pm

Works fine in discourse chat, strange behaviour .At the moment our community doesn’t use the chat.

merefield · March 2, 2023, 12:02am

Did you try it with the default bot name?

merefield · March 2, 2023, 10:38am

ChatGPT model in test: (coming soon )

merefield · March 2, 2023, 12:54pm

Support for ChatGPT along with some other improvements has been merged:

github.com/merefield/discourse-chatbot

FEATURE: Support ChatGPT model

merefield:main ← merefield:support_chat_gpt_model

opened 12:38PM - 02 Mar 23 UTC

merefield

+83 -42

- Adds support for the model behind ChatGPT ("gpt-3.5-turbo") - improves resp…onses and avoids prompt 'hacking' - reduces cost of use by a factor of 10! - provides option to modify nature of response by setting up a freeform expectation, e.g. "You are a helpful assistant", or "You are a mad scientist"?! - Adds setting for delay in seconds for paying customers who want a faster response and have better rate limits - Cleans up some code and improves console feedback.

Adds support for the model behind ChatGPT (“gpt-3.5-turbo”)
- improves responses and avoids some significant prompt ‘hacking’
- reduces cost of use by a factor of 10!
- provides option to modify nature of response by setting up a freeform expectation, e.g. “You are a helpful assistant” is default, or perhaps try “You are a mad scientist”?!
Changing models is now possible via a helpful dropdown, no need to type in the exact name.
Adds setting for delay in seconds for paying customers who want a faster response and have better rate limits
Cleans up some code and improves console feedback.

@jimmynewtron please let me know if this resolves your issue - if not could you please share with me the list of plugins you are using? Feel free to PM.

MarcP · March 2, 2023, 3:05pm

Hey @merefield this is so cool and I love the group limits.

However, the behavior reported above, happened to me while testing in a slightly different way.

But I can conclude, somehow it got context from a very different topic (XYZ) and is using that in the reply in topic ABC. From front-end, it looks like it’s using the context from a wrong topic IP, is this possible?

merefield · March 2, 2023, 3:16pm

That’s interesting. Bugs are possible, it’s early days too.

Is it working most of the time? Can you reproduce on the latest version?

MarcP · March 2, 2023, 3:23pm

I always used the latest version, never installed the one before.

I am still playing around, I will let you know.

Also, I notice it replies even if it’s not mentioned in topics. I assume you want it to only reply if @ or pressed Reply on one of his posts, not if you press Reply on the topic:

merefield · March 2, 2023, 3:26pm

That’s intended behaviour so not a bug as such. If that’s demonstrably a problem I’m happy to reconsider that behaviour.

MarcP · March 2, 2023, 3:28pm

Of course, if you have a long topic and want to ask AI for input once, you don’t want it do keep participating with the topic for the rest of it’s lifetime, every time someone replies, right?

merefield · March 2, 2023, 3:29pm

Right. That’s great feedback. I’ll see what I can do.

Presumably same will apply to chat? The only downside is the interface there makes replies more manual effort.

MarcP · March 2, 2023, 3:35pm

There might be some info (you probably already know) in this topic, I made a simple prompt-reply bot and it only listens for type 1/2 notifications and that works great.

In channels you’d want the same behavior, for private chat with the bot it can always reply but there should probably be a different limit in prompt history, and perhaps a “magic word” to reset the context history for the bot?

merefield · March 2, 2023, 4:00pm

I think there is a special case too, where if the bot is the only other actor in the topic, it makes sense to assume a further post is meant for the bot regardless of the means of replying if the last post was a bot post? In other words, retain the current behaviour if the only participants so far are the bot and the user and the same user is replying?

ie: add a clause that if you reply to the Topic, and the last post is owned by the bot, do not assume user is replying to the bot if there is more than one human user in the Topic*, otherwise do?

(*which would also be the case if the person replying has not yet posted, ie there’s about to be two users in the Topic)

This logic would then work for one to one PMs or chats as well as open topics/channels.

EDIT: in fact this is simple: count human participants in topic (and include typist of new Post) and if this is more than one switch off convenient reply? (convenient because it also has a hotkey).

MarcP · March 2, 2023, 4:25pm

Randomly, after testing a bit back and forth, the following response is given:

Sorry, I’m not well right now. Lets talk some other time. Meanwhile, please ask the admin to check the logs, thank you!

Error log: OpenAIBot: There was a problem: Couldn't find Post with 'id'=99 [WHERE "posts"."deleted_at" IS NULL]

That would be a possible approach - as long as so far is always respected.

I think behavior should be as follows:

Topic - only reply if mentioned or "reply button to an earlier reply of the bot is clicked.
- I don’t think we should encourage long AI chats on a forum - AI is useful if you call him and ask him a prompt, if it always replies in a topic it would be more towards entertainment because you don’t really need it, right?
- However, I could see this working if you can specify a category as “AI will always reply”-setting?
PM - always reply
Chat channels - only reply if mentioned or "reply button to an earlier reply of the bot is clicked. Unless a chat-category is specified as “always reply”?
Private chat with bot - always reply

This is one way to approach it. But I still believe on the forums the AI would be better off (more functional) if it’s call-based only by default and specify categories where you want AI to be “active aggressive”, lol.

EDIT: Looks like AIBot is hijacking every private-chat without being mentioned. To repro chat a random user and AIBot will reply. Also noticed that mentioning Alice in another chat-category then where it’s allowed to reply, it still replies, not sure if that is intended.

Got to say this is way cooler than the simple bot I created and I hope the team will include this in official, I can see some very useful appliances with this bot (what if AIBot knows every (public!) thing that’s on your forum and can utilize that in the chat without a huge token usage. Keeps dreaming…