Discourse Chatbot šŸ¤– (supporting ChatGPT)

:information_source: Summary A cloud chatbot adaptor for Discourse, currently supporting OpenAI
:hammer_and_wrench: Repository Link GitHub - merefield/discourse-chatbot: A cloud chatbot adaptor for Discourse, currently supporting OpenAI
:open_book: Install Guide How to install plugins in Discourse

EARLY ACCESS: This is early days for this plugin. It’s great to have help with testing and feedback. Production use should be ok now but expect minor issues. The most egregious ones should be fixed now!

Known Issues:

  • Persistent notifcation on Bot DM channel that won’t go away (almost certainly a core bug Discourse team is working on … to do with length of messages not fitting in screen - workaround on desktop is to maximise the chat window :tada)
  • May not work in multi-site configurations.
  • Code doesn’t handle it well when you delete the bot user. :white_check_mark: fixed!
  • Deleted posts and messages: the bot can’t handle these at present :white_check_mark: should be fixed
  • in Direct Message channels the bot is over intrusive and will presently try to reply to everything. This is being worked on. I’ve released a fix for this, so you might want to test it out to confirm. :white_check_mark:
  • Strange responses that appear to suggest the post history is being gathered from different topics I’ve since released a fix. :white_check_mark:

What is it?

A plugin that uses a cloud-based chatbot to provide a fun and sometimes useful interactive AI experience.


(Sorry China! :wink: )

Is extensible and supporting other cloud bots is intended (hence the generic name for the plugin), but currently ā€˜only’ supports interaction with Open AI Large Language Models (LLM) such as ā€œChatGPTā€ and GPT3. This may change in the future. Please contact me if you wish to add additional bot types or want to support me to add more. PR welcome.

Setup

There is some minor setup required before you can use the bot: take a moment to read through the entire set of Plugin settings.

For AIbot to work in Chat you must have Chat enabled.

OpenAI

You must get a token from https://platform.openai.com/ in order to use the current bot. A default language model is set (one of the most sophisticated), but you can try a cheaper alternative, the list is here

There is an automated part of the setup: upon addition to a Discourse, the plugin currently sets up a AI bot user with the following attributes

  • Name: ā€˜AIBot’
  • User Id: -4
  • Bio: ā€œHi, I’m not a real person. I’m a bot that can discuss things with you. Don’t take me too seriously. Sometimes, I’m even right about stuff!ā€
  • Group Name: ā€œai_bot_groupā€
  • Group Full Name: ā€œAI Botsā€

You can edit the name, avatar and bio (see locale string in admin → customize → text) as you wish but make it easy to mention.

It’s not free, so there’s a quota system, and you have to set this up

Initially no-one will have access to the bot, not even staff.

Calling the Open AI API is not free after an initial free allocation has expired! So, I’ve implemented a quota system to keep this under control, keep costs down and prevent abuse. The cost is not crazy with these small interactions, but it may add up if it gets popular. You can read more about OpenAI pricing on their pricing page.

Example calculations can be found in this post:

In order to interact with the bot you must belong to a group that has been added to one of the three levels of trusted sets of groups, low, medium & high trust group sets. You can modify each of the number of allowed interactions per week per trusted group sets in the corresponding settings.

You must populate the groups too. That configuration is entirely up to you. They start out blank so initially no-one will have access to the bot:

image

In this example I’ve made staff have high trust access, whilst trust_level_0 have low trust. They get the corresponding quotas in three additional settings.

Note the user gets the quota based on the highest trusted group they are a member of.

ā€œPrompt Engineeringā€

There are several locale text ā€œsettingsā€ that influence what the bot receives and how the bot responds.

The most important one you should consider changing is the bot’s system prompt. This is sent every time you speak to the bot.

You can try a system prompt like:

’You are an extreme Formula One fan, you love everything to do with motorsport and its high octane levels of excitement’ instead of the default.

Try one that is most appropriate for the subject matter of your forum. Be creative!

Changing these locale strings can make the bot behave very differently but cannot be amended on the fly. I would recommend changing only the system prompt as the others play an important role in providing information on who said what to the bot.

NB In Topics, the first Post and Topic Title are sent in addition to the window of Posts (determined by the lookback setting) to give the bot more context.

You can edit these strings in Admin → Customize → Text under chatbot.prompt.

Supports both Posts & Chat Messages!

The bot supports Chat Messages and Topic Posts, including Private Messages (if configured).

You can prompt the bot to respond by replying to it, or @ mentioning it. You can set how far the bot looks behind to get context for a response. The bigger the value the more costly will be each call.

There’s a floating quick chat button that connects you immediately to the bot. Its styling is a little experimental (modifying some z-index values of your base forum on mobile) and it may clash on some pages. This can be disabled in settings. PR welcome to improve how it behaves.

image

Disclaimer: I’m not responsible for what the bot responds with. Consider the plugin to be at Beta stage and things could go wrong. It will improve with feedback. But not necessarily the bots response :rofl: Please understand the pro’s and con’s of a LLM and what they are and aren’t capable of and their limitations. They are very good at creating convincing text but can often be factually wrong.

Important Privacy Note: whatever you write on your forum may get forwarded to Open AI as part of the bots scan of the last few posts once it is prompted to reply (obviously this is restricted to the current Topic or Chat Channel). Whilst it almost certainly won’t be incorporated into their pre-trained models, they will use the data in their analytics and logging. Be sure to add this fact into your forum’s TOS & privacy statements. Related links: Terms of use, Privacy policy, OpenAI API

Copyright: Open AI made a statement about Copyright here: Will OpenAI claim copyright over what outputs I generate with the API? | OpenAI Help Center

TODO/Roadmap Items

  • Add front and back-end tests :construction:

  • Add extra logic to convert suspected usernames into @ mentions (@frold )

  • forgot to mention the bot? Get bot to respond to edits that add its @ mention (@frold )

  • Add a badge? You did mention @botname (@frold )

  • Add setting to include Category and Pinned Posts prompt? (@Ed_S)

  • Ditto Bios to each message history prompt? (@Ed_S , @codergautam). Will this even work. Let’s get evidence.

  • Update Discourse Frotz with this better codebase?

  • Add support for ChatGPT :white_check_mark:

  • Lint the plugin to Discourse core standards :white_check_mark:

  • Add CI workflows :white_check_mark:

  • Add settings to influence the nature of the bots response (e.g. how wacky it is). :white_check_mark:

  • include Topic Title & first Posts to prompt :white_check_mark:

  • Add setting to switch from raw Post/Message data to cooked to potentially leverage web training data better (suggestion by @MarcP). NB May cost more and limit what is returned as input tokens are counted and cooked is much bigger. think we’ve abandoned this idea

Credits:

52 Likes

Wow! This is super interesting. I am still trying to wrap my head around all the use cases of GPT. I think so many people have become experts and know the best phrases and keywords and even booleans to use when searching Google. But we also know the limitations and our search queries have been within that box of what we know Google can actually spit out for us. GPT seems to take this same search functionality to a whole new level… even a tactical level where it can help you take action with the results you’re given.

Great work on this @merefield!

5 Likes

And would you believe the timing? This just turned up in my inbox:

image

I will try to squeeze in some work to support this at some point.

Present model is pretty good though.

Now back to work :sweat_smile:

8 Likes

We get some pretty strange responses at the moment:

It also sometimes responds like this, with the ā€œCREATE_POST_RESPONSESā€ and also I had one with a flag notice or something along the lines of that:

Any ideas?

2 Likes

That’s quite odd! I’ve had some odd responses but never that odd.

The responses aren’t always factually correct but rarely so bizarre. The demo in the screenshot is fairly typical.

Are you embedding anything in posts? Any images that might be throwing it off? Try it on the end of just a series of text posts?

I just tried this:

and this:

Both were requested at the beginning of a Topic.

You are better to ask it specific things, not just ā€œhelp us outā€: help us out with what?

If your posts are cluttered with links and images, try Chat.

3 Likes

This is just chat and nothing else & I still get these very peculiar responses. I feel like the AI lacks it’s usual knowledge and just seems kind of ā€œstupidā€.

1 Like

I cannot explain your experience. It is far more capable. I’d be keen to know if others are having this issue.

Could you please try Discourse Chat?

1 Like

Works fine in discourse chat, strange behaviour .At the moment our community doesn’t use the chat. :confused:

1 Like

Did you try it with the default bot name?

1 Like

ChatGPT model in test: (coming soon :rocket: )

3 Likes

Support for ChatGPT along with some other improvements has been merged:

  • Adds support for the model behind ChatGPT (ā€œgpt-3.5-turboā€)
    • improves responses and avoids some significant prompt ā€˜hacking’
    • reduces cost of use by a factor of 10!
    • provides option to modify nature of response by setting up a freeform expectation, e.g. ā€œYou are a helpful assistantā€ is default, or perhaps try ā€œYou are a mad scientistā€?!
  • Changing models is now possible via a helpful dropdown, no need to type in the exact name.
  • Adds setting for delay in seconds for paying customers who want a faster response and have better rate limits
  • Cleans up some code and improves console feedback.

@jimmynewtron please let me know if this resolves your issue - if not could you please share with me the list of plugins you are using? Feel free to PM.

4 Likes

Hey @merefield this is so cool and I love the group limits.

However, the behavior reported above, happened to me while testing in a slightly different way.

But I can conclude, somehow it got context from a very different topic (XYZ) and is using that in the reply in topic ABC. From front-end, it looks like it’s using the context from a wrong topic IP, is this possible?

3 Likes

That’s interesting. Bugs are possible, it’s early days too.

Is it working most of the time? Can you reproduce on the latest version?

2 Likes

I always used the latest version, never installed the one before.

I am still playing around, I will let you know.

Also, I notice it replies even if it’s not mentioned in topics. I assume you want it to only reply if @ or pressed Reply on one of his posts, not if you press Reply on the topic:

1 Like

That’s intended behaviour so not a bug as such. If that’s demonstrably a problem I’m happy to reconsider that behaviour.

1 Like

Of course, if you have a long topic and want to ask AI for input once, you don’t want it do keep participating with the topic for the rest of it’s lifetime, every time someone replies, right?

3 Likes

Right. That’s great feedback. I’ll see what I can do.

Presumably same will apply to chat? The only downside is the interface there makes replies more manual effort.

1 Like

There might be some info (you probably already know) in this topic, I made a simple prompt-reply bot and it only listens for type 1/2 notifications and that works great.

In channels you’d want the same behavior, for private chat with the bot it can always reply but there should probably be a different limit in prompt history, and perhaps a ā€œmagic wordā€ to reset the context history for the bot?

1 Like

I think there is a special case too, where if the bot is the only other actor in the topic, it makes sense to assume a further post is meant for the bot regardless of the means of replying if the last post was a bot post? In other words, retain the current behaviour if the only participants so far are the bot and the user and the same user is replying?

ie: add a clause that if you reply to the Topic, and the last post is owned by the bot, do not assume user is replying to the bot if there is more than one human user in the Topic*, otherwise do?

(*which would also be the case if the person replying has not yet posted, ie there’s about to be two users in the Topic)

This logic would then work for one to one PMs or chats as well as open topics/channels.

EDIT: in fact this is simple: count human participants in topic (and include typist of new Post) and if this is more than one switch off convenient reply? (convenient because it also has a hotkey).

3 Likes

Randomly, after testing a bit back and forth, the following response is given:

Sorry, I’m not well right now. Lets talk some other time. Meanwhile, please ask the admin to check the logs, thank you!

Error log: OpenAIBot: There was a problem: Couldn't find Post with 'id'=99 [WHERE "posts"."deleted_at" IS NULL]

That would be a possible approach - as long as so far is always respected.

I think behavior should be as follows:

  • Topic - only reply if mentioned or "reply button to an earlier reply of the bot is clicked.
    • I don’t think we should encourage long AI chats on a forum - AI is useful if you call him and ask him a prompt, if it always replies in a topic it would be more towards entertainment because you don’t really need it, right?
    • However, I could see this working if you can specify a category as ā€œAI will always replyā€-setting?
  • PM - always reply
  • Chat channels - only reply if mentioned or "reply button to an earlier reply of the bot is clicked. Unless a chat-category is specified as ā€œalways replyā€?
  • Private chat with bot - always reply

This is one way to approach it. But I still believe on the forums the AI would be better off (more functional) if it’s call-based only by default and specify categories where you want AI to be ā€œactive aggressiveā€, lol.

EDIT: Looks like AIBot is hijacking every private-chat without being mentioned. To repro chat a random user and AIBot will reply. Also noticed that mentioning Alice in another chat-category then where it’s allowed to reply, it still replies, not sure if that is intended.

Got to say this is way cooler than the simple bot I created and I hope the team will include this in official, I can see some very useful appliances with this bot (what if AIBot knows every (public!) thing that’s on your forum and can utilize that in the chat without a huge token usage. Keeps dreaming…

2 Likes