Discourse AI Topic Summary : automated summaries and smart tagging

[merefield (Robert)]

	Summary	Uses a remote AI language model to prepare and post a summary of a topic.
	Repository Link	GitHub - merefield/discourse-ai-topic-summary: Uses a remote AI language model to prepare and post a summary of a Topic
	Install Guide	How to install plugins in Discourse

Features

(NB, this is just a UI preview using some horrible random dev fixtures, results are vastly better on real data)

Experimental, but stable.

After a minimum number of Posts, if a Topic lives in an in-scope Category, the plugin will send the topic text to an Open AI Large Language Model with a prompt and post the response, intended to be a summary of the contents which it succeeds in doing pretty well. There is a choice of model.

The summaries are often surprisingly good, occasionally sublime. However when one is not good enough, I’ve added a community downvoting mechanic to force a refresh of poor summaries after a set threshold.

Summaries are in any case re-sought when a set number of additional posts have been made.

You can modify the prompt and the thresholds

IMPORTANT NOTE: the summaries are never exposed to anon so the text will not be crawlable (you may or may not think this is a good thing, but at least the crawlers will only operate on your genuine human data).

EXPERIMENTAL FEATURE: auto-tagging support:

Tired of tagging Topics? Let the AI do it for you! (Relatively intelligently!)

Sometimes the AI gets too creative despite the direction we are giving it here , so you can restrict it to the set of existing ones.

you can quickly create a non-admin user for this purpose from the rails console by using:

rake admin:create (don’t give this user admin priviliges)

(This is a bit of a hacky workaround because the current Discourse “internal API” for tagging currently does not allow you to specify “no new tags”, so the only way of easily preventing this at present is by creating them as a user that does not have that privilege (ie trust level less that min_trust_to_create_tag)

Limitations

Due to token limits, it’s only good for about 40 Posts max currently, so it will not include material in Posts after a certain point. That will almost certainly change in the future as the models get more powerful and the services more sophisticated.

Rate limits and costs

Retrieving data from Open AI is not free. However, rate limits are implicitly based on Posting. You have control on how many posts it takes before new summaries are sought. This is unlikely to be an issue.

Configuration

Please take a look through the settings, they should be pretty self explanatory. You will need a token from https://openai.com. The link is also in the settings.

Layouts support!!

There’s now a Layouts compatible widget which ships with the plugin. You just have to install the Layouts plugin and configure it. Once installed you can turn off the standard top-of-topic summary and rely on the widget in the sidebar.

Notes

Disclaimer: I’m not responsible for what the LLM responds with. Please understand the pro’s and con’s of a LLM and what they are and aren’t capable of and their limitations. They are very good at creating convincing, context aware, text but can be factually wrong.

Important Privacy Note: Your forum data is being sent to Open AI. Whilst it almost certainly won’t be incorporated into their pre-trained models, they will use the data in their analytics and logging. Be sure to add this fact into your forum’s TOS & privacy statements. Related links: Terms of use, Privacy policy

Copyright: Open AI made a statement about Copyright here: Will OpenAI claim copyright over what outputs I generate with the API? | OpenAI Help Center

TODO

• Add front & back tests
• Add more user configuration to affect the style of response
• Add GPT-4 support when available
• Make the model setting a drop-down list
• Add widget support for the Layouts plugin

[jimmynewtron]

This was just a matter of time until we got this!

But my primary question is how expensive can this get for us?

[merefield (Robert)]

It’s been live on one of my forums for a few days now. It’s active on nearly all public categories.

It’s hard to answer because I don’t know how active your forum is.

My suggestion is give it a go in one Category and see how you get on.

There’s a decent dashboard.

After developing two plugins based on GPT3 (the most expensive model) and going live on one site late last month, mine looks like this for Feb:

So probably less than your email costs.

[matenauta]

Oh, that could be good to growth our glossary topics.

It should work in spanish?

[merefield (Robert)]

I have no idea, you might need to change the prompt in settings. But give it a go!

(this should be a localised string, I’ll change that soon)

[ChrisBeach (Chris)]

Thanks @merefield - this looks like a great plugin!

I’ve enabled it and entered my API key, enabled categories etc, but haven’t seen any summaries yet. Is there some way I can monitor its progress on my Discourse forum? Can’t see anything in logs or sidekiq (although admittedly haven’t had a deep look yet)

[merefield (Robert)]

Hey Chris, it reacts to new Posts. If they are within the thresholds and fits the Category location criteria, it will create a summary.

[ChrisBeach (Chris)]

I see, thanks for clarifying @merefield

[JamesPick]

Hi,

My rebuild fails when I enable this summary plugin. I don’t know if this is something to do with my underlying setup or conflicting plugins. I attach some details in case it is helpful.

Commenting out the ai-topic-summary plugin resolves the issue and the build is successful.

Many thanks for any guidance.

I, [2023-03-12T17:24:25.390052 #1]  INFO -- : > cd /var/www/discourse && su discourse -c 'LOAD_PLUGINS=0 bundle exec rake plugin:pull_compatible_all'
I, [2023-03-12T17:24:31.668769 #1]  INFO -- : discourse-ai-topic-summary is already at latest compatible version
discourse-assign is already at latest compatible version
discourse-chat-integration is already at latest compatible version
discourse-chatbot is already at latest compatible version
discourse-checklist is already at latest compatible version
discourse-solved is already at latest compatible version
docker_manager is already at latest compatible version

I, [2023-03-12T17:24:31.677801 #1]  INFO -- : > cd /var/www/discourse && su discourse -c 'bundle exec rake db:migrate'
rake aborted!
Gem::LoadError: can't activate ruby-openai-3.4.0, already activated ruby-openai-3.3.0
/var/www/discourse/lib/plugin_gem.rb:26:in `load'
/var/www/discourse/lib/plugin/instance.rb:796:in `gem'
/var/www/discourse/plugins/discourse-chatbot/plugin.rb:9:in `activate!'
/var/www/discourse/lib/plugin/instance.rb:693:in `instance_eval'
/var/www/discourse/lib/plugin/instance.rb:693:in `activate!'
/var/www/discourse/lib/discourse.rb:352:in `block in activate_plugins!'
/var/www/discourse/lib/discourse.rb:349:in `each'
/var/www/discourse/lib/discourse.rb:349:in `activate_plugins!'
/var/www/discourse/config/application.rb:216:in `block in <class:Application>'
/var/www/discourse/lib/plugin.rb:6:in `initialization_guard'
/var/www/discourse/config/application.rb:216:in `<class:Application>'
/var/www/discourse/config/application.rb:75:in `<module:Discourse>'
/var/www/discourse/config/application.rb:74:in `<top (required)>'
/var/www/discourse/Rakefile:7:in `require'
/var/www/discourse/Rakefile:7:in `<top (required)>'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
/usr/local/bin/bundle:25:in `load'
/usr/local/bin/bundle:25:in `<main>'
(See full trace by running task with --trace)
I, [2023-03-12T17:24:38.020166 #1]  INFO -- : gem install httparty -v 0.21.0 -i /var/www/discourse/plugins/discourse-ai-topic-summary/gems/3.2.1 --no-document --ignore-dependencies --no-user-install
When you HTTParty, you must party hard!
Successfully installed httparty-0.21.0
1 gem installed
gem install ruby-openai -v 3.3.0 -i /var/www/discourse/plugins/discourse-ai-topic-summary/gems/3.2.1 --no-document --ignore-dependencies --no-user-install
Note if upgrading: The `::Ruby::OpenAI` module has been removed and all classes have been moved under the top level `::OpenAI` module. To upgrade, change `require 'ruby/openai'` to `require 'openai'` and change all references to `Ruby::OpenAI` to `OpenAI`.
Successfully installed ruby-openai-3.3.0
1 gem installed
gem install httparty -v 0.21.0 -i /var/www/discourse/plugins/discourse-chatbot/gems/3.2.1 --no-document --ignore-dependencies --no-user-install
When you HTTParty, you must party hard!
Successfully installed httparty-0.21.0
1 gem installed
gem install ruby-openai -v 3.4.0 -i /var/www/discourse/plugins/discourse-chatbot/gems/3.2.1 --no-document --ignore-dependencies --no-user-install
Note if upgrading: The `::Ruby::OpenAI` module has been removed and all classes have been moved under the top level `::OpenAI` module. To upgrade, change `require 'ruby/openai'` to `require 'openai'` and change all references to `Ruby::OpenAI` to `OpenAI`.
Successfully installed ruby-openai-3.4.0
1 gem installed

I, [2023-03-12T17:24:38.021523 #1]  INFO -- : Terminating async processes
I, [2023-03-12T17:24:38.022106 #1]  INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main pid: 41
2023-03-12 17:24:38.022 UTC [41] LOG:  received fast shutdown request
I, [2023-03-12T17:24:38.022874 #1]  INFO -- : Sending TERM to exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf pid: 102
102:signal-handler (1678641878) Received SIGTERM scheduling shutdown...
2023-03-12 17:24:38.035 UTC [41] LOG:  aborting any active transactions
2023-03-12 17:24:38.046 UTC [41] LOG:  background worker "logical replication launcher" (PID 50) exited with exit code 1
2023-03-12 17:24:38.049 UTC [45] LOG:  shutting down
102:M 12 Mar 2023 17:24:38.053 # User requested shutdown...
102:M 12 Mar 2023 17:24:38.054 * Saving the final RDB snapshot before exiting.
102:M 12 Mar 2023 17:24:38.120 * DB saved on disk
102:M 12 Mar 2023 17:24:38.122 # Redis is now ready to exit, bye bye...
2023-03-12 17:24:38.153 UTC [41] LOG:  database system is shut down


FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && su discourse -c 'bundle exec rake db:migrate' failed with return #<Process::Status: pid 1043 exit 1>
Location of failure: /usr/local/lib/ruby/gems/3.2.0/gems/pups-1.1.1/lib/pups/exec_command.rb:117:in `spawn'
exec failed with the params {"cd"=>"$home", "hook"=>"db_migrate", "cmd"=>["su discourse -c 'bundle exec rake db:migrate'"]}
bootstrap failed with exit code 1

[merefield (Robert)]

Oops my bad. It’s a conflict between my two AI plugins. Will resolve and revert! Apologies!

[merefield (Robert)]

Try that: FIX: conflict with chatbot · merefield/discourse-ai-topic-summary@0067eaf · GitHub

[JamesPick]

Many thanks for getting back so quickly. I will try this tomorrow. I’m looking forward to seeing this plugin in action!

[JamesPick]

Quick update I had my first real summary triggered and it did a pretty reasonable job. @merefield I think you’ve nailed this plugin :-). I think in fact putting chatgpt to work like this has far better future potential than just adding into a discussion thread.

Here’s the summary that was generated which placed at the top of the topic does a great job at encouraging others to read - or not.

I did get an error (error 500) on posting an update that would have triggered the summary. This was because I’d tried to use gpt-3.5-turbo in settings. Perhaps if there is a chatgpt api error it would be better to allow the post submission rather than the user getting a 500 error they don’t know what to do with.

In the logs it was clear what I’d done wrong

StandardError (This is a chat model and not supported in the v1/completions endpoint. Did you mean to use v1/chat/completions?)

Potential evolutions:

• Allow topic owner / admin to remove summary if it isn’t relevant / appropriate?
• Perhaps a place to go to view all summaries in a category / overall

Other than that I look forward to seeing how this evolves. Many thanks for what you’ve done so far. My site is very small with only a few users but I can really see it’s potential.

[merefield (Robert)]

Great feedback thanks. Those are nice ideas.

@JamesPick

Allow topic owner / admin to remove summary if it isn’t relevant / appropriate?

Yeah an admin/author button might be quite nice. I’ll add to roadmap.

@JamesPick

Perhaps a place to go to view all summaries in a category / overall

Nice!

I wonder if “the place to see the summaries” list is actually an alternative excerpt on the topic list?

I’ll add to roadmap.

@JamesPick

I did get an error (error 500) on posting an update that would have triggered the summary. This was because I’d tried to use gpt-3.5-turbo in settings.

Yeah I should convert this to a drop-down like I did with Chatbot. Good try though

GPT-4 is coming btw which will vastly improve the summaries and cope with much longer Topics.

[hoangphuctran93 (Hoàng Phúc Trần)]

Can you tell me if the plugin currently supports ChatGPT 4 and the API of the POE application (AI chat platform developed by Quora)?

[merefield (Robert)]

@hoangphuctran93

ChatGPT 4

Lol, give me a chance! afaia it was only announced yesterday?

Very few developers have been given access to the API so far. There is a waiting list. So users of the plugin are unlikely to be able to connect in any case.

(Did you notice the Topic title btw?)

@hoangphuctran93

the API of the POE application (AI chat platform developed by Quora)?

No and no plans to at present. What would be the benefit? Is it known to excel at summarisation?

[hoangphuctran93 (Hoàng Phúc Trần)]

@merefield

Lol, give me a chance! afaia it was only announced yesterday?

^^
sorry for being unclear ^^ i just wanted to know if you are interested about integrating them in the future? I’m following the news around GPT 04 and it’s surprising and special to us Vietnamese community

[merefield (Robert)]

Thanks for you interest. As far as GPT-4 support is concerned, let me respond this way:

I will probably be adding it to Chatbot first, but will surely get around to AI Topic Summary soon after.

I’m waiting for the email from Open AI … (and then a little spare time!)

[merefield (Robert)]

@JamesPick

Allow topic owner / admin to remove summary if it isn’t relevant / appropriate?

Actually, I’ve thought of a slightly simpler solution to this so we don’t have to crowd the UI:

How about we just give settings for “vote weight” for Admin and Author?

So you can set the Author voting weight to 3 and if the threshold is 3 that results in a new summary being generated.

The downside of both solutions (button, or weighting that matches the threshold) is that the Author could just spam for new summaries, costing you a lot in API calls.

So I suggest you make the authors vote carry a weight just below the threshold (but more than a regular user) and the admin at threshold?

Then we don’t have to change the UI and just bake this into the back-end logic and settings.

Admin downvotes will always create a new summary and reset the count.

Thoughts @JamesPick ?

[JamesPick]

I agree with your concern about spamming of new summaries.
I think it’s less about the ai providing a bad summary and more that the topic is not conducive to a good summary. Perhaps the topic lost it’s focus or became more a chat and the key posts get lost from what is sent to chatgpt.

Ultimately when admin / topic owner clearly sees that the summary is not going to produce anything of meaning for the topic they are able to cancel it.

I suppose in time the features could be added to select the specific posts that could be chosen to send to api for summary.

I was thinking of this for something like this thread. You could flag your first post to be the one that always gets included in a summary? And then perhaps add a few more that most suit a summary.

You could even have a ‘generate summary’ option where even when a topic only has one post a summary can be generated. This fits well with your earlier comment

@merefield

I wonder if “the place to see the summaries” list is actually an alternative excerpt on the topic list?