Discourse AI Topic Summary : automated summaries and smart tagging

I agree with your concern about spamming of new summaries.
I think it’s less about the ai providing a bad summary and more that the topic is not conducive to a good summary. Perhaps the topic lost it’s focus or became more a chat and the key posts get lost from what is sent to chatgpt.

Ultimately when admin / topic owner clearly sees that the summary is not going to produce anything of meaning for the topic they are able to cancel it.

I suppose in time the features could be added to select the specific posts that could be chosen to send to api for summary.

I was thinking of this for something like this thread. You could flag your first post to be the one that always gets included in a summary? And then perhaps add a few more that most suit a summary.

You could even have a ‘generate summary’ option where even when a topic only has one post a summary can be generated. This fits well with your earlier comment

2 Likes

I’ve been trying to put summary into a working flow a couple of things I’ve found:

  • There is no reason not to have a summary from the first post as the summary will just evolve as the topic increases. Of course those refreshes limited by settings.
  • When you are working on a topic the summary is positioned at the top of the page but the user will always be sent to the bottom (last post, last post read). So invariably the summary will never be read by these users. Its a really cool thing to have but I suspect never seen as a user will never / have no reason to scroll all the way back up to the top to see what the summary says even if it’s useful. Could the summary be positioned on the right column underneath the toc / date line? So that it’s always visible?
1 Like

Except cost and having too much duplicate information at same time taking up space?

Surely a summary adds most value once things have really got going and you have several Posts at least?

In any case, I suspect you can just set your thresholds as you please here, and don’t need any action from me?

Are you sure existing readers will bother reading the summary? I note that the summary is probably more valuable to new readers who will see the summary first and can use it as a guide to read further down?

This is a nice idea though so long as the width of the presentation was wide enough to make reading comfortable? That may be a challenge?

My experimental solution would be to leverage the Layouts Plugin and create a compatible widget that can appear on the right or left of a Topic (indeed if we add Top support we can put it there too in the original location!).

Note that there are other ways to deal with this:

  • add a move to top button (there’s a theme component). You could even modify that to include a label saying “go to summary”?

I’ll add it to roadmap in any case, but let me know what you think of above.

2 Likes

Agree on the cost aspect but if it was ever going to be used as the ‘alternative excerpt’ a summary would have to be generated. I’m not sure I was able to get it to work by updating settings. Perhaps I missed something I will go back to look.

Very good point… I suppose time will tell. But if users start to get comfortable with the summary then I see overtime them taking a quick glance. In fact if some users are working on a number of topics I think it’s a good quick memory jog of what’s happened. Perhaps in the future a user could click to generate a summary of what’s happened since they were last in a particular topic? (Again cost permitting on the api call)

I like this idea yes. Better than the button to jump to top as you then need to go back to where you were before (which might not be the last post)

2 Likes

OK version 0.1 of that is done:

You will need to install the Layouts plugin and configure the right sidebar for it to work.

There is now an additional setting in the plugin to switch off the standard summary at the top.

As an almost free by-product this means that the summary can be invoked on mobile in a convenient slide-drawer.

@JamesPick

4 Likes

It’s brilliant! :tada:

Except I just had summary created that contained details of an internal “Whisper” post. Any chance these posts could be excluded from what gets sent to openai?

3 Likes

Good catch. I’ll address that.

2 Likes
  • Only summarise normal user posts (ignoring “staff actions” (like “Topic Opened”) and whispered posts)
  • Allow prompt engineering via locale settings ai_topic_summary.prompt.* on Admin → Customize → Text
    • Get rid of now redundant Site Setting
  • Make links to openai in settings clickable (thanks @MarcP)
  • Make model selection a nicer drop down

@JamesPick

5 Likes

Thanks for this.
I do have a question though are you excluding all admin posts? Does this mean it’s good practice for an admin user that posts to have 2 separate accounts one the admin and then other for posting and interacting with the community? On my site I often post to topics and would want that to be included in the summary (I am a site admin).

2 Likes

Let me clarify:

“Admin” posts such as “Topic Closed”, “Topic Opened” etc. These are actually Posts in the post stream believe it or not.

Not regular Posts posted by users that happen to be moderators or admins.

2 Likes

Many thanks for that clarification

2 Likes

No problem. Thanks for helping me make this better.

2 Likes

Thanks. I’ve tried to install and start this plugin, but tries to leave a comment in some posts returns HTTP error 500. In production.log i see the message:
StandardError (This model’s maximum context length is 4097 tokens, however you requested 15682 tokens (15482 in your prompt; 200 for the completion). Please reduce your prompt; or completion length.)

1 Like

Quite, so … you need to reduce the amount your are summarising using the settings.

I don’t believe this is a bug, you are simply being too greedy :hamburger: :hamburger: :wink:

You need to tune the settings to match your setup. There’s a Post and a Character limit you can control. A token is usually 4 chars.

It’s possible the defaults are a bit too generous … but they are set to approximate the token limit, so this is a bit strange.

Can you please share these settings?:

Follow the information in the error to guide your changes.

GPT-4 (when I get access) will up to 8x the token limits, so look forward to that.

1 Like

Merged:

  • now we strip quotes from posts to avoid duplication and optimise token budget usage
  • If you preferred prior behaviour you can turn off the new setting.
3 Likes

@merefield quick question on the language model you are using for this. Why didn’t you decide to enable gpt-3.5-turbo?

We have been testing ourselves on the use of gpt-3.5 for email and phone transcription summaries and it does a very good job. The email message / transcription itself we put in the user content.

For phone transcription for example we add the following into the ‘system’ content

Identify if this transcript is either a telephone conversation or if the caller is leaving a voicemail.
If it is a telephone conversation then you should provide a summary of the telephone conversation and then create the following sections with bulleted items

  • Agreements Made
  • Questions Raised
  • Possible inconsistencies / concerns
  • Sentiment score: (for each participant)
  • Team Lead Review: (Please flag if this call should be reviewed by a team leader)

If it is a voicemail then you should provide a title ‘Voicemail left [for]’ and on a new line summarise the voicemail with one sentence

1 Like

Because it was optimised for chat and at the time Open AI were still recommending Davinci 003 for Summarisation. Happy to review this when GPT-4 stuff becomes available more broadly.

Note that ChatGPT and Davinci-003 are the same generation, so I’m not sure from a results perspective there is much between them?

1 Like

Have you analized the possibility to summarize content from a link and propose the post befor it was sent?

I tried to chat to ChatGPT chatbot for that kind of tasks but it just goes crazy and I was need to stop trying that way.

The workflow could be completed at the end with something like

Content generated by ChatGPT

I see a lot of forums with issues around people that are using ChatGPT in their literal replies (so a lot of people are almost robots).

Remote content not currently in scope. Maybe something for future.

1 Like

Here’s a thing - twice in the past week I’ve read some kind of contribution, to find at the end a disclosure that it’s from a LLM. Both times I felt bad - I’d invested in understanding the sense of something thinking that there was a somebody behind the words.

So I would recommend (strongly!) that the disclosure should come ahead of the text.

1 Like