Do you use AI Summaries?

Feature
, ,
1

:information_source: Context

AI summaries have been out for some time now. We’ve seen some early feedback but wanted to revisit the discussion. As we are thinking about investing in further development of AI Summaries, I’m trying to understand user feedback starting with us right here in Meta

Are you using AI Summaries here and in your communities?

  • Yes, all the time
  • Sometimes, depending on the topic
  • No, it’s not useful for me
0 voters

:speaking_head: Feedback

Feel free to speak your mind and be honest. Some guiding questions I’m interested in knowing…

  • How often are you using it?
  • How useful is the generated summary? do you find it accurate to what you’ve read?
  • Do you find it easy to access?
  • If no, why?
1 Like
4

I would say “no” but I think the language in the poll is a bit presumptive. It’s not a conscious decision on my part. It’s moreso that in the context where it could be useful, I’ve yet to organically think “oh yeah I should use AI summarization”

3 Likes
5

My issue is two folded:

  • language; english isn’t working language on my forum even practically everyone can read it
  • I can’t tune system prompt, because now it is quite often too long, too hallucinated and/or there is unrelevant things

I can go around language barrier, though. I will ask a GPT that uses same model than summarising something in finnish, coming back and now if I generate summary it will be in Finnish.

2 Likes
6

I use it once every few days. As a moderator, I read every topic. However, the summaries are not always useful for me but are created for others.

At times very useful, at other times it would miss critical information. The summaries sometimes percolate up the noise rather than the signal. If I really need information I don’t rely on the summaries to help me decide, I read the entire topic as there may be a diamond in the rough and the summary will miss it.

I often use GPT-4-based summaries, and since they update the models periodically, there are instances when it appears that the language model is struggling to comprehend the topic.

Not always. Since I read many posts, often when I return to a post, I find myself at the bottom of the topic rather than at the top where the button is. This often requires me to scroll to the top to access the button. It would be more convenient to have it also located at the bottom.

2 Likes
7

I do understand hallucinations, language support and accessibility can be big deterring factors from using it at all

On the point of the LLM not comprehending and/or giving incorrect responses, perhaps smaller and more concise summaries might help combat some of that :thinking:

1 Like
8

I wish something more creative (not lineal) for catch attention, like telling a story on 1/5 messages and/or asking for feedback using polls for engagement.

Sometimes chosing single topics using analytics (switching between the top X hot*, by visits, likes).

With those features I think that could replace the e-mail sent by Discourse, giving a refresh to this feature too.

I like the feature, there is room to improve :slight_smile:

9

I didn’t answer the poll, because my answer is different. I don’t know how much it will cost to connect my forums to an AI API, and haven’t looked into it. When I looked into translation, it was too expensive, so I’m assuming this could get expensive too.

I haven’t used it here, because the topics aren’t long and I’m usually looking for something specific.

10

I’ve got an approach for parsing long topics that works for me:

  • I’m fairly good at speed reading
  • I find the “(n) posts in topic” button that’s on the usercard super useful

In general I find AI generated text off-putting.

I’m sure I’d use AI summaries in a pinch. Other than clicking the button out of curiosity, that situation hasn’t come up yet.

What I’d really like to see is a mechanism for limiting the number of posts and users in a topic. If that was done, there’s be less of a need for AI generated summaries.

2 Likes
11

One of my hopes for Discourse AI integration is that it will allow conversations to occur in a natural, human manner while still allowing a forum to function as a knowledge base for users who have not been following the full conversation. Semantic search (especially when searching within a topic) will help with that. Another approach would be to use AI generated topic summaries to break a topic into subjects.

For topics that have a sufficient number of posts, the summary could outline the OP, then have a “subjects discussed in this topic” section. In a similar way to how clicking the “(n) posts in topic” button on the usercard extracts that user’s posts from the full topic, each subject discussed could have a button that allowed the sub-conversation to be extracted from the full topic.

This might also be useful for moderators who were wanting to separate long topics into multiple topics.

Note that this is less testable on Meta than it used to be. An effort has been made over the past few years to keep topics relevant to the OP.

2 Likes
12

I think has certainly been one of our North Stars as we develop Discourse AI

I visualize this in my head as sort of like thought bubbles that pop-up depending on relevant themes for a topic

1 Like