Feedback on AI Summarisation

UX
,
1

Feedback on the summarisation. This looks very promising :pray:

image
image713×884 31.3 KB

:point_up: the popup is not wide enough

  • It would be great to offer options about how long the summary should be. It sounds like this is targeting the 150-200 word count which is not a lot for some topics.
  • I would love to customise the prompt. For example I would probably default to bullet points.
  • I would love to experiment with a prompt that quote who says what and references posts
  • I would decouple the summarize with AI minimum post setting from the general one. The summarise with AI is probably useful with fewer posts

Note that gpt-3.5 now has a version with a 16k tokens limit which may be very relevant here.

And now the 2 killer features I would like to see:

  • daily summaries with the summarise with AI feature
  • summaries when I hover on the topic (eg pre-computed)
4 Likes
2

We are using a simplistic prompt at the moment (when OpenAI or Anthropic are selected), but will change it very soon as we are working on it at the moment.

Customizing it is an interesting feature, we may add this in the future.

We just made it show for posts with at least one reply this week.

We added for OpenAI large context models the day they were released. Currently you can pick:

  • GPT 3.5 Turbo with 16k
  • GPT 4 with 32k
  • Claude with 100k

There is work underway to cache the generated summaries so they can be used elsewhere, which unlocks features like this.

6 Likes
3

I don’t see that on my site, is there a setting that controls it?

More suggestions:

  • showing the summarize with AI button in the pipeline.
  • have a way to replace normal summarize with summarize with AI
  • Allow to point GPT models at Azure. We have observed significantly improved latencies in my company
2 Likes
4

It’s a change we just made this week, so it will rollout to your site in the next few days.

What “pipeline” here means?

As you can see here on Meta we already did this. Old summary is now called “Show top Replies”. We are still working on the design and order of the buttons.

That is on our roadmap!

5 Likes
5

Sorry for the lack of clarity, I have no idea why I picked that word :facepalm:

I meant here

image
image679×1030 17.6 KB

1 Like
6

This is done per:

It is rather annoying to configure cause you need to create a bunch of deployments, but it does work nicely.

1 Like
7

EDITED

Using Gemini 1.5 Flash, the summaries often contain links to multiple posts, but the multiple links are not rendered as links. I don’t remember seeing other models try to use multiple links.

Links to single posts are rendered properly.

image
image532×227 21.1 KB

The corresponding response payload includes this text:

[Username redacted](/t/-/56915/8, /t/-/56915/13, /t/-/56915/28, /t/-/56915/33, /t/-/56915/36)

I really like the idea of citing multiple links, if they could be rendered properly.

8

Well, that is simply invalid markdown that is why it doesn’t work.

It is a shame Gemini Flash can’t produce valid markdown, here on Meta we use Llama 3.1 quantized down to FP8 for summaries, and it can handle producing proper markdown links just fine.

1 Like
9

And that haiku apparently can’t return a valid status code. Those are the two I’ve tried for summarization so far. I’ll try Llama 3.1 next.

I don’t know if you would consider patching known issues, like checking for Mondays and for this particular markdown. I realize it’s not ideal, but it might be better than losing otherwise good LLM options.

10

Can you see invalid summaries at https://discuss.ai.google.dev/ ?

We are using flash there , if you can link me to a few I would be happy to do a bit of prompt engineering

11

Unreliability of Gemini API - Error while creating cache - Gemini API - Build with Google AI

image
image546×192 20.4 KB

That’s the only one I see offhand, but those topics are short, so any one user would not usually be cited multiple times.

1 Like
12

This looks very fixable to me cause we can improve our example here, at the moment the prompt is:

You are an advanced summarization bot that generates concise, coherent summaries of provided text.\n\n- Only include the summary, without any additional commentary.\n- You understand and generate Discourse forum Markdown; including links, _italics_, **bold**.\n- Maintain the original language of the text being summarized.\n- Aim for summaries to be 400 words or less.\n- Each post is formatted as \"<POST_NUMBER>) <USERNAME> <MESSAGE>\"\n- Cite specific noteworthy posts using the format [NAME](/t/-/31793/POST_NUMBER)\n - Example: link to the 3rd post by sam: [sam](/t/-/31793/3)\n - Example: link to the 6th post by jane: [agreed with](/t/-/31793/6)\n - Example: link to the 13th post by joe: [#13](/t/-/31793/13)\n- When formatting usernames either use @USERNMAE OR [USERNAME](/t/-/31793/POST_NUMBER)

This feels a bit ambiguous. We can ground it with something like this:

Fundementally though @Roman_Rizzi / @Falco we are going to have to evict these prompts from the source code tree and maybe have them in a persona so communities can tweak them and share what works better.

I will not be surprised if the long term is that we need special case prompts for the weaker models.

Persona would work great here if we had support for “examples” which does not feel like a massive stretch. Either that or we move it to CompletionPrompt and give that a UI.

4 Likes
13

I like the idea of customizing the prompt to meet your needs and leverage the model’s capabilities. I also think turning personas into a more feature-agnostic concept is a good idea.

1 Like
14

This is basically the idea of the “persona functions”, “prepared statements” that we’ve been discussing this year. We would move everything like AI Helper, Summary, Search, etc to use those, allow for creating custom ones and have an UI to associate those to features in the UI.

It gets trick when we update the seeded ones to ship a different format, eg XML to JSON, so hopefully something we can consider building next year.

1 Like