Topic Summarization doesn't work :-(

Support
1

Hey guys,

This is about a selfhosted instance

I´ve activated the Discourse AI Plugin and had a few problems there.
Working in an enterprise environment and AI access is though an internal endpoint with additional authentication.
I implemented an AWS API Gateway endpoint with a Lambda, that takes care of enriching the headers with the required auth info and pass the request on to the OPENAI compatible internal endpoint.

The

The image depicts a settings page for Large Language Models (LLMs) in a software application, detailing how they are used and listing a configured LLM named GPT-4o, associated with the vLLM provider. (Captioned by AI)
The image depicts a settings page for Large Language Models (LLMs) in a software application, detailing how they are used and listing a configured LLM named GPT-4o, associated with the vLLM provider. (Captioned by AI)1457×433 27 KB

LLM Test gives me a

image
image536×127 5.39 KB

and I thought I´m good to go.

I also see the related calls in my CloudWatch Logs for the API Gateway call.

Then I related the Summarization Personas (Smmmarization and … (short form) with this LLM and activated the Topic Summarization.

What I´m seeing now, though, is not what I wanted to see :smiley:

The image shows a summary suggestion tool within a writing application, highlighting a template and options to sort by votes or activity. (Captioned by AI)
The image shows a summary suggestion tool within a writing application, highlighting a template and options to sort by votes or activity. (Captioned by AI)1394×664 13 KB

And that´s not changing :frowning:

The strangest thing is, that I don´t see a request in CloudWatch Logs for the attempt to summarize.

Ok, again what I did.

  • Activated AI
  • Created vLLM with my API Gateway endpoint
  • Created 2 Personas (see above) and related them with my LLM config
  • Activated Summarization and related there the Persona (tried both)

Not sure what I can try next :frowning:

Thanks guys and Greetings,

JP

2

Our test endpoint is a bit misleading and this is something I would like to fix.

We currently only test “non streaming” endpoints. “streaming” is not tested.

Does everything work if you disable streaming on your LLM, maybe it is not implemented by your proxy.

3

Hey @sam,

I disabled the streaming (hard coded) in my LAMBDA while passing the request on.

Thing is, that the LAMBDA doesn´t even get triggered by the summarization API (i see no requests).

The LLM Test issues a call, but the summarization button in a topic doesn´t.

Nothing, no errors (i switched debug level on with DISCOURSE_LOG_LEVEL=debug) in discourse log, nor any logs in my LAMBDA (CloudWatch).

As if I didn´t press anything at all :frowning:

Thanks and Cheers,

Julian

4

Why is the provider set to vLLM if it is an OpenAI compatible endpoint?

1 Like