We have many topics that exceed typical context lengths (say 128k) and bad things happen. Have you considered summarizing based on only the most-liked posts? Start with the most-liked post, add the next-most-liked, and continue until the target context length is reached, then pass the resulting text to the GPT? For our forum at least, “likes” are a very good proxy for “most relevant”.
That is how it works on long topics, we use the top 50 most liked posts in the topic.
1 Like
Hmm, summarizing appears to hang on long topics, so I assumed it was related to the content length. Any tips for debugging this? I’m using gpt-o4. I don’t see a way to view specific API requests in my OpenAI dashboard.
You can see requests on the ai_api_audit_logs table.
Regarding the hanging mid summarization, we had some front end bugs on that front recently, so please update your site to latest.
2 Likes
I found this in the error logs:
Request too large for gpt-4o in organization org-xxxxxxxxxxxxxxxxx on tokens per min (TPM): Limit 30000, Requested 45906.
To increase my limits, I believe I have to actually spend $50 (not just budget it) and wait something like a month plus 7 days:
So far I’ve only spent $2.00 and no payment is due yet. I also don’t see how to cap the tokens in Discourse. So I don’t see how to use summarization for long topics at this point.
Ohhh so the issue is that you have a restricted OpenAI account that limits you on tokens per minute, which impact using AI summarization on Discourse.
That’s is such a bummer. I’d look into alternative providers, like Antrhopic, AWS, Azure, Google Cloud or even Llama API providers. I believe there are providers with a free tier that will give you a higher limit than that.
2 Likes
I switched from using 4o to 4o-mini for summarization, and that appears to work on my longest topics. Apparently OpenAI has different limits for the different GPTs.
2 Likes