putty asks for recommendations to reduce Agent response time for a single conversation, noting that aside from trimming prompts or RAG, Thinking Effort appears to be the primary lever. They use GPT Sol 6.1 with product documentation and are considering using subagents to split product versions for a streamlined RAG.
Reasoning Effort: Use “low” effort for Sol, which is often capable enough.
Service Tier: Enable the priority tier (Fast mode) for 2.5x speed, with an ultrafast option possible.
Prompt Caching: Move dynamic prompt parts to the end to maximize cacheable content and reduce prefill time.
RAG vs. Hardcoded Prompt: For small knowledge bases, hardcoding information may be faster than dynamic loading.
Faster Tools: Optimize custom tools used for dynamic grounding.
Smaller Models: Switch to Luna if acceptable quality trade-offs exist.
Regarding subagents, Falco advises that while they can help, they are labor-intensive and hard to maintain, suggesting they be used only after simpler optimizations are exhausted.
putty shares test results showing significant improvements through experimentation:
Sol with high thinking: 56.3s (too slow).
Sol with low agent thinking effort: 18.9s, but caused quality degradation on specific questions.
Luna (smaller model) with High effort: 35.2s, but weaker knowledge usage.
Luna + Quick Facts (hardcoded in prompt): 24.8s–29.6s, restoring correct answers for collection and root-DC questions.
Adding troubleshooting checklist: Further reduced response time to 24.0s for most turns, though complex turns still took ~27s.
The results highlight that combining a smaller model (Luna) with hardcoding critical “Quick Facts” into the prompt offered the best balance of speed and accuracy compared to adjusting thinking effort alone or relying solely on RAG chunk reduction.
What settings do you recommend tweaking or increasing to reduce response time? Assume I’m only talking about a single conversation, not dozens running at the same time. Aside from trimming down the prompt/RAG, it seems like Thinking Effort is the only setting that really affects timing for an individual conversation.
For background, I have an agent equipped with GPT Sol 6.1 and loaded with our product documentation, knowledge base articles, and some “gotcha” questions and answers.
I just noticed a subagent setting. I wonder if I can use this to split up some product versions/components and streamline the RAG?
Reasoning effort
Sol on low is very capable; I use it extensively for many tasks, so I’d try that first.
Service Tier
Discourse also supports the service tier priority, which is the same as “Fast mode” and makes it 2.5 times faster. We could add ultrafast as an option there if you want it faster.
Prompt caching
Move all the dynamic parts of your prompt as late as possible to increase what is cacheable and reduce prefill time.
Experiment with the RAG + tools vs hardcoded prompt tradeoff
Depending on how big your KB is, sometimes using a very large prompt will be quicker and cheaper than dynamic loading of knowledge.
Faster tools
If you, like we do on ask.discourse.com, make extensive use of custom tools for dynamic grounding sources, optimizing those to respond ASAP is very beneficial.
Smaller Models
Depending on your use case, switching to Luna may close the speed gap with acceptable quality.
It can definitely help, but it is also the hardest one in terms of labor, and maintenance going forward, so I’d reach for it only if it still doesn’t feel right after you explored the low hanging ones.
Thank you for the ideas! I think we’re in a much better spot now. I tweaked some settings and ran the same set of questions. Here are some results of my testing today: