How to speed up Agent's response time?

AI-generated summary

putty asks for recommendations to reduce Agent response time for a single conversation, noting that aside from trimming prompts or RAG, Thinking Effort appears to be the primary lever. They use GPT Sol 6.1 with product documentation and are considering using subagents to split product versions for a streamlined RAG.

Falco provides six optimization strategies:

  1. Reasoning Effort: Use “low” effort for Sol, which is often capable enough.
  2. Service Tier: Enable the priority tier (Fast mode) for 2.5x speed, with an ultrafast option possible.
  3. Prompt Caching: Move dynamic prompt parts to the end to maximize cacheable content and reduce prefill time.
  4. RAG vs. Hardcoded Prompt: For small knowledge bases, hardcoding information may be faster than dynamic loading.
  5. Faster Tools: Optimize custom tools used for dynamic grounding.
  6. Smaller Models: Switch to Luna if acceptable quality trade-offs exist.

Regarding subagents, Falco advises that while they can help, they are labor-intensive and hard to maintain, suggesting they be used only after simpler optimizations are exhausted.

putty shares test results showing significant improvements through experimentation:

  • Sol with high thinking: 56.3s (too slow).
  • Sol with low agent thinking effort: 18.9s, but caused quality degradation on specific questions.
  • Luna (smaller model) with High effort: 35.2s, but weaker knowledge usage.
  • Luna + Quick Facts (hardcoded in prompt): 24.8s–29.6s, restoring correct answers for collection and root-DC questions.
  • Adding troubleshooting checklist: Further reduced response time to 24.0s for most turns, though complex turns still took ~27s.

The results highlight that combining a smaller model (Luna) with hardcoding critical “Quick Facts” into the prompt offered the best balance of speed and accuracy compared to adjusting thinking effort alone or relying solely on RAG chunk reduction.

What settings do you recommend tweaking or increasing to reduce response time? Assume I’m only talking about a single conversation, not dozens running at the same time. Aside from trimming down the prompt/RAG, it seems like Thinking Effort is the only setting that really affects timing for an individual conversation.

For background, I have an agent equipped with GPT Sol 6.1 and loaded with our product documentation, knowledge base articles, and some “gotcha” questions and answers.

I just noticed a subagent setting. I wonder if I can use this to split up some product versions/components and streamline the RAG?

  1. Reasoning effort
    Sol on low is very capable; I use it extensively for many tasks, so I’d try that first.

  2. Service Tier
    Discourse also supports the service tier priority, which is the same as “Fast mode” and makes it 2.5 times faster. We could add ultrafast as an option there if you want it faster.

  3. Prompt caching
    Move all the dynamic parts of your prompt as late as possible to increase what is cacheable and reduce prefill time.

  4. Experiment with the RAG + tools vs hardcoded prompt tradeoff
    Depending on how big your KB is, sometimes using a very large prompt will be quicker and cheaper than dynamic loading of knowledge.

  5. Faster tools
    If you, like we do on ask.discourse.com, make extensive use of custom tools for dynamic grounding sources, optimizing those to respond ASAP is very beneficial.

  6. Smaller Models
    Depending on your use case, switching to Luna may close the speed gap with acceptable quality.

Thank you! I’ll definitely try all of these.

Any thoughts on the effectiveness of subagents in this case? That would really help me trim some fat on the RAG.

It can definitely help, but it is also the hardest one in terms of labor, and maintenance going forward, so I’d reach for it only if it still doesn’t feel right after you explored the low hanging ones.

Thank you for the ideas! I think we’re in a much better spot now. I tweaked some settings and ran the same set of questions. Here are some results of my testing today:

A little additional info
Change Avg time Verdict
Original: Sol model, high thinking 56.3 s Good answers, too slow
Model reasoning set to low 47.6 s Small gain
Agent thinking effort low 18.9 s Fast, but it stopped answering collection-frequency questions
Agent thinking effort medium 29.8 s Same quality problem, no good middle
Fewer RAG chunks (7 to 3) 54.8 s No gain, and one hard_overflow error
Smaller model (Luna) at high effort 35.2 s Faster, but weaker use of the support knowledge
Luna + Quick Facts block in the prompt 29.6 s and 24.8 s Collection and root-DC answers correct again
+ report-impact line 25.6 s Report impact now correct
+ troubleshooting checklist 24.0 s Group-change turn 49.7 s to 27.2 s

Quick Facts was offloading some answers to the prompt out of the RAG.