How to speed up Agent's response time?

AI-generated summary

In the Product Support discussion, putty seeks advice on reducing Agent response time for a single conversation powered by GPT Sol 6.1 and extensive RAG. Falco provides several optimization strategies:

  1. Reasoning Effort: Try lowering the setting (e.g., Sol on low) for better speed.
  2. Service Tier: Enable the priority tier (“Fast mode”) for a 2.5x speed increase, with ultrafast potentially available.
  3. Prompt Caching: Structure prompts to maximize cacheability and reduce prefill time.
  4. RAG vs. Hardcoded Prompt: Depending on the knowledge base size, a large static prompt may be faster than dynamic RAG loading.
  5. Optimize Tools: Ensure custom tools for dynamic grounding are highly efficient.
  6. Model Choice: Consider switching to a smaller model like Luna if quality trade-offs are acceptable.

putty also asks about using subagents to split up product components and streamline RAG. Falco cautions that while subagents can help, they are complex and high-maintenance, recommending they be considered only after trying the easier optimizations above.

What settings do you recommend tweaking or increasing to reduce response time? Assume I’m only talking about a single conversation, not dozens running at the same time. Aside from trimming down the prompt/RAG, it seems like Thinking Effort is the only setting that really affects timing for an individual conversation.

For background, I have an agent equipped with GPT Sol 6.1 and loaded with our product documentation, knowledge base articles, and some “gotcha” questions and answers.

I just noticed a subagent setting. I wonder if I can use this to split up some product versions/components and streamline the RAG?

  1. Reasoning effort
    Sol on low is very capable; I use it extensively for many tasks, so I’d try that first.

  2. Service Tier
    Discourse also supports the service tier priority, which is the same as “Fast mode” and makes it 2.5 times faster. We could add ultrafast as an option there if you want it faster.

  3. Prompt caching
    Move all the dynamic parts of your prompt as late as possible to increase what is cacheable and reduce prefill time.

  4. Experiment with the RAG + tools vs hardcoded prompt tradeoff
    Depending on how big your KB is, sometimes using a very large prompt will be quicker and cheaper than dynamic loading of knowledge.

  5. Faster tools
    If you, like we do on ask.discourse.com, make extensive use of custom tools for dynamic grounding sources, optimizing those to respond ASAP is very beneficial.

  6. Smaller Models
    Depending on your use case, switching to Luna may close the speed gap with acceptable quality.

Thank you! I’ll definitely try all of these.

Any thoughts on the effectiveness of subagents in this case? That would really help me trim some fat on the RAG.

It can definitely help, but it is also the hardest one in terms of labor, and maintenance going forward, so I’d reach for it only if it still doesn’t feel right after you explored the low hanging ones.