-
Reasoning effort
Sol on low is very capable; I use it extensively for many tasks, so I’d try that first. -
Service Tier
Discourse also supports the service tierpriority, which is the same as “Fast mode” and makes it 2.5 times faster. We could addultrafastas an option there if you want it faster. -
Prompt caching
Move all the dynamic parts of your prompt as late as possible to increase what is cacheable and reduce prefill time. -
Experiment with the RAG + tools vs hardcoded prompt tradeoff
Depending on how big your KB is, sometimes using a very large prompt will be quicker and cheaper than dynamic loading of knowledge. -
Faster tools
If you, like we do on ask.discourse.com, make extensive use of custom tools for dynamic grounding sources, optimizing those to respond ASAP is very beneficial. -
Smaller Models
Depending on your use case, switching to Luna may close the speed gap with acceptable quality.