Add {username} as a template param in AI persona system prompts

I discussed this with @falco as well, but would like to warn that this is a very sharp implement.

Particularly we already include the user name in the “User” message:

This is visible from in the ai api logs.

The danger around inserting name at the system prompt is that in cases where there is a multi user interaction in a chain then you can cause system prompt thrashing which causes expiry of cache and which is not ideal and can lead to bad perf.

Now … one thing I noticed in our vllm templates is that we are throwing away name, that explains why CDCK hosted models are missing this feature. I am working on a patch now to fix this.

3 Likes