Usage not showing LLM models

Fout
1

As the heading says, the Usage tab is supposed to help us figure out which features are using which models. However instead of listing the Configured LLM names, it’s listing the model id.

See this example, the Usage is showing under All Models → gemini-flash-latest and gemini-flash-lite-latest

image
image1216×625 43.7 KB

If I see the Configured LLM’s page I don’t see any such names

image
image1362×935 26.9 KB

So now I’m not even sure if the usage information is per Configured LLM or it per LLM model id? The purpose of using multiple Configured LLM’s (using different keys and projects) is to avoid overloading the LLM’s if one feature is taking too many requests or token it doesn’t disrupt other features.

2

It should be across features, but I see what you are reporting here.

You are creating duplicate LLMs that point at the same model and want it broken down by that. Interesting use case. The counts will be right but you may not be getting the split you want.

We will think about a way of cleaning this up.

2 likes
3

I have this sorted now, the reports should be :chefs_kiss: Update to latest.

That said, we simply did not have proper storage in the past (we did not store llm_id in the audit log) so the report will only be 100% correct for you going forward.