The discussion revolves around a bug in the Usage tab, where instead of displaying configured LLM names, it shows model IDs like gemini-flash-latest and gemini-flash-lite-latest. RBoy reports that the Usage tab incorrectly lists model IDs rather than the configured LLM names, which makes it difficult to track usage by feature. sam acknowledges the issue, noting that duplicate LLMs pointing to the same model are causing confusion. sam confirms the issue has been resolved and advises updating to the latest version, explaining that the problem was due to improper handling of model IDs in the reporting system.
As the heading says, the Usage tab is supposed to help us figure out which features are using which models. However instead of listing the Configured LLM names, it’s listing the model id.
See this example, the Usage is showing under All Models → gemini-flash-latest and gemini-flash-lite-latest
So now I’m not even sure if the usage information is per Configured LLM or it per LLM model id? The purpose of using multiple Configured LLM’s (using different keys and projects) is to avoid overloading the LLM’s if one feature is taking too many requests or token it doesn’t disrupt other features.
It should be across features, but I see what you are reporting here.
You are creating duplicate LLMs that point at the same model and want it broken down by that. Interesting use case. The counts will be right but you may not be getting the split you want.
I have this sorted now, the reports should be Update to latest.
That said, we simply did not have proper storage in the past (we did not store llm_id in the audit log) so the report will only be 100% correct for you going forward.
Update to latest.