I suggest collecting and displaying information about when each access was last used, ideally per granted permission. (This data could also be used to automatically revoke unused accesses after some time.)
We only started tracking “last use” after this change. This means that the user profile says “Last use: 8 hours ago” on some keys you may have not used in years.
This will “fix itself up” over time, no real choice here but to start somewhere.