I agree with this, and I’ve definitely received questions and feedback from my site’s users who are getting the same impression.
As already suggested, it might be good to limit where it’s displayed to only the main pages and not within categories. OR have it so it’s a global leaderboard on main pages and it shifts to a category leaderboard when within a category. Either way would be an improvement to top contributors.