I’d also like to make the point that it’s actually not such a specific use case.
Just as @mattdm already pointed out, the color is not only used in the badges or the icons. It can be used in banners and practically any page element when combined with the Colorful Categories component. Another advantage of using colors right from the category settings is that you enable community managers to change colors used in a theme themselves, rather then needing to reach out to the theme designer just for that.
Right now when you don’t want to use the core badges with colors, but still use category colors in other theme elements, the only way to change the category color is to enable a different badge style, change the color, then re-enable none as badge-style.
I saw @codinghorror mentioned this concern in another topic about it:
But as said, it can have many effects in a theme. Also, there’s a hint already, stating:
So why not keep the setting available and rather give a hint like:
You might not see colors because you have a category style of none.