Yeah this is exactly the problem - we don’t (yet) have any automatic tree-shaking. Admin modules are separated out manually so that regular users don’t have to pay the cost of loading them over the network.
That’s probably the best bet. IIRC the color selector is fairly self-contained so it should be easy to copy. That also has the advantage of insulating your plugin from any changes made to the core component (which isn’t really considered “public api”)