To me, it seems as if the data the color picker needs may not be available when lazy-loaded categories are enabled. It only shows the colors of the categories in my sidebar as used.
Yes Im aware there are limits with lazy loaded, Im not sure we are going to fix them though. Lazy loaded is designed for A LOT of categories, and no one is going to look at hundreds of colors. We need a different system here if we want to support this, which I dont think will be a priority, sites requiring lazy loading of categories are quite rare in the big scheme of things.
Is it possible to simply not separate the colors when lazy loaded categories are enabled?
Testing with lazy loaded categories is also more of an oversight. It’s now difficult to test without the setting being enabled because I can’t easily see which groups it’s enabled for.