This does seem to be causing an issue in one of my plugins on 3.2.1 stable.
When I inspect Site.current().categories and lazy loading is enabled, I do see (only) the categories I need (like the category for the current topic), but all preloaded_category_custom_fields seem to be missing.
They do appear when I turn off lazy loading (together with all the other categories).
So the correct category is being preloaded but it misses its preloaded fields.