Custom User Field based on Group

i am searching for a way to enable / disable a Custom Field for Users based on their group.
I searched in the config, but did not find anything.

The only thing that i can think of is to use some CSS to hide the field in the UI.
Should be hidden under [domain]/u/[username]/preferences/profile.

Any ideas or how to?