When you remove someone from a group that grants a trust level it recalculates their trust level and sets it to what it would be without the group bump. 
The trust levels are adjusted on both group add and group remove.
But unfortunately that does mean, currently at least, that anyone added before the extra trust level perk was set won’t be automatically granted it.