So, as the PM owner with trust level 2 or higher it is possible to add people to the PM but not remove them? 
I may be misremembering, but I think the PM owner (assuming correct trust level) should be able to add or remove people from the PM at will?