It’s not at the moment, no.
We’ve had a few requests lately to make groups ‘private’ so that only group members know who is in the group. I think that would make a lot of sense for several use cases.
Your request for limiting PMs to group members would be a natural extension of that. I’m not sure if we’ve actually road mapped this though. @jomaxro can you remember?