After giving this some more thought, I’m not sure that a new site setting would be required. Instead, the group’s “Who can message this group” setting could be respected and the values of the enable personal messages and min trust to send messages site settings could be ignored for sending messages to a group. Possibly some more options could be added to the “Who can message this group” setting on the group’s page. For example, messaging a particular group could be limited by trust level.
Somewhat related to this topic, in some ways the enable personal messages setting seems to be redundant. If a “staff” option was added to the min trust to send messages setting, there might not be a need for the enable personal messages setting.