Tricky situation. Messages are for the most part private and should pretty much be left alone.
The current approach relies on members reporting bad messages. So how to do damage control?
I don’t think it would be the best idea to send them off to Akismet, I assume they respect confidentiality, but maybe not.
I guess you could tweak up min trust to send messages but that would be punishing the good members as well as the bad.
Likewise with tweaking down max private messages per day
If you have changed any of your Groups “Who can message and @mention this group?” to be more liberal you should probably reconsider if the group has a lot of members so that the group can’t be bulk messaged.
Maybe the existing “similar to” that exists for posts could be tied into? But instead of an education modal put in a hold queue and notify moderators.