Moderation by topic?

Then they should be moved to group PMs if that granular level of access is needed.