This is only true for admins, not moderators, so I am editing your title which is incorrect. Moderators also only have access to PMs when they are flagged.
If this is a concern, demote yourself to moderator (by logging in under a different account to taste), or enable logging of PM visits by admins in your site settings.