This isn’t true, at least not in my experience- whispered posts aren’t visible, even as a “hidden reply”, to users without access. I just tested it to make sure; a topic with one whispered reply shows no replies (it doesn’t even show the infobox that gets added with replies) to a non-moderator
You can now use whispers allowed groups to make them available to more than just staff users, but @not-ethan has found a description that hasn’t been updated to match yet.