If the rules prohibit nudes then you have a reason to police this. Users can’t really complain about that.
The messages are not ‘private’ they are personal messages (always have). My suggestion is to remind users of the community rules and emphasise that they apply in messages and that you will be enforcing it.
You can enable the log personal messages views site setting to keep tabs on what messages and users are constantly doing this so you can keep an eye on them.
Are they just images or is there also text in the message? If there is then you can pick out the most used words and add them to watched words or consider using this plugin
Other than that I cannot think of another way to effectively moderate images in a PM