So if I am understanding correctly the anonymous users are replying/creating topics by email?
In Category Settings Tab there is setting for email posts. I can’t see the options as I do not have reply by email setup. Here is a pic
I am guessing you might be able to set it here whether or not a reply/post can be done by email maybe.