You can switch off these:
Or just allow certain people to message you.
It’s under Preferences → Users at bottom.
(Note, it might be nice if the second option allowed you to select groups too?)