There are two user preferences that control what emails get sent to a user. The settings are found in the Emails section of the user’s preferences page:
The first setting in the screenshot controls when personal messages should be sent as an email. The second setting controls when emails to do with activity in a topic should get sent to a user. For your case you might want to set the second setting to “never”.
You can set site wide default values for these preferences. They are set by the default email messages level and default email level site settings.
Maybe there is another way to handle this. I’ll see if I can come up with something. It might require you to simplify the HTML that is currently being used in your invite email though.