When inviting a user or a group to a message, that invitation counts as one reply in the group mailbox. Since an invitation is not a reply, it should’t count. For instance, whispers or assigning a message to someone doesn’t count as replies (correctly).
This problem is very relevant in the context of using Discourse as a ticketing system. Messages received with zero replies have top priority from the group. The fact that a message appears as replied (because of an invitation, not a real reply) might cause that it is not addressed on time, because the attention will keep going to other messages without replies.