Did those old groups previously have messages in them? I have a group on my personal site that just had one message in it and the inbox appears in the sidebar. I just deleted the message and the group inbox is still showing.
The deleted message no longer shows up in the inboxes, even to me as admin. I was able to browse back to it in my browser history and see it still exists as a deleted message. I tried removing the group and it did not let me. I undeleted it and removed the group. Then deleted the message for good measure.
The inbox is still showing - so I suspect there is still some other deleted message in that group inbox.
So yes, there is some bug here, methinks - if the only messages in a group inbox are deleted then the group inbox should not appear in the sidebar.
The only fix at the moment methinks is to delete the group and recreate it.
It would be interesting to use data explorer to get the post id of any deleted messages in a group inbox to confirm my theory, and get a look at those posts.