Oh, yeah I have seen this as well @pmusaraj this is tricky.
The easy fix is simply to archive the messages in the group inbox /my/messages/group/moderators by selecting them all and hitting archive. There is a slight delay here but it will fix it.
However it gets extra tricky if you have PMs disabled on your site. If you do, then the only workaround is to temporarily enable PMs so the archive button shows up, so you can move it.
I think a concrete bug here is that if for any reason PMs are disabled we should still show the archive button on group mail boxes.