An oldie but a goodie!
I just came across this issue today for myself! I am creating groups on my site for my clients to give them access to their own category, to which I want them to be allowed to invite as many people as they like to join the site and their private group+category.
Even with all the improvements to the invite system recently, this is still not possible as far as I can see. Non-moderators/admins can be given invite privileges, but then have to wait for the invitee to join before adding them to the group. Unless I am missing something, hopefully? 
The workaround is for me to create the invite as moderator or admin, or to promote them temporarily, masquerade as them and create the invite on their behalf.