I’m afraid this isn’t going to work at all!
Primarily because invites don’t work without local logins enabled. So, You’re probably looking at a very custom plugin that will handle all of this or an external authentication system to suffice your requirements.