Maybe turn off local logins? I’m not sure about the invite flow, can’t the click the Facebook login, or you’re saying that on account creation it asks for password?
You could customize the language to say something that makes it clearer that your can skip it.