You have a few options here
-
Hide that field via CSS overrides on your site so users don’t have any opportunity to enter a password
-
Educate your users that they don’t need to enter a password, either just in time (by using CSS rules to add text on that invite accept page), or by modifying the invite email text in Admin, Customize, Text.