I think making the invite-only process clearer by keeping the padlock locked and simplifying the description could reduce confusion, something like:
