Disabling that setting isn’t a good idea.
Email addresses are always validated. Each reply key is bound to a topic and user. If they do not match, the email gets rejected. There’s only one exception. If the system knows that an email was forwarded to another user, because that other user was in the To or CC header of a previous reply, than it will allow that user to reply with the forwarded reply key.
In your case, the user needs to send with the correct email address. There’s no way around it. Most email clients should be smart enough to select the right sender address anyway.