My CSS is pretty basic, but I think you can target just the box on the sign-up page with this:
.d-modal.create-account .user-field-YOUR-FIELD-NAME {
display: none;
}
.invites-show .user-field-YOUR-FIELD-NAME {
display: none;
}
And that should let you use the field as normal elsewhere.
Edit: I put a bit in for the invite page too, if you also use invites.