What are the implications of letting oauth2 users change their email addresses?

Oh, that would be bad! Is there, perhaps, a way to allow the addition of a secondary email address while making the primary one fixed?