Is there a way to allow sign-up only for Twitter integration?

Yes, That’s doable. Just make site invite only and configure twitter logins.