Try disabling the enable local logins settings. This should remove the ability to login leaving your users only the option of logging in with OAuth2.
I would recommend doing this this in a separate browser session though. That way you can revert any changes you make if you have configuration problems without losing the ability to login.