In your case, and assuming your don’t have other manual groups in the Discourse instance, you should enable the sso overrides groups site setting, and pass the 3 groups the user is currently a member of in the groups parameter of the SSO payload.
On login we will ensure that the user is a member of those 3 groups and not a member of every other manual group.