I’m guessing a bit here, but I think that to keep serving external avatars with SSO and the discourse connect overrides avatar setting, or by adding a URL to the external system avatars url setting, you need to disable the download remote images to local site setting. So what you are wanting to do can be accomplished now, but only by preventing the downloading of all remote images.
But, I think Discourse has the functionality you are looking for with Gravatars. You can disable the automatically download gravatars site setting while keeping the download remote images to local site setting enabled.
Edit: If I am correct about how things work now, the feature you want is to have a new site setting: automatically download external system avatars.
When that setting was disabled, avatars could be served from the external system avatars url without them being downloaded by Discourse.