SSO & external avatars

I believe disabling download_remote_images_to_local should solve your issue, although this will also serve images within posts remotely instead of locally.