Are you using a plugin for avatars, or are you using the default WordPress Gravatar avatars? When the default WordPress Gravatars are being used, the expected behaviour is that an avatar URL with the parameter d=404 will be passed from WordPress to Discourse during SSO login. The d=404 parameter will cause Discourse to display a letter avatar for users who have not set a Gravatar.
This is working correctly when I test it. If it’s not working correctly for you, any details about your avatar setup would be helpful.