Flipping the setting will remove the dependence on the central service, but it will require a full rebake of all posts to update all the cached avatar URLs.
You’re absolutely right that an outage in the letter avatar service shouldn’t affect the rest of the site. As @RGJ noted, this seems to be related to NGINX capacity when the proxied requests block for a long time - we’ll certainly look into whether any improvements can be made there.