# Avatar service somewhat broken
**URL:** https://meta.discourse.org/t/avatar-service-somewhat-broken/215061
**Category:** Support
**Created:** [January 17, 2022, 8:41am UTC](https://meta.discourse.org/t/avatar-service-somewhat-broken/215061 "2022-01-17T08:41:10Z")
**Posts on this page:** 11
**Page:** 1
### Author:  [@calmh](https://meta.discourse.org/u/calmh)
#### Post date: [January 17, 2022, 8:41am UTC](https://meta.discourse.org/t/avatar-service-somewhat-broken/215061/1 "2022-01-17T08:41:11Z")
It seems like as of recently, users who sign up (edit: or maybe just have the “system avatar” but not cached in my browser) don’t get the typical colored-character system avatar, instead they just get a white / nothing picture. In some cases it seems like it loads, refreshing asynchronously. I see this on my instance, but also here:
Edit:
Looks like the avatar thing is sometimes returning a 502 error.
---
### Author:  [@gerhard](https://meta.discourse.org/u/gerhard)
#### Post date: [January 17, 2022, 9:20am UTC](https://meta.discourse.org/t/avatar-service-somewhat-broken/215061/2 "2022-01-17T09:20:43Z")
A fire/explosion in the vicinity of the datacenter where the avatar service is hosted has caused the outage. We are working on getting everything back online.
---
### Author:  [@kinetiksoft](https://meta.discourse.org/u/kinetiksoft)
#### Post date: [January 17, 2022, 11:44am UTC](https://meta.discourse.org/t/avatar-service-somewhat-broken/215061/3 "2022-01-17T11:44:48Z")
Thanks for the update.
Is there any status page of the datacenter to track the incident ourselves?
And is it just us having admin reports not working?
> **Browser Network**
>
> 
> **Admin reports loading...**
>
> 
> **Admin reports failed to load**
>
> 
---
### Author:  [@gerhard](https://meta.discourse.org/u/gerhard)
#### Post date: [January 17, 2022, 11:48am UTC](https://meta.discourse.org/t/avatar-service-somewhat-broken/215061/4 "2022-01-17T11:48:55Z")
I’m not seeing any problems with admin reports on our sites.
You could track the incident on our status page at [https://status.discourse.org/](https://status.discourse.org/)
---
### Author:  [@simonk](https://meta.discourse.org/u/simonk)
#### Post date: [January 17, 2022, 12:11pm UTC](https://meta.discourse.org/t/avatar-service-somewhat-broken/215061/5 "2022-01-17T12:11:44Z")
I have seen the same problem (admin reports not loading) this morning. The logs show lots of timeout errors in the avatar proxy service and the version check job. I wonder if that is causing other background tasks to be delayed, including the generation of reports.
---
### Author:  [@RGJ](https://meta.discourse.org/u/RGJ)
#### Post date: [January 17, 2022, 12:14pm UTC](https://meta.discourse.org/t/avatar-service-somewhat-broken/215061/6 "2022-01-17T12:14:53Z")
The fact that avatars are being proxied through nginx, and those are now taking 15 seconds to timeout instead of milliseconds to load can under some circumstances seriously saturate your nginx capacity, resulting in errors in other, unrelated, requests as well.
It does seem to help to temporarily disable Admin - Settings - Files - `external_system_avatars_enabled`. (thx @gerhard)
---
### Author:  [@kinetiksoft](https://meta.discourse.org/u/kinetiksoft)
#### Post date: [January 17, 2022, 12:17pm UTC](https://meta.discourse.org/t/avatar-service-somewhat-broken/215061/7 "2022-01-17T12:17:57Z")
Thank you guys. I’m forwarding this to our sysadmin team for them to check what may be causing reporting errors and if temp fix you provided should be applied in our case
---
### Author:  [@kinetiksoft](https://meta.discourse.org/u/kinetiksoft)
#### Post date: [January 17, 2022, 12:24pm UTC](https://meta.discourse.org/t/avatar-service-somewhat-broken/215061/8 "2022-01-17T12:24:51Z")
It took me some time to realize this is a forum setting. Can report that disabling external avatar service does **not** help admin dashboard modules to load.
---
### Author:  [@david](https://meta.discourse.org/u/david)
#### Post date: [January 17, 2022, 3:54pm UTC](https://meta.discourse.org/t/avatar-service-somewhat-broken/215061/10 "2022-01-17T15:54:49Z")
The central avatars service should now be back online 🚀
---
### Author:  [@calmh](https://meta.discourse.org/u/calmh)
#### Post date: [January 17, 2022, 4:30pm UTC](https://meta.discourse.org/t/avatar-service-somewhat-broken/215061/11 "2022-01-17T16:30:09Z")
It’s interesting to me that our self hosted instance suffered a lot of random flakiness during this issue, on top of the obvious one of just avatars not loading… Turning off “external avatar service” in the settings also did not help – avatars then did not render at all, and the API calls still blocked for a long time before failing. I thought our instance was more or less independent, or could be, but apparently not so.
---
### Author:  [@david](https://meta.discourse.org/u/david)
#### Post date: [January 17, 2022, 4:36pm UTC](https://meta.discourse.org/t/avatar-service-somewhat-broken/215061/12 "2022-01-17T16:36:08Z")
Flipping the setting will remove the dependence on the central service, but it will require a full rebake of all posts to update all the cached avatar URLs.
You’re absolutely right that an outage in the letter avatar service shouldn’t affect the rest of the site. As @RGJ noted, this seems to be related to NGINX capacity when the proxied requests block for a long time - we’ll certainly look into whether any improvements can be made there.