CDN 503 on user avatars and CSS files

Hey guys

My forum (https://forum.billysbilling.dk) just went down. As far as I can see it’s due to the CDN. I hope someone can help me - maybe @sam or @codinghorror?

It’s up for me. Is it still down for you?

Yes, unfortunately it is.

Can you try clearing your browser’s cache and refreshing?

That does not help.

I still get this site after several minutes of loading time:

@zogstrip it isn’t a caching issue. We’re located in Denmark, and right now I’m getting a 504 Gateway Time-out across different devices on different networks (while trying to entering the page - not on loading resources).

Sometimes the page will load, but at most times with 404 and 50* errors on resource loading.

Our customer-base (entering this forum) is based in Denmark also, but I don’t see it only as a CDN issue anymore.

Confirmed … investigating … will have it sorted shortly

@christianjohnsen / @nicklasmoeller is it still down now?

Seems to work now! Thanks.

Will get back to you, if the problem occurs again :smile:

2 Likes

Looks OK, apologies for the difficulty – we are having some issues with our glusterfs network file system:

We’ve moved more essential resources (stylesheet, etc) to an internal site path so they won’t be stored in the network file system and vulnerable to issues like this. I believe @sam also moved the default letter avatars to an internal site path.

(It’s OK for a picture of a lolcat in a topic, or even some avatars, to be temporarily missing, but it is much more traumatic if the CSS file for the site is missing!)

2 Likes

Ahh okay. I was in the process of making a topic about this. It’s happening here on Meta and the Imgur community too. Images appear broken from Discourse’s fastly CDN.

1 Like

OK, so that’s not just me. Sometimes when my phone is running out of space some images won’t load, and since I was getting the low space warnings I initially chalked it up to that.

Yeah we are still working on fixing the missing images but stylesheets should be very solid now, no longer served from a shared fs.

2 Likes

Note this was sitting in my bookmarks list, this particular failure mode is no longer likely at all:

https://github.com/discourse/discourse/commit/f58d85edea0f85a9da52798aa45bf6a202c10a61

(this is already deployed to all our customers)

Our filesystem migration is over, if any of our customers reading this see anything amiss PM me.

2 Likes