Missing brotli assets

bug
1

I’ve got a site that I upgraded yesterday (and moved to a new server).

Some assets are failing and have URLs like:

https://xyz-91d3.kxcdn.com/brotli_asset/locales/en-2767e8d4317a5d4ca60e61187a74d85225d46c78928d59aacab8b5df54a4ad8b.js

I did a rebuild yesterday which seemed to have fixed it, but it’s happening still.

It’s particularly pernicious because without these assets the site is inaccessible and my normal test of pulling the site and looking for “Discourse” implies that the site is just fine (this is a separate issue, however).

Oh, and it’s not the CDN’s fault, as the file isn’t on the server when accessed directly.

I just did another rake assets:precompile and that didn’t fix it either.

EDIT: I rebuilt again and everything is fine, but I swear this is exactly what happened about 20 hours ago.

2 Likes
2

So just a while ago I rebuilt and everything was fine and now some resources are again missing.

They seem to be different ones than before.

I’m changing this to a bug.

I’m disabling the babble plugin. Retort and whosonline are still installed, but are working just fine on another site. This doesn’t seem like it’d be affected by the plugin. . .

The rebuild again has everything working, but twice before things were working after a rebuild and then some js resources went missing (or something went looking for the wrong ones).

1 Like
3

thanks for following up on this @pfaffman! :+1:t3:

4

If the resource is on disk and not on CDN that is a CDN problem.

Can you share an example of a brotli resource that is 404 right now?

2 Likes
5

Thanks, Rafael!

I checked this, but tried both using the hostname directly to bypass the CDN and also doing a find /var/www/discourse -name missing-filename (I’m 90% sure I did that inside thecontainer and not just in /var/discourse/shared, or maybe I did find /shared inside the container).

I am continually annoyed by people who don’t share URLs when they are essential for troubleshooting the problem. And now I’m doing it. :frowning_face:

https://thebitcoin.pub/brotli_asset/locales/en-2767e8d4317a5d4ca60e61187a74d85225d46c78928d59aacab8b5df54a4ad8b.js

is one that was missing a couple rebuilds ago (the same one not working for the CDN link above)

Here is a pair that are working now:
https://thebitcoinpub-91d3.kxcdn.com/brotli_asset/locales/en-135ea415cc2ff9976adc16526c564bcfba4023af2c4760b230b3ebd0449ab533.js
https://thebitcoin.pub/brotli_asset/locales/en-135ea415cc2ff9976adc16526c564bcfba4023af2c4760b230b3ebd0449ab533.js

1 Like
6

:wink:

That won’t do it. I mean an asset that is currently valid (from the live version) and is missing in the CDN.

1 Like
7

No. That was my first assumption as well, bu it doesn’t seem to be the CDN’s fault. When the file is missing from the CDN is it also missing from the host. It seems that Discourse is linking to files that don’t exist. I included the CDN/non-CDN pair to convince myself that I know how to generate the local server URL from the CDN url.

Does something clean up those assets that might be deleting them when it shouldn’t?

2 Likes
8

I’ve seen this on two sites that I tried to rebuild today, rebuilding from the web updater fails per the pic … but rebuilding from command line works OK:

image
image.jpg1570×981 374 KB

8 Likes
9

I didn’t know before, but I’m pretty sure that both of the problems I’ve seen also started with an upgrade from the web interface.

2 Likes
10

I can confirm that thebitcoin.pub gets links to missing assets when an upgrade is done from the web interface and rebuild from the command line fixes it.

What I can’t tell is if this is limited to this site and @codinghorror’s two or if it’s all of them. It seems like it it were the latter, there would be a bigger outcry, especially with two beta releases in less than a week.

Jeff you don’t have any plugins that could be to blame, do you? This site has who’s online and retort.

1 Like
11

I upgraded 2 sites yesterday with no issues, was there anything in the upgrade log when this happened?

3 Likes
12

Not that I noticed. It seems to happen every time, though, so I’ll try to have a look and see if I get any clues.

2 Likes
13

Also please take note of the pre-upgrade version of the container and discourse.

1 Like
14

This happened to me again last night with both commonmark and my own site. Web rebuild resulted in white page with weird 404s from CDN. Rebuilding from command line fixed it.

2 Likes
15

The container is, uh, new. I’m somehow not sure where I find the version.

This has happened 4 times now starting last Thursday.

1 Like
16

This is fixed per:

https://github.com/discourse/docker_manager/commit/ada600b8522c64a36873512833d3c3d61162b602

https://github.com/discourse/discourse_docker/commit/77edaf675a47729bb693d09b94713a2a98b5d686

https://github.com/discourse/discourse/commit/be59c1559d77cc8028265b8d43477b79b6efccef

:arrow_double_up: (backported to stable and beta)

https://github.com/discourse/discourse/commit/baa7a9836cf547b1ae9eefa51096718706c00602

The root of the problem is that we carried around a strangely named COMPRESS_BROTLI env var to denote we should use brotli compression. This var got eaten by the web upgrader.

I think (not sure) when @david refactored some of the way we do brotli assets our path that used to fallback to non brotli assets stopped doing so.

Anyway… having these two things are a problem:

  1. An oddly named global setting
  2. Brotli even be optional

So now… brotli is unconditionally enabled everywhere including stable and beta.

Keep in mind for about 2-3 years brotli has been “optional default on” and nobody ever really turned it off.

14 Likes