Discourse Assets Missing

Hi,
I have installed Discourse on a Fedora 31 server with the recommended install guide, however used podman instead of docker for running the container. It all went perfectly fine this morning when I initially set it up, however I had troubles getting plain HTTP to my Nginx on the host to work and had to use the SSL certs on both nginx instances. I configured Discourse and all was fine.
I later decided to rebuild Discourse with SSL turned off and build it on a non-root user account. Plain HTTP to host worked fine, however I got a white page (title was fine, favicon worked).


I checked the directory in the container:

root@xena-app:/var/www/discourse# ls public/assets -al
total 15640
drwxr-xr-x. 2 discourse discourse      10 Jan 11 13:54 .
drwxr-xr-x. 3 discourse discourse      29 Jan 11 13:57 ..
-rw-r--r--. 1 discourse discourse     595 Jan 11 13:49 _activate-account-7a1f8d46287194007ee607f79af56217e6e6238e4ca159c5da6974005a1254c5.js
-rw-r--r--. 1 discourse discourse     250 Jan 11 13:54 activate-account-7a1f8d46287194007ee607f79af56217e6e6238e4ca159c5da6974005a1254c5.js.br
-rw-r--r--. 1 discourse discourse     420 Jan 11 13:54 activate-account-7a1f8d46287194007ee607f79af56217e6e6238e4ca159c5da6974005a1254c5.js.gz
-rw-r--r--. 1 discourse discourse     695 Jan 11 13:54 activate-account-7a1f8d46287194007ee607f79af56217e6e6238e4ca159c5da6974005a1254c5.js.map
-rw-r--r--. 1 discourse discourse  990519 Jan 11 13:50 _admin-91c2a24a61c0ecd0200db1e808dbadafb9f0f1bb5179acea5b2b7dbd4b4cc5f9.js
-rw-r--r--. 1 discourse discourse   83903 Jan 11 13:54 admin-91c2a24a61c0ecd0200db1e808dbadafb9f0f1bb5179acea5b2b7dbd4b4cc5f9.js.br
-rw-r--r--. 1 discourse discourse  106465 Jan 11 13:54 admin-91c2a24a61c0ecd0200db1e808dbadafb9f0f1bb5179acea5b2b7dbd4b4cc5f9.js.gz
-rw-r--r--. 1 discourse discourse  366446 Jan 11 13:54 admin-91c2a24a61c0ecd0200db1e808dbadafb9f0f1bb5179acea5b2b7dbd4b4cc5f9.js.map
drwxr-xr-x. 2 discourse discourse    4096 Jan 11 13:54 admin-login
...

The files that were requested are indeed missing, even Nginx says so:

2020/01/11 13:59:49 [error] 60#60: *20 open() "/var/www/discourse/public/assets/plugins/poll-57c9059384e3daeaee3d974c985d85a2b8462bb2629eb0b1834451056d1e1459.js" failed (2: No such file or directory), client: 10.0.2.2, server: _, request: "GET /assets/plugins/poll-57c9059384e3daeaee3d974c985d85a2b8462bb2629eb0b1834451056d1e1459.js HTTP/1.0", host: "idlerpg.xyz", referrer: "https://idlerpg.xyz/"

I thought that this would be an issue with my install, because they were here on the old install this morning as root on the host! I have wiped the shared folder and rebuilt as non-root with the same results over and over. Then I tried again, this time as root with identical settings like this morning, and I got the same issues.
A non-root install always shows the “Congratulations, you have installed Discourse!” page, but after I finish the install, I get the blank page because the asset files are missing.

Is there any way to fix the missing assets?

In case anyone wonders if it really worked this morning, here’s a screenshot from my mobile


Discourse won’t allow me to use 2 images in one post yet :slight_smile:

Changed this as a bug. I cannot manage to get a version with all assets when building tests-passed.
I tried entering the container and doing a rake assets:precompile, but the same assets are still gone

Seems after tons of rebuilds, it magically works now. Weird.

Hi Adriany, welcome to the forum

The last I knew, docker is a requirement not an option. Where did you see this newer install guide?

I just replaced docker with podman, works fine. Their command line syntax is compatible

1 Like

Okay, after some testing, I can now tell that it always shows a blank page after every rebuild. If I run RAILS_ENV=production bin/rake assets:precompile, it always works afterwards. No idea why, it should compile them in a rebuild anyways?

Did you see this topic and this post in particular about podman incompatibility with docker?

1 Like

Well it indeed works. Podman should not influence which files will be the result of the initial asset precompile.

In the assets compilation phase of the rebuild do you see any backtraces on the output?

1 Like

We’re here because something doesn’t work. Should and does are different states.

You are confidently asserting that Podman is not implicated. It would good to prove that to clearly eliminate it as a factor. AFAIK, you haven’t completed the official install method with Docker to see if that eliminates the problem.

Yea, I do.

I, [2020-01-12T20:58:31.800117 #1]  INFO -- : > cd /var/www/discourse && su discourse -c 'bundle exec rake assets:precompile'
Purging temp files
Bundling assets
I, [2020-01-12T20:58:38.208107 #758]  INFO -- : Writing /var/www/discourse/public/assets/vendor-3b0a3ac5e0bc628422d6e99d3563fad4c474398644f70683eff0893b7a241513.js
I, [2020-01-12T20:59:02.428630 #758]  INFO -- : Writing /var/www/discourse/public/assets/admin-91c2a24a61c0ecd0200db1e808dbadafb9f0f1bb5179acea5b2b7dbd4b4cc5f9.js
I, [2020-01-12T20:59:02.447192 #758]  INFO -- : Writing /var/www/discourse/public/assets/preload-store-d32dcf974dddcac742f8a7a6aa7fcd686185920b201029d0ecb2b85527ef9034.js
I, [2020-01-12T20:59:02.707559 #758]  INFO -- : Writing /var/www/discourse/public/assets/browser-update-1741a2ed67a367faeb0a582af064457e8b1b1354e52e6efcf8bf26301166dec8.js
I, [2020-01-12T20:59:02.753632 #758]  INFO -- : Writing /var/www/discourse/public/assets/break_string-cc617154cd957804f2f6a1f3bc68258c9cdca3d4b9a322bf777d145fed04790e.js
I, [2020-01-12T20:59:04.073148 #758]  INFO -- : Writing /var/www/discourse/public/assets/ember_jquery-c2cca4a19fd4c70eabc95b010a6d7643955ec2a2251aeb1d70f625773eebea21.js
I, [2020-01-12T20:59:06.303026 #758]  INFO -- : Writing /var/www/discourse/public/assets/pretty-text-bundle-fa6ea00e2000bf2f9ed3f069be9253477fed5e7290b5d109a3e2d9498ac70e81.js
... [snip]
108798.339928812 Compressing: vendor-3b0a3ac5e0bc628422d6e99d3563fad4c474398644f70683eff0893b7a241513.js
uglifyjs '/var/www/discourse/public/assets/_vendor-3b0a3ac5e0bc628422d6e99d3563fad4c474398644f70683eff0893b7a241513.js' -m -c -o '/var/www/discourse/public/assets/vendor-3b0a3ac5e0bc628422d6e99d3563fad4c474398644f70683eff0893b7a241513.js' --source-map "root='/assets',url='/assets/vendor-3b0a3ac5e0bc628422d6e99d3563fad4c474398644f70683eff0893b7a241513.js.map'" --output '/var/www/discourse/public/assets/vendor-3b0a3ac5e0bc628422d6e99d3563fad4c474398644f70683eff0893b7a241513.js'
gzip -f -c -9 /var/www/discourse/public/assets/vendor-3b0a3ac5e0bc628422d6e99d3563fad4c474398644f70683eff0893b7a241513.js > /var/www/discourse/public/assets/vendor-3b0a3ac5e0bc628422d6e99d3563fad4c474398644f70683eff0893b7a241513.js.gz

brotli -f --quality=11 /var/www/discourse/public/assets/vendor-3b0a3ac5e0bc628422d6e99d3563fad4c474398644f70683eff0893b7a241513.js --output=/var/www/discourse/public/assets/vendor-3b0a3ac5e0bc628422d6e99d3563fad4c474398644f70683eff0893b7a241513.js.br


Done compressing vendor-3b0a3ac5e0bc628422d6e99d3563fad4c474398644f70683eff0893b7a241513.js : 3.33 secs

108801.669283898 Compressing: admin-91c2a24a61c0ecd0200db1e808dbadafb9f0f1bb5179acea5b2b7dbd4b4cc5f9.js
uglifyjs '/var/www/discourse/public/assets/_admin-91c2a24a61c0ecd0200db1e808dbadafb9f0f1bb5179acea5b2b7dbd4b4cc5f9.js' -m -c -o '/var/www/discourse/public/assets/admin-91c2a24a61c0ecd0200db1e808dbadafb9f0f1bb5179acea5b2b7dbd4b4cc5f9.js' --source-map "root='/assets',url='/assets/admin-91c2a24a61c0ecd0200db1e808dbadafb9f0f1bb5179acea5b2b7dbd4b4cc5f9.js.map'" --output '/var/www/discourse/public/assets/admin-91c2a24a61c0ecd0200db1e808dbadafb9f0f1bb5179acea5b2b7dbd4b4cc5f9.js'
gzip -f -c -9 /var/www/discourse/public/assets/admin-91c2a24a61c0ecd0200db1e808dbadafb9f0f1bb5179acea5b2b7dbd4b4cc5f9.js > /var/www/discourse/public/assets/admin-91c2a24a61c0ecd0200db1e808dbadafb9f0f1bb5179acea5b2b7dbd4b4cc5f9.js.gz
[...]

It does compile them, but I always have to do it again after the build to make the missing files appear

Interesting so the files are crafted just fine, but the persistence fails?

I’d look into how the podman volume is working, and what is the driver that is backing it. In the Docker world we have a lot of weird issues with storage drivers over the years, which made we even keep a list of “blessed” drivers in our launcher script.

4 Likes

To be clear, my experiments with podman were really testing launcher not discourse. At least when I tested it, it depended on output details of docker that podman doesn’t try to emulate.

@Adriany, I’m guessing you are using the default storage driver, which is the same one that is recommended for discourse, which in docker is "overlay2* — but I don’t know that to be the case.

3 Likes

I am using overlayfs, which is the equivalent to overlay2.
Interestingly, it just built fine with no missing assets after some more builds with missing assets when I tried to add plugins. When I rebuilt without plugins, it went butter smooth. Huh?

Edit: It did work for me and now the assets are cached here. During running, it stopped working for other users.

Edit 2: Another rebuild made it break assets like before and the fix was to rebuild as before.

1 Like

If files are crafted successfully by our assets pipeline and the OS says there is no error, but they disappear after the rebuild (which effectively restarts the container) I’d (re-)place my bet on the podman storage driver.

Unless someone can reproduce this same behavior using Docker (and there would be several topics if this was the case), I’m afraid you need to open a ticket on the Podman project about this.

3 Likes

Hey,
I’ve done more investigation and it seems I was wrong on my bets on where the files are stored or just dumb at finding them.

The files are all a 404, however take this as example:

root@idlerpg:/var/www/discourse# find . -type f -name "*discourse-solved*"
./tmp/stylesheet-cache/discourse-solved_9_cb2984706f4402e95fd787160a5588ed0972ccc3.css
./tmp/stylesheet-cache/discourse-solved_9_cb2984706f4402e95fd787160a5588ed0972ccc3.css.map
./public/assets/plugins/_discourse-solved-5b7312e4207059eb2654a46e804b4c899cfc1e2a362d0d60e580943f9eb9d419.js
./public/assets/plugins/discourse-solved-5b7312e4207059eb2654a46e804b4c899cfc1e2a362d0d60e580943f9eb9d419.js.gz
./public/assets/plugins/discourse-solved-5b7312e4207059eb2654a46e804b4c899cfc1e2a362d0d60e580943f9eb9d419.js.br
./public/assets/plugins/discourse-solved-5b7312e4207059eb2654a46e804b4c899cfc1e2a362d0d60e580943f9eb9d419.js.map
./app/assets/javascripts/plugins/discourse-solved.js.erb
./plugins/discourse-solved/test/javascripts/acceptance/discourse-solved-test.js.es6

They are there, so I guess this is a Discourse issue and not podman, I just did not manage to look for the files properly.

I thought it might have been an issue in Cloudflare, but when I do a quick curl on localhost on the server, I get a 404 as well.

1 Like

Can you gives us a repro of this happening on Docker?

4 Likes