Sure thing, and thank you again.
@joffreyjaffeux Should we update tonight to get that debugging patch or wait for your go-ahead?
I will let you know as you as I have something, might not be today, hopefully yes. Be sure I will update this when ready.
Not a problem, thank you.
Could you try to update please?
I pushed https://github.com/discourse/discourse/commit/2b2a506a7bccd6eb03159dbe13e733867930c5e5 which I expect to either fix your issue, or give us a hint. If it doesn’t it will at least narrow the possibilities.
Just upgraded to 849f0d0 and unfortunately, got the exact same error.
can you look in /logs
if you see something please ?
Also can you confirm https://forum.quartertothree.com/admin/dashboard/general.json is failing to load for you ?
From unicorn.stderr.log:
E, [2018-08-01T11:09:51.384566 #10594] ERROR -- : worker=4 PID:11082 timeout (31s > 30s), killing I, [2018-08-01T11:09:57.218680 #2724] INFO -- : worker=4 ready D, [2018-08-01T11:10:21.455417 #10594] DEBUG -- : waiting 16.0s after suspend/hibernation E, [2018-08-01T11:11:19.500626 #10594] ERROR -- : worker=2 PID:10856 timeout (31s > 30s), killing I, [2018-08-01T11:11:25.014635 #2995] INFO -- : worker=2 ready D, [2018-08-01T11:11:49.583321 #10594] DEBUG -- : waiting 16.0s after suspend/hibernation E, [2018-08-01T11:12:17.611403 #10594] ERROR -- : worker=5 PID:12472 timeout (31s > 30s), killing I, [2018-08-01T11:12:23.631837 #3211] INFO -- : worker=5 ready
Don’t see anything notable in production.log, unicorn.stdout.log, or anything in the var-logs dir.
Yes, that URL times out and eventually gives a 502.
I think the DiskSpace caching job is taking too much time on your machine. This is the only thing which is not covered by recent changes I think. I’m still unsure about this though, as a background job is not supposed to make this fail.
Can you try these commands:
df -Pk /path/to/public/backups/default | awk 'NR==2 {print $4;}'
du -s /path/to/public/backups/default
If you struggle to find the correct path, start a rails console on your server and type: DiskSpace.uploads_path
Does it return? Does it return fast?
Both of these commands response essentially instantaneously. We’re on SSD storage.
root@forum:/var/discourse/shared/standalone/backups/default# time df -Pk /var/discourse/shared/standalone/backups/default | awk 'NR==2 {print $4;}'
174096600
real 0m0.004s
user 0m0.001s
sys 0m0.006s
root@forum:/var/discourse/shared/standalone/backups/default# time du -s /var/discourse/shared/standalone/backups/default
33923456 /var/discourse/shared/standalone/backups/default
real 0m0.002s
user 0m0.001s
sys 0m0.001s
Or did you mean inside the container? The du is several orders of magnitude slower there-- I assume because it’s iterating through thousands of cat pictures uploaded to the forum rather than just a couple large backup files.
root@forum-app:/var/www/discourse# time df -Pk /var/www/discourse/public/uploads/default | awk 'NR==2 {print $4;}'
174113284
real 0m0.009s
user 0m0.007s
sys 0m0.000s
root@forum-app:/var/www/discourse# time du -s /var/www/discourse/public/uploads/default
12446396 /var/www/discourse/public/uploads/default
real 0m0.378s
user 0m0.032s
sys 0m0.337s
@joffreyjaffeux helped us fix this issue, which now is resolved. Thanks!