502 Bad Gateway for only two Topics

niccokunzmann · October 26, 2022, 4:24pm

I have a 502 Bad gateway for a topic (maybe more than one) on the discourse forum I host:

I just document my way here and see if someone can help.

I did

./launcher rebuild app - Discourse stuck on "Currently Upgrading" - #2 by sam
upgrading all and canceling the stuck upgrade
- Stuck on “currently upgrading...”
- "Currently upgrading" even after a rebuild

I wonder what to do. I will search a bit more.

restarting the docker container - "Currently upgrading" even after a rebuild - #3 by codinghorror

HEAD requests

Here is a list of pages:

$ for i in `seq 100`; do ( status="`curl -L --head -o /dev/null -s -w '%{http_code}' https://ammanvalley.foss.wales/t/$i`"; echo "https://ammanvalley.foss.wales/t/$i - $status" ) done
https://ammanvalley.foss.wales/t/1 - 200
https://ammanvalley.foss.wales/t/2 - 404
https://ammanvalley.foss.wales/t/3 - 200
https://ammanvalley.foss.wales/t/4 - 404
https://ammanvalley.foss.wales/t/5 - 404
https://ammanvalley.foss.wales/t/6 - 404
https://ammanvalley.foss.wales/t/7 - 200 -- no 502
https://ammanvalley.foss.wales/t/8 - 404
https://ammanvalley.foss.wales/t/9 - 404
https://ammanvalley.foss.wales/t/10 - 200
https://ammanvalley.foss.wales/t/11 - 200
https://ammanvalley.foss.wales/t/12 - 200
https://ammanvalley.foss.wales/t/13 - 200
https://ammanvalley.foss.wales/t/14 - 200
https://ammanvalley.foss.wales/t/15 - 200
https://ammanvalley.foss.wales/t/16 - 404
https://ammanvalley.foss.wales/t/17 - 404
https://ammanvalley.foss.wales/t/18 - 200
https://ammanvalley.foss.wales/t/19 - 200
https://ammanvalley.foss.wales/t/20 - 404
https://ammanvalley.foss.wales/t/21 - 200
https://ammanvalley.foss.wales/t/22 - 200
https://ammanvalley.foss.wales/t/23 - 200
https://ammanvalley.foss.wales/t/24 - 200
https://ammanvalley.foss.wales/t/25 - 200
https://ammanvalley.foss.wales/t/26 - 404
https://ammanvalley.foss.wales/t/27 - 404
https://ammanvalley.foss.wales/t/28 - 200
https://ammanvalley.foss.wales/t/29 - 404
https://ammanvalley.foss.wales/t/30 - 200
https://ammanvalley.foss.wales/t/31 - 200
https://ammanvalley.foss.wales/t/32 - 200
https://ammanvalley.foss.wales/t/33 - 200
https://ammanvalley.foss.wales/t/34 - 200
https://ammanvalley.foss.wales/t/35 - 200
https://ammanvalley.foss.wales/t/36 - 404
https://ammanvalley.foss.wales/t/37 - 200
https://ammanvalley.foss.wales/t/38 - 200
https://ammanvalley.foss.wales/t/39 - 200
https://ammanvalley.foss.wales/t/40 - 404
https://ammanvalley.foss.wales/t/41 - 200
https://ammanvalley.foss.wales/t/42 - 200
https://ammanvalley.foss.wales/t/43 - 200
https://ammanvalley.foss.wales/t/44 - 200
https://ammanvalley.foss.wales/t/45 - 200
https://ammanvalley.foss.wales/t/46 - 200
https://ammanvalley.foss.wales/t/47 - 404
https://ammanvalley.foss.wales/t/48 - 200
https://ammanvalley.foss.wales/t/49 - 200
https://ammanvalley.foss.wales/t/50 - 200
https://ammanvalley.foss.wales/t/51 - 200
https://ammanvalley.foss.wales/t/52 - 200
https://ammanvalley.foss.wales/t/53 - 200
https://ammanvalley.foss.wales/t/54 - 200
https://ammanvalley.foss.wales/t/55 - 200
https://ammanvalley.foss.wales/t/56 - 200
https://ammanvalley.foss.wales/t/57 - 200
https://ammanvalley.foss.wales/t/58 - 200 -- no 502
https://ammanvalley.foss.wales/t/59 - 200
https://ammanvalley.foss.wales/t/60 - 200
https://ammanvalley.foss.wales/t/61 - 200
https://ammanvalley.foss.wales/t/62 - 404
https://ammanvalley.foss.wales/t/63 - 200
https://ammanvalley.foss.wales/t/64 - 404
https://ammanvalley.foss.wales/t/65 - 404
https://ammanvalley.foss.wales/t/66 - 200
https://ammanvalley.foss.wales/t/67 - 200
https://ammanvalley.foss.wales/t/68 - 404
https://ammanvalley.foss.wales/t/69 - 200
https://ammanvalley.foss.wales/t/70 - 404
https://ammanvalley.foss.wales/t/71 - 404
https://ammanvalley.foss.wales/t/72 - 404
https://ammanvalley.foss.wales/t/73 - 200
https://ammanvalley.foss.wales/t/74 - 404
https://ammanvalley.foss.wales/t/75 - 404
https://ammanvalley.foss.wales/t/76 - 404
https://ammanvalley.foss.wales/t/77 - 404
https://ammanvalley.foss.wales/t/78 - 404
https://ammanvalley.foss.wales/t/79 - 404
https://ammanvalley.foss.wales/t/80 - 404
https://ammanvalley.foss.wales/t/81 - 404
https://ammanvalley.foss.wales/t/82 - 404
https://ammanvalley.foss.wales/t/83 - 404
https://ammanvalley.foss.wales/t/84 - 404
https://ammanvalley.foss.wales/t/85 - 404
https://ammanvalley.foss.wales/t/86 - 404
https://ammanvalley.foss.wales/t/87 - 404
https://ammanvalley.foss.wales/t/88 - 404
https://ammanvalley.foss.wales/t/89 - 404
https://ammanvalley.foss.wales/t/90 - 404
https://ammanvalley.foss.wales/t/91 - 404
https://ammanvalley.foss.wales/t/92 - 404
https://ammanvalley.foss.wales/t/93 - 404
https://ammanvalley.foss.wales/t/94 - 404
https://ammanvalley.foss.wales/t/95 - 404
https://ammanvalley.foss.wales/t/96 - 404
https://ammanvalley.foss.wales/t/97 - 404
https://ammanvalley.foss.wales/t/98 - 404
https://ammanvalley.foss.wales/t/99 - 404
https://ammanvalley.foss.wales/t/100 - 404

GET requests

$ for i in `seq 100`; do ( status="`curl -L -o /dev/null -s -w '%{http_code}' https://ammanvalley.foss.wales/t/$i`"; echo "https://ammanvalley.foss.wales/t/$i - $status" ) done
https://ammanvalley.foss.wales/t/1 - 200
https://ammanvalley.foss.wales/t/2 - 404
https://ammanvalley.foss.wales/t/3 - 200
https://ammanvalley.foss.wales/t/4 - 404
https://ammanvalley.foss.wales/t/5 - 404
https://ammanvalley.foss.wales/t/6 - 404
https://ammanvalley.foss.wales/t/7 - 502  -- 502
https://ammanvalley.foss.wales/t/8 - 404
https://ammanvalley.foss.wales/t/9 - 404
https://ammanvalley.foss.wales/t/10 - 200
https://ammanvalley.foss.wales/t/11 - 200
https://ammanvalley.foss.wales/t/12 - 200
https://ammanvalley.foss.wales/t/13 - 200
https://ammanvalley.foss.wales/t/14 - 200
https://ammanvalley.foss.wales/t/15 - 200
https://ammanvalley.foss.wales/t/16 - 404
https://ammanvalley.foss.wales/t/17 - 404
https://ammanvalley.foss.wales/t/18 - 200
https://ammanvalley.foss.wales/t/19 - 200
https://ammanvalley.foss.wales/t/20 - 404
https://ammanvalley.foss.wales/t/21 - 200
https://ammanvalley.foss.wales/t/22 - 200
https://ammanvalley.foss.wales/t/23 - 200
https://ammanvalley.foss.wales/t/24 - 200
https://ammanvalley.foss.wales/t/25 - 200
https://ammanvalley.foss.wales/t/26 - 404
https://ammanvalley.foss.wales/t/27 - 404
https://ammanvalley.foss.wales/t/28 - 200
https://ammanvalley.foss.wales/t/29 - 404
https://ammanvalley.foss.wales/t/30 - 200
https://ammanvalley.foss.wales/t/31 - 200
https://ammanvalley.foss.wales/t/32 - 200
https://ammanvalley.foss.wales/t/33 - 200
https://ammanvalley.foss.wales/t/34 - 200
https://ammanvalley.foss.wales/t/35 - 200
https://ammanvalley.foss.wales/t/36 - 404
https://ammanvalley.foss.wales/t/37 - 200
https://ammanvalley.foss.wales/t/38 - 200
https://ammanvalley.foss.wales/t/39 - 200
https://ammanvalley.foss.wales/t/40 - 404
https://ammanvalley.foss.wales/t/41 - 200
https://ammanvalley.foss.wales/t/42 - 200
https://ammanvalley.foss.wales/t/43 - 200
https://ammanvalley.foss.wales/t/44 - 200
https://ammanvalley.foss.wales/t/45 - 200
https://ammanvalley.foss.wales/t/46 - 200
https://ammanvalley.foss.wales/t/47 - 404
https://ammanvalley.foss.wales/t/48 - 200
https://ammanvalley.foss.wales/t/49 - 200
https://ammanvalley.foss.wales/t/50 - 200
https://ammanvalley.foss.wales/t/51 - 200
https://ammanvalley.foss.wales/t/52 - 200
https://ammanvalley.foss.wales/t/53 - 200
https://ammanvalley.foss.wales/t/54 - 200
https://ammanvalley.foss.wales/t/55 - 200
https://ammanvalley.foss.wales/t/56 - 200
https://ammanvalley.foss.wales/t/57 - 200
https://ammanvalley.foss.wales/t/58 - 502 -- 502
https://ammanvalley.foss.wales/t/59 - 200
https://ammanvalley.foss.wales/t/60 - 200
https://ammanvalley.foss.wales/t/61 - 200
https://ammanvalley.foss.wales/t/62 - 404
https://ammanvalley.foss.wales/t/63 - 200
https://ammanvalley.foss.wales/t/64 - 404
https://ammanvalley.foss.wales/t/65 - 404
https://ammanvalley.foss.wales/t/66 - 200
https://ammanvalley.foss.wales/t/67 - 200
https://ammanvalley.foss.wales/t/68 - 404
https://ammanvalley.foss.wales/t/69 - 200
https://ammanvalley.foss.wales/t/70 - 404
https://ammanvalley.foss.wales/t/71 - 404
https://ammanvalley.foss.wales/t/72 - 404
https://ammanvalley.foss.wales/t/73 - 200
https://ammanvalley.foss.wales/t/74 - 404
https://ammanvalley.foss.wales/t/75 - 404
https://ammanvalley.foss.wales/t/76 - 404
https://ammanvalley.foss.wales/t/77 - 404
https://ammanvalley.foss.wales/t/78 - 404
https://ammanvalley.foss.wales/t/79 - 404
https://ammanvalley.foss.wales/t/80 - 404
https://ammanvalley.foss.wales/t/81 - 404
https://ammanvalley.foss.wales/t/82 - 404
https://ammanvalley.foss.wales/t/83 - 404
https://ammanvalley.foss.wales/t/84 - 404
https://ammanvalley.foss.wales/t/85 - 404
https://ammanvalley.foss.wales/t/86 - 404
https://ammanvalley.foss.wales/t/87 - 404
https://ammanvalley.foss.wales/t/88 - 404
https://ammanvalley.foss.wales/t/89 - 404
https://ammanvalley.foss.wales/t/90 - 404
https://ammanvalley.foss.wales/t/91 - 404
https://ammanvalley.foss.wales/t/92 - 404
https://ammanvalley.foss.wales/t/93 - 404
https://ammanvalley.foss.wales/t/94 - 404
https://ammanvalley.foss.wales/t/95 - 404
https://ammanvalley.foss.wales/t/96 - 404
https://ammanvalley.foss.wales/t/97 - 404
https://ammanvalley.foss.wales/t/98 - 404
https://ammanvalley.foss.wales/t/99 - 404
https://ammanvalley.foss.wales/t/100 - 404

Logs

A list of errors from ./launcher logs app:

...
[Wed 26 Oct 2022 04:28:27 PM UTC] Run reload cmd: sv reload nginx
fail: nginx: runsv not running
[Tue 25 Oct 2022 01:07:47 PM UTC] Run reload cmd: sv reload nginx
warning: nginx: unable to open supervise/ok: file does not exist
[Tue 25 Oct 2022 01:07:47 PM UTC] ESC[1;31mReload error for :ESC[0m
Started runsvdir, PID is 552
[Tue 25 Oct 2022 01:07:45 PM UTC] Run reload cmd: sv reload nginx
warning: nginx: unable to open supervise/ok: file does not exist
[Tue 25 Oct 2022 01:07:45 PM UTC] ESC[1;31mReload error for :ESC[0m
[Tue 25 Oct 2022 01:07:46 PM UTC] Domains not changed.

Further investigation

I am unsure about how to proceed with this. It affects only two topics.
The previews work, the data does not:

How do I restore the posts?

pfaffman · October 26, 2022, 7:50pm

Are those topics deleted? Could they be PMs? Are you requesting them as an admin?

Oh, sorry, it’s a 502; You can have a look at /var/discourse/shared/logs/rails/production.log and see if there are any clues there. See also https://ammanvalley.foss.wales/logs

niccokunzmann · October 26, 2022, 8:59pm

Hm rails is happy and logs very fast.

Started GET "/t/wip-arduino-workshop/58" for 8.8.8.8 at 2022-10-26 20:57:4
8 +0000
Processing by TopicsController#show as HTML
  Parameters: {"slug"=>"wip-arduino-workshop", "topic_id"=>"58"}
start
done

Then, the website still loads for a while until the 502 appears.

However, I get a lot of errors here /var/discourse/shared/standalone/log/rails/production.log:

Failed to report error: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL) 3 Error fetching job: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL) 
Failed to report error: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL) 3 Error fetching job: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL) 
Failed to report error: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL) 3 Error fetching job: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL) 
Failed to report error: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL) 2 Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL) subscribe failed, reconnecting in 1 second. Call stack /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.7.1/lib/redis/client.rb:398:in `rescue in establish_connection'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.7.1/lib/redis/client.rb:379:in `establish_connection'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.7.1/lib/redis/client.rb:115:in `block in connect'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.7.1/lib/redis/client.rb:344:in `with_reconnect'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.7.1/lib/redis/client.rb:114:in `connect'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.7.1/lib/redis/client.rb:417:in `ensure_connected'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.7.1/lib/redis/client.rb:269:in `block in process'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.7.1/lib/redis/client.rb:356:in `logging'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.7.1/lib/redis/client.rb:268:in `process'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.7.1/lib/redis/client.rb:175:in `block in call_loop'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.7.1/lib/redis/client.rb:331:in `with_socket_timeout'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.7.1/lib/redis/client.rb:174:in `call_loop'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.7.1/lib/redis/subscribe.rb:44:in `subscription'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.7.1/lib/redis/subscribe.rb:14:in `subscribe'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.7.1/lib/redis.rb:287:in `_subscription'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.7.1/lib/redis/commands/pubsub.rb:20:in `block in subscribe'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.7.1/lib/redis.rb:264:in `block in synchronize'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.7.1/lib/redis.rb:264:in `synchronize'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.7.1/lib/redis.rb:264:in `synchronize'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.7.1/lib/redis/commands/pubsub.rb:19:in `subscribe'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/message_bus-4.2.0/lib/message_bus/backends/redis.rb:302:in `global_subscribe'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/message_bus-4.2.0/lib/message_bus.rb:763:in `global_subscribe_thread'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/message_bus-4.2.0/lib/message_bus.rb:734:in `block in new_subscriber_thread' 
Failed to report error: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL) 3 Error fetching job: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL) 
Failed to report error: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL) 3 Error fetching job: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL) 
Failed to report error: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL) 3 Job exception: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL)
 
Failed to report error: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL) 3 Job exception: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL)
 
2022-10-26 16:31:02 +0000: Issuing stop to sidekiq
I, [2022-10-26T16:31:04.563256 #2079]  INFO -- : executing ["/var/www/discourse/vendor/bundle/ruby/2.7.0/bin/unicorn", "-E", "production", "-c", "config/unico
rn.conf.rb", {10=>#<Unicorn::TCPSrv:fd 10, AF_INET, 127.0.0.1, 3000>}] (in /var/www/discourse)
I, [2022-10-26T16:31:04.563661 #2079]  INFO -- : forked child re-executing...
I, [2022-10-26T16:31:07.250986 #2079]  INFO -- : inherited addr=127.0.0.1:3000 fd=10
I, [2022-10-26T16:31:07.252783 #2079]  INFO -- : Refreshing Gem list
I, [2022-10-26T16:31:44.617034 #2079]  INFO -- : starting 1 supervised sidekiqs
I, [2022-10-26T16:31:47.752185 #2079]  INFO -- : master process ready
I, [2022-10-26T16:31:55.330063 #589]  INFO -- : reaped #<Process::Status: pid 704 exit 0> worker=1
I, [2022-10-26T16:31:55.333326 #589]  INFO -- : reaped #<Process::Status: pid 714 exit 0> worker=2
I, [2022-10-26T16:31:55.333928 #589]  INFO -- : reaped #<Process::Status: pid 1740 exit 0> worker=0
I, [2022-10-26T16:31:55.335869 #589]  INFO -- : master complete
I, [2022-10-26T16:32:26.504213 #2414]  INFO -- : worker=0 ready
2022-10-26 16:32:27 +0000: Issuing stop to sidekiq
I, [2022-10-26T16:32:28.194731 #2429]  INFO -- : worker=1 ready
I, [2022-10-26T16:32:29.025684 #2445]  INFO -- : worker=2 ready
Process would not terminate cleanly, force quitting. pid: 2402 Demon::Sidekiq
I, [2022-10-26T16:32:38.212437 #2079]  INFO -- : reaped #<Process::Status: pid 2429 exit 0> worker=1
I, [2022-10-26T16:32:38.212904 #2079]  INFO -- : reaped #<Process::Status: pid 2445 exit 0> worker=2
oxipng worker: could not get version of oxipng at /usr/local/bin/oxipng; please provide proper binary or disable this worker (--no-oxipng argument or `:oxipng => false` through options)
oxipng worker: could not get version of oxipng at /usr/local/bin/oxipng; please provide proper binary or disable this worker (--no-oxipng argument or `:oxipng => false` through options)
E, [2022-10-26T17:47:01.151902 #5094] ERROR -- : worker=0 PID:7920 timeout (31s > 30s), killing
I, [2022-10-26T21:01:44.288907 #10359]  INFO -- : worker=1 ready
I, [2022-10-26T21:02:06.344069 #10391]  INFO -- : worker=0 ready
I, [2022-10-26T21:02:10.336134 #10410]  INFO -- : worker=2 ready
D, [2022-10-26T21:02:14.022594 #595] DEBUG -- : waiting 16.0s after suspend/hibernation
E, [2022-10-26T21:02:39.047922 #595] ERROR -- : worker=1 PID:10359 timeout (31s > 30s), killing
E, [2022-10-26T21:02:39.175049 #595] ERROR -- : reaped #<Process::Status: pid 10359 SIGKILL (signal 9)> worker=1
E, [2022-10-26T21:02:44.202628 #595] ERROR -- : worker=0 PID:10391 timeout (31s > 30s), killing
E, [2022-10-26T21:02:44.327961 #595] ERROR -- : reaped #<Process::Status: pid 10391 SIGKILL (signal 9)> worker=0
I, [2022-10-26T21:03:08.803095 #10488]  INFO -- : worker=1 ready
E, [2022-10-26T21:03:11.378475 #595] ERROR -- : worker=2 PID:10410 timeout (31s > 30s), killing
E, [2022-10-26T21:03:11.456868 #595] ERROR -- : worker=2 PID:10410 timeout (31s > 30s), killing
E, [2022-10-26T21:03:11.548047 #595] ERROR -- : reaped #<Process::Status: pid 10410 SIGKILL (signal 9)> worker=2

Falco · October 26, 2022, 9:03pm

Also check /var/log/nginx/ files for clues.

pfaffman · October 26, 2022, 9:04pm

I can see that https://ammanvalley.foss.wales/t/welcome-to-amman-valley-makerspace/7.json returns nothing and then gives the 502.

Do you have any plugins installed?

Have you rebooted recently? How much RAM and swap do you have?

It looks like there is some issue with Redis. I can’t think of a reason that these two topics would cause that.

niccokunzmann · October 27, 2022, 5:49am

No but I have restarted the app, rebuild it and updates it through the web interface.

So the nginx output waits for the timeout in access.log and then produces this:

[27/Oct/2022:05:45:48 +0000] "ammanvalley.foss.wales" 8.8.8.8 "GET /t/wip-arduino-workshop/58 HTTP/2.0" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:105.
0) Gecko/20100101 Firefox/105.0" "-" 502 217 "https://meta.discourse.org/" 30.516 30.516 "-" "-" "-" "-" "-" "-" "-"

In the error.log, I can see this:

2022/10/26 21:03:48 [error] 578#578: *509 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 46.59.192.44, se
rver: _, request: "GET /t/wip-arduino-workshop/58 HTTP/2.0", upstream: "http://127.0.0.1:3000/t/wip-arduino-workshop/58", host: "ammanvalley.foss.wales", refe
rrer: "https://meta.discourse.org/"
2022/10/26 21:05:58 [error] 575#575: *591 upstream prematurely closed connection while reading response header from upstream, client: 189.79.134.158, server:
_, request: "GET /t/wip-arduino-workshop/58 HTTP/2.0", upstream: "http://127.0.0.1:3000/t/wip-arduino-workshop/58", host: "ammanvalley.foss.wales", referrer:
"https://meta.discourse.org/"
2022/10/26 21:52:07 [error] 578#578: *1117 upstream prematurely closed connection while reading response header from upstream, client: 216.68.248.7, server: _
, request: "GET /t/welcome-to-amman-valley-makerspace/7 HTTP/1.1", upstream: "http://127.0.0.1:3000/t/welcome-to-amman-valley-makerspace/7", host: "ammanvalle
y.foss.wales"
2022/10/26 22:33:23 [error] 578#578: *1534 upstream prematurely closed connection while reading response header from upstream, client: 64.222.212.196, server:
 _, request: "GET /t/wip-arduino-workshop/58 HTTP/1.1", upstream: "http://127.0.0.1:3000/t/wip-arduino-workshop/58", host: "ammanvalley.foss.wales"
2022/10/27 05:21:21 [crit] 578#578: *2066 SSL_do_handshake() failed (SSL: error:141CF06C:SSL routines:tls_parse_ctos_key_share:bad key share) while SSL handsh
aking, client: 64.62.197.155, server: 0.0.0.0:443
2022/10/27 05:44:54 [error] 578#578: *2073 upstream prematurely closed connection while reading response header from upstream, client: 92.40.170.157, server:
_, request: "GET /t/wip-arduino-workshop/58 HTTP/2.0", upstream: "http://127.0.0.1:3000/t/wip-arduino-workshop/58", host: "ammanvalley.foss.wales", referrer:
"https://meta.discourse.org/"
2022/10/27 05:45:48 [error] 578#578: *2073 upstream prematurely closed connection while reading response header from upstream, client: 92.40.170.157, server:
_, request: "GET /t/wip-arduino-workshop/58 HTTP/2.0", upstream: "http://127.0.0.1:3000/t/wip-arduino-workshop/58", host: "ammanvalley.foss.wales", referrer:
"https://meta.discourse.org/"
2022/10/27 05:48:48 [error] 575#575: *2161 upstream prematurely closed connection while reading response header from upstream, client: 92.40.170.156, server:
_, request: "GET /t/wip-arduino-workshop/58 HTTP/2.0", upstream: "http://127.0.0.1:3000/t/wip-arduino-workshop/58", host: "ammanvalley.foss.wales", referrer:
"https://meta.discourse.org/"

SWAP: 311M/2.97G RAM: 1.13G/1.93G

Benjamin_D · October 27, 2022, 6:07am

the raw is there though, and seems harmless.
I wonder what the rails console would show…

rails c
Topic.where(id: 58)

niccokunzmann · October 28, 2022, 8:23am

I went into the rails console.

pry(main)> Topic.where(id: 58)
=> [#<Topic:0x0000aaab01b22260
  id: 58,
  title: "[WIP] Arduino Workshop",
  last_posted_at: Sun, 23 Oct 2022 22:13:09.895053000 UTC +00:00,
  created_at: Fri, 23 Sep 2022 10:30:18.827248000 UTC +00:00,
  updated_at: Sun, 23 Oct 2022 22:13:10.291385000 UTC +00:00,
  views: 26,
  posts_count: 6,
  user_id: 1,
  last_post_user_id: 1,
  reply_count: 4,
  featured_user1_id: 2,
  featured_user2_id: nil,
  featured_user3_id: nil,
  deleted_at: nil,
  highest_post_number: 6,
  like_count: 1,
  incoming_link_count: 16,
  category_id: 6,
  visible: true,
  moderator_posts_count: 0,
  closed: false,
  archived: false,
  bumped_at: Sun, 23 Oct 2022 22:13:09.895053000 UTC +00:00,
  has_summary: false,
  archetype: "regular",
  featured_user4_id: nil,
  notify_moderators_count: 0,
  spam_count: 0,
  pinned_at: nil,
  score: 11.233333333333333,
  percent_rank: 1.0,
  subtype: nil,
  slug: "wip-arduino-workshop",
  deleted_by_id: nil,
  participant_count: 2,
  word_count: 184,
  excerpt:
   "I can run a workshop to introduce into programming arduinos. I post this here, so that we can organize \n\nOrganization\n\n\n make a poster @DomAman made one\n\n get material down from the community center\n\n make a list of what&hellip;",
  pinned_globally: false,
  pinned_until: nil,
  fancy_title: "[WIP] Arduino Workshop",
  highest_staff_post_number: 6,
  featured_link: nil,
  reviewable_score: 0.0,
  image_upload_id: nil,
  slow_mode_seconds: 0,
  bannered_until: nil,
  external_id: nil>]

RGJ · October 30, 2022, 5:17pm

Seems like https://ammanvalley.foss.wales/raw/58 works fine.
When did this start?

niccokunzmann · November 5, 2022, 12:03am

Thanks so much for the raw link! This helps backing the topics up.

A new link is 502 now: Hovercraft Project - Projects - Amman Valley MakerSpace

Hm. I did updates, disabled plugins. The new link however was created shortly. No updates, no disables, nothing administrative.

No! Now it works! Hm. I do not get it.
Now, it seems to be resolved: All three posts display in HTML without any error. Should I close this now? If anyone is interested in pursuing this, please comment. I leave closing to the time.

niccokunzmann · November 6, 2022, 5:42pm

No! It is not resolved. The forum has restarted 30 minutes ago and the 502 appears on other posts:

https://ammanvalley.foss.wales/t/hovercraft-project/93
https://ammanvalley.foss.wales/t/welcome-to-amman-valley-makerspace/7
https://ammanvalley.foss.wales/t/wip-arduino-workshop/58

The forum generally loads slowly at times. Is there a way to increase the timeout? The raw links work!

I do not really know what to do with this.

RGJ · November 6, 2022, 6:48pm

So the 502 indeed happens because of the timeout, it happens just over 30 seconds.

What’s going on in Sidekiq? Is there a large queue?

pfaffman · November 6, 2022, 6:53pm

Then that’s what I would address.

I’d double your RAM or get a faster CPU. If you’re on shared hosting (like most of Digital Ocean’s droplets, you could have a “noisy neighbor” that’s slowing things down for you).

Also, if you still haven’t rebooted your OS, it might have upgraded some libraries that are taking up ram.

Stephen · November 6, 2022, 6:54pm

Are you actually running on an arm CPU, or is that tag erroneous?

niccokunzmann · November 6, 2022, 9:31pm

Yes, it is an ARM CPU. I accidentially inquired further and found that when I open a 502 page, I see this process spiking:

unicorn worker[1] -E production -c config/unicorn.conf.rb

99.0 is the cpu percentage:

And this also stays up several seconds after a 502 page is returned to the browser. What is it doing?

How can I check this?

Hm. These are the minimal requirements. Also, the RAM never spikes up or even seems to move.

RGJ · November 6, 2022, 9:50pm

So the unicorn process is what’s responsible for rendering the page and handing it over to nginx. After 30 seconds, nginx gives up, but the unicorn process does not know that, so it continues rendering the page.

The question is: why does unicorn take so long on rendering certain topics?

I think your RAM is fine.

You could try to use the mini-profiler to see what is going on. Here’s how to do that (it’s in the context of an unrelated problem).

You can check on Sidekiq by going to /sidekiq as an admin.

niccokunzmann · November 6, 2022, 9:57pm

There seems to be nothing really happening for sidekiq: The one process spike comes up when the browser sends the GET.

I notice that I hit this with reply topics. This might also just be because I am more likely to visit them.

niccokunzmann · November 9, 2022, 3:57pm

I have trouble opening it. The shortcut does not work. When I clicked on a post that generates a 502, it was open on the page where I clicked it and then it went away. I do not know how to open it at the moment. I can execute some JavaScript though.

niccokunzmann · November 15, 2022, 11:19pm

My question at the moment is: Is there a way to increase the time-out for nginx from say the 30 seconds to about two minutes? This would solve it probably.

Nginx Timeout 20s Causes 502 Bad Gateway - Stack Overflow
templates/web.letsencrypt.ssl.template.yml - accesses the nginx config
Module ngx_http_proxy_module

My conclusion is that I can write a template that adds a .conf file to /etc/nginx/sites-enabled/ that sets the timeouts.

Also, I noticed that I observe this 502 with a higher likelihood when an answer was posted. My guess is that the forum needs to rebuild the page and that takes a while. Is this correct? I wonder: can I trigger a rebuild of a discourse page manually?

niccokunzmann · November 17, 2022, 6:27pm

My try:

/var/discourse# cat templates/web.proxy_read_timeout.template.yml 
#
# Set the timeout for nginx proxy reads on slow systems.
# See https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_read_timeout
# See https://meta.discourse.org/t/502-bad-gateway-for-only-two-topics/242981/20?u=niccokunzmann
#
run:
  - file:
     path: "/etc/nginx/conf.d/proxy_read_timeout.conf"
     contents: |
       proxy_read_timeout 180s;

Then, I added it in the containers/app.yml:

templates:
# ...
  - "templates/web.proxy_read_timeout.template.yml"

Then, I did

./launcher rebuild app

For now, I do not see these errors appear.

Topic		Replies	Views
No errors, instance not coming back up after rebuild installation	18	742	April 7, 2023
502 bad gateway with clean rebuild and no plugins except discourse docker installation	5	245	February 14, 2024
Redis connection timed out installation	29	8881	July 2, 2018
Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL) installation	22	3698	March 24, 2021
FAILED to rebuild discourse -- OpenURI::HTTPError: 401 Error: Invalid license key installation	11	1222	February 10, 2021

502 Bad Gateway for only two Topics

HEAD requests

GET requests

Logs

Further investigation

Related Topics