특정 두 주제에서만 502 Bad Gateway 오류 발생

niccokunzmann · 10월 26, 2022, 4:24오후

디스코urses 포럼에서 특정 주제(하나 이상)에서 502 Bad Gateway 오류가 발생합니다. 이 포럼은 제가 직접 호스팅하고 있습니다:

https://ammanvalley.foss.wales/t/wip-arduino-workshop/58

여기서 제 해결 과정을 기록해 보고 도움을 구하고자 합니다.

다음과 같은 작업을 수행했습니다.

./launcher rebuild app 실행 - Discourse stuck on "Currently Upgrading" - #2 by sam
모든 업그레이드를 수행하고 멈춰 있는 업그레이드를 취소함
- Stuck on “currently upgrading...”
- "Currently upgrading" even after a rebuild

어떤 조치를 취해야 할지 고민 중입니다. 조금 더 검색해 보겠습니다.

도커 컨테이너 재시작 - "Currently upgrading" even after a rebuild - #3 by codinghorror

HEAD 요청

다음은 페이지 목록입니다:

$ for i in `seq 100`; do ( status="`curl -L --head -o /dev/null -s -w '%{http_code}' https://ammanvalley.foss.wales/t/$i`"; echo "https://ammanvalley.foss.wales/t/$i - $status" ) done
https://ammanvalley.foss.wales/t/1 - 200
https://ammanvalley.foss.wales/t/2 - 404
https://ammanvalley.foss.wales/t/3 - 200
https://ammanvalley.foss.wales/t/4 - 404
https://ammanvalley.foss.wales/t/5 - 404
https://ammanvalley.foss.wales/t/6 - 404
https://ammanvalley.foss.wales/t/7 - 200 -- 502 오류 없음
https://ammanvalley.foss.wales/t/8 - 404
https://ammanvalley.foss.wales/t/9 - 404
https://ammanvalley.foss.wales/t/10 - 200
https://ammanvalley.foss.wales/t/11 - 200
https://ammanvalley.foss.wales/t/12 - 200
https://ammanvalley.foss.wales/t/13 - 200
https://ammanvalley.foss.wales/t/14 - 200
https://ammanvalley.foss.wales/t/15 - 200
https://ammanvalley.foss.wales/t/16 - 404
https://ammanvalley.foss.wales/t/17 - 404
https://ammanvalley.foss.wales/t/18 - 200
https://ammanvalley.foss.wales/t/19 - 200
https://ammanvalley.foss.wales/t/20 - 404
https://ammanvalley.foss.wales/t/21 - 200
https://ammanvalley.foss.wales/t/22 - 200
https://ammanvalley.foss.wales/t/23 - 200
https://ammanvalley.foss.wales/t/24 - 200
https://ammanvalley.foss.wales/t/25 - 200
https://ammanvalley.foss.wales/t/26 - 404
https://ammanvalley.foss.wales/t/27 - 404
https://ammanvalley.foss.wales/t/28 - 200
https://ammanvalley.foss.wales/t/29 - 404
https://ammanvalley.foss.wales/t/30 - 200
https://ammanvalley.foss.wales/t/31 - 200
https://ammanvalley.foss.wales/t/32 - 200
https://ammanvalley.foss.wales/t/33 - 200
https://ammanvalley.foss.wales/t/34 - 200
https://ammanvalley.foss.wales/t/35 - 200
https://ammanvalley.foss.wales/t/36 - 404
https://ammanvalley.foss.wales/t/37 - 200
https://ammanvalley.foss.wales/t/38 - 200
https://ammanvalley.foss.wales/t/39 - 200
https://ammanvalley.foss.wales/t/40 - 404
https://ammanvalley.foss.wales/t/41 - 200
https://ammanvalley.foss.wales/t/42 - 200
https://ammanvalley.foss.wales/t/43 - 200
https://ammanvalley.foss.wales/t/44 - 200
https://ammanvalley.foss.wales/t/45 - 200
https://ammanvalley.foss.wales/t/46 - 200
https://ammanvalley.foss.wales/t/47 - 404
https://ammanvalley.foss.wales/t/48 - 200
https://ammanvalley.foss.wales/t/49 - 200
https://ammanvalley.foss.wales/t/50 - 200
https://ammanvalley.foss.wales/t/51 - 200
https://ammanvalley.foss.wales/t/52 - 200
https://ammanvalley.foss.wales/t/53 - 200
https://ammanvalley.foss.wales/t/54 - 200
https://ammanvalley.foss.wales/t/55 - 200
https://ammanvalley.foss.wales/t/56 - 200
https://ammanvalley.foss.wales/t/57 - 200
https://ammanvalley.foss.wales/t/58 - 200 -- 502 오류 없음
https://ammanvalley.foss.wales/t/59 - 200
https://ammanvalley.foss.wales/t/60 - 200
https://ammanvalley.foss.wales/t/61 - 200
https://ammanvalley.foss.wales/t/62 - 404
https://ammanvalley.foss.wales/t/63 - 200
https://ammanvalley.foss.wales/t/64 - 404
https://ammanvalley.foss.wales/t/65 - 404
https://ammanvalley.foss.wales/t/66 - 200
https://ammanvalley.foss.wales/t/67 - 200
https://ammanvalley.foss.wales/t/68 - 404
https://ammanvalley.foss.wales/t/69 - 200
https://ammanvalley.foss.wales/t/70 - 404
https://ammanvalley.foss.wales/t/71 - 404
https://ammanvalley.foss.wales/t/72 - 404
https://ammanvalley.foss.wales/t/73 - 200
https://ammanvalley.foss.wales/t/74 - 404
https://ammanvalley.foss.wales/t/75 - 404
https://ammanvalley.foss.wales/t/76 - 404
https://ammanvalley.foss.wales/t/77 - 404
https://ammanvalley.foss.wales/t/78 - 404
https://ammanvalley.foss.wales/t/79 - 404
https://ammanvalley.foss.wales/t/80 - 404
https://ammanvalley.foss.wales/t/81 - 404
https://ammanvalley.foss.wales/t/82 - 404
https://ammanvalley.foss.wales/t/83 - 404
https://ammanvalley.foss.wales/t/84 - 404
https://ammanvalley.foss.wales/t/85 - 404
https://ammanvalley.foss.wales/t/86 - 404
https://ammanvalley.foss.wales/t/87 - 404
https://ammanvalley.foss.wales/t/88 - 404
https://ammanvalley.foss.wales/t/89 - 404
https://ammanvalley.foss.wales/t/90 - 404
https://ammanvalley.foss.wales/t/91 - 404
https://ammanvalley.foss.wales/t/92 - 404
https://ammanvalley.foss.wales/t/93 - 404
https://ammanvalley.foss.wales/t/94 - 404
https://ammanvalley.foss.wales/t/95 - 404
https://ammanvalley.foss.wales/t/96 - 404
https://ammanvalley.foss.wales/t/97 - 404
https://ammanvalley.foss.wales/t/98 - 404
https://ammanvalley.foss.wales/t/99 - 404
https://ammanvalley.foss.wales/t/100 - 404

GET 요청

$ for i in `seq 100`; do ( status="`curl -L -o /dev/null -s -w '%{http_code}' https://ammanvalley.foss.wales/t/$i`"; echo "https://ammanvalley.foss.wales/t/$i - $status" ) done
https://ammanvalley.foss.wales/t/1 - 200
https://ammanvalley.foss.wales/t/2 - 404
https://ammanvalley.foss.wales/t/3 - 200
https://ammanvalley.foss.wales/t/4 - 404
https://ammanvalley.foss.wales/t/5 - 404
https://ammanvalley.foss.wales/t/6 - 404
https://ammanvalley.foss.wales/t/7 - 502  -- 502 오류 발생
https://ammanvalley.foss.wales/t/8 - 404
https://ammanvalley.foss.wales/t/9 - 404
https://ammanvalley.foss.wales/t/10 - 200
https://ammanvalley.foss.wales/t/11 - 200
https://ammanvalley.foss.wales/t/12 - 200
https://ammanvalley.foss.wales/t/13 - 200
https://ammanvalley.foss.wales/t/14 - 200
https://ammanvalley.foss.wales/t/15 - 200
https://ammanvalley.foss.wales/t/16 - 404
https://ammanvalley.foss.wales/t/17 - 404
https://ammanvalley.foss.wales/t/18 - 200
https://ammanvalley.foss.wales/t/19 - 200
https://ammanvalley.foss.wales/t/20 - 404
https://ammanvalley.foss.wales/t/21 - 200
https://ammanvalley.foss.wales/t/22 - 200
https://ammanvalley.foss.wales/t/23 - 200
https://ammanvalley.foss.wales/t/24 - 200
https://ammanvalley.foss.wales/t/25 - 200
https://ammanvalley.foss.wales/t/26 - 404
https://ammanvalley.foss.wales/t/27 - 404
https://ammanvalley.foss.wales/t/28 - 200
https://ammanvalley.foss.wales/t/29 - 404
https://ammanvalley.foss.wales/t/30 - 200
https://ammanvalley.foss.wales/t/31 - 200
https://ammanvalley.foss.wales/t/32 - 200
https://ammanvalley.foss.wales/t/33 - 200
https://ammanvalley.foss.wales/t/34 - 200
https://ammanvalley.foss.wales/t/35 - 200
https://ammanvalley.foss.wales/t/36 - 404
https://ammanvalley.foss.wales/t/37 - 200
https://ammanvalley.foss.wales/t/38 - 200
https://ammanvalley.foss.wales/t/39 - 200
https://ammanvalley.foss.wales/t/40 - 404
https://ammanvalley.foss.wales/t/41 - 200
https://ammanvalley.foss.wales/t/42 - 200
https://ammanvalley.foss.wales/t/43 - 200
https://ammanvalley.foss.wales/t/44 - 200
https://ammanvalley.foss.wales/t/45 - 200
https://ammanvalley.foss.wales/t/46 - 200
https://ammanvalley.foss.wales/t/47 - 404
https://ammanvalley.foss.wales/t/48 - 200
https://ammanvalley.foss.wales/t/49 - 200
https://ammanvalley.foss.wales/t/50 - 200
https://ammanvalley.foss.wales/t/51 - 200
https://ammanvalley.foss.wales/t/52 - 200
https://ammanvalley.foss.wales/t/53 - 200
https://ammanvalley.foss.wales/t/54 - 200
https://ammanvalley.foss.wales/t/55 - 200
https://ammanvalley.foss.wales/t/56 - 200
https://ammanvalley.foss.wales/t/57 - 200
https://ammanvalley.foss.wales/t/58 - 502 -- 502 오류 발생
https://ammanvalley.foss.wales/t/59 - 200
https://ammanvalley.foss.wales/t/60 - 200
https://ammanvalley.foss.wales/t/61 - 200
https://ammanvalley.foss.wales/t/62 - 404
https://ammanvalley.foss.wales/t/63 - 200
https://ammanvalley.foss.wales/t/64 - 404
https://ammanvalley.foss.wales/t/65 - 404
https://ammanvalley.foss.wales/t/66 - 200
https://ammanvalley.foss.wales/t/67 - 200
https://ammanvalley.foss.wales/t/68 - 404
https://ammanvalley.foss.wales/t/69 - 200
https://ammanvalley.foss.wales/t/70 - 404
https://ammanvalley.foss.wales/t/71 - 404
https://ammanvalley.foss.wales/t/72 - 404
https://ammanvalley.foss.wales/t/73 - 200
https://ammanvalley.foss.wales/t/74 - 404
https://ammanvalley.foss.wales/t/75 - 404
https://ammanvalley.foss.wales/t/76 - 404
https://ammanvalley.foss.wales/t/77 - 404
https://ammanvalley.foss.wales/t/78 - 404
https://ammanvalley.foss.wales/t/79 - 404
https://ammanvalley.foss.wales/t/80 - 404
https://ammanvalley.foss.wales/t/81 - 404
https://ammanvalley.foss.wales/t/82 - 404
https://ammanvalley.foss.wales/t/83 - 404
https://ammanvalley.foss.wales/t/84 - 404
https://ammanvalley.foss.wales/t/85 - 404
https://ammanvalley.foss.wales/t/86 - 404
https://ammanvalley.foss.wales/t/87 - 404
https://ammanvalley.foss.wales/t/88 - 404
https://ammanvalley.foss.wales/t/89 - 404
https://ammanvalley.foss.wales/t/90 - 404
https://ammanvalley.foss.wales/t/91 - 404
https://ammanvalley.foss.wales/t/92 - 404
https://ammanvalley.foss.wales/t/93 - 404
https://ammanvalley.foss.wales/t/94 - 404
https://ammanvalley.foss.wales/t/95 - 404
https://ammanvalley.foss.wales/t/96 - 404
https://ammanvalley.foss.wales/t/97 - 404
https://ammanvalley.foss.wales/t/98 - 404
https://ammanvalley.foss.wales/t/99 - 404
https://ammanvalley.foss.wales/t/100 - 404

로그

./launcher logs app에서 확인한 오류 목록:

...
[Wed 26 Oct 2022 04:28:27 PM UTC] Run reload cmd: sv reload nginx
fail: nginx: runsv not running
[Tue 25 Oct 2022 01:07:47 PM UTC] Run reload cmd: sv reload nginx
warning: nginx: unable to open supervise/ok: file does not exist
[Tue 25 Oct 2022 01:07:47 PM UTC] ESC[1;31mReload error for :ESC[0m
Started runsvdir, PID is 552
[Tue 25 Oct 2022 01:07:45 PM UTC] Run reload cmd: sv reload nginx
warning: nginx: unable to open supervise/ok: file does not exist
[Tue 25 Oct 2022 01:07:45 PM UTC] ESC[1;31mReload error for :ESC[0m
[Tue 25 Oct 2022 01:07:46 PM UTC] Domains not changed.

추가 조사

이제 어떻게 진행해야 할지 막막합니다. 두 개의 주제에만 영향을 미치고 있습니다.
미리보기는 작동하지만 데이터는 로드되지 않습니다:
https://ammanvalley.foss.wales/t/how-this-forum-is-hosted/13/5?u=niccokunzmann

게시물을 어떻게 복구할 수 있을까요?

pfaffman · 10월 26, 2022, 7:50오후

그 주제들이 삭제된 건가요? 개인 메시지(PM)일 수도 있나요? 관리자 권한으로 요청하신 건가요?

아, 죄송합니다. 502 오류네요. /var/discourse/shared/logs/rails/production.log 파일을 확인해 보시고, 거기서 단서를 찾을 수 있는지 살펴보세요. 또한 https://ammanvalley.foss.wales/logs 도 참고해 보세요.

niccokunzmann · 10월 26, 2022, 8:59오후

음, rails는 잘 돌아가고 로그 기록도 매우 빠릅니다.

Started GET "/t/wip-arduino-workshop/58" for 8.8.8.8 at 2022-10-26 20:57:4
8 +0000
Processing by TopicsController#show as HTML
  Parameters: {"slug"=>"wip-arduino-workshop", "topic_id"=>"58"}
start
done

그런데 502 오류가 나타날 때까지 웹사이트가 여전히 잠시 동안 로드됩니다.

하지만 /var/discourse/shared/standalone/log/rails/production.log에서 많은 오류를 확인합니다:

Failed to report error: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL) 3 Error fetching job: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL) 
Failed to report error: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL) 3 Error fetching job: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL) 
Failed to report error: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL) 3 Error fetching job: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL) 
Failed to report error: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL) 2 Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL) subscribe failed, reconnecting in 1 second. Call stack /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.7.1/lib/redis/client.rb:398:in `rescue in establish_connection'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.7.1/lib/redis/client.rb:379:in `establish_connection'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.7.1/lib/redis/client.rb:115:in `block in connect'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.7.1/lib/redis/client.rb:344:in `with_reconnect'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.7.1/lib/redis/client.rb:114:in `connect'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.7.1/lib/redis/client.rb:417:in `ensure_connected'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.7.1/lib/redis/client.rb:269:in `block in process'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.7.1/lib/redis/client.rb:356:in `logging'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.7.1/lib/redis/client.rb:268:in `process'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.7.1/lib/redis/client.rb:175:in `block in call_loop'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.7.1/lib/redis/client.rb:331:in `with_socket_timeout'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.7.1/lib/redis/client.rb:174:in `call_loop'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.7.1/lib/redis/subscribe.rb:44:in `subscription'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.7.1/lib/redis/subscribe.rb:14:in `subscribe'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.7.1/lib/redis.rb:287:in `_subscription'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.7.1/lib/redis/commands/pubsub.rb:20:in `block in subscribe'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.7.1/lib/redis.rb:264:in `block in synchronize'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.7.1/lib/redis.rb:264:in `synchronize'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.7.1/lib/redis.rb:264:in `synchronize'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.7.1/lib/redis/commands/pubsub.rb:19:in `subscribe'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/message_bus-4.2.0/lib/message_bus/backends/redis.rb:302:in `global_subscribe'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/message_bus-4.2.0/lib/message_bus.rb:763:in `global_subscribe_thread'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/message_bus-4.2.0/lib/message_bus.rb:734:in `block in new_subscriber_thread' 
Failed to report error: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL) 3 Error fetching job: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL) 
Failed to report error: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL) 3 Error fetching job: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL) 
Failed to report error: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL) 3 Job exception: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL)
 
Failed to report error: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL) 3 Job exception: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL)
 
2022-10-26 16:31:02 +0000: Issuing stop to sidekiq
I, [2022-10-26T16:31:04.563256 #2079]  INFO -- : executing ["/var/www/discourse/vendor/bundle/ruby/2.7.0/bin/unicorn", "-E", "production", "-c", "config/unico
rn.conf.rb", {10=>#<Unicorn::TCPSrv:fd 10, AF_INET, 127.0.0.1, 3000>}] (in /var/www/discourse)
I, [2022-10-26T16:31:04.563661 #2079]  INFO -- : forked child re-executing...
I, [2022-10-26T16:31:07.250986 #2079]  INFO -- : inherited addr=127.0.0.1:3000 fd=10
I, [2022-10-26T16:31:07.252783 #2079]  INFO -- : Refreshing Gem list
I, [2022-10-26T16:31:44.617034 #2079]  INFO -- : starting 1 supervised sidekiqs
I, [2022-10-26T16:31:47.752185 #2079]  INFO -- : master process ready
I, [2022-10-26T16:31:55.330063 #589]  INFO -- : reaped #<Process::Status: pid 704 exit 0> worker=1
I, [2022-10-26T16:31:55.333326 #589]  INFO -- : reaped #<Process::Status: pid 714 exit 0> worker=2
I, [2022-10-26T16:31:55.333928 #589]  INFO -- : reaped #<Process::Status: pid 1740 exit 0> worker=0
I, [2022-10-26T16:31:55.335869 #589]  INFO -- : master complete
I, [2022-10-26T16:32:26.504213 #2414]  INFO -- : worker=0 ready
2022-10-26 16:32:27 +0000: Issuing stop to sidekiq
I, [2022-10-26T16:32:28.194731 #2429]  INFO -- : worker=1 ready
I, [2022-10-26T16:32:29.025684 #2445]  INFO -- : worker=2 ready
Process would not terminate cleanly, force quitting. pid: 2402 Demon::Sidekiq
I, [2022-10-26T16:32:38.212437 #2079]  INFO -- : reaped #<Process::Status: pid 2429 exit 0> worker=1
I, [2022-10-26T16:32:38.212904 #2079]  INFO -- : reaped #<Process::Status: pid 2445 exit 0> worker=2
oxipng worker: could not get version of oxipng at /usr/local/bin/oxipng; please provide proper binary or disable this worker (--no-oxipng argument or `:oxipng => false` through options)
oxipng worker: could not get version of oxipng at /usr/local/bin/oxipng; please provide proper binary or disable this worker (--no-oxipng argument or `:oxipng => false` through options)
E, [2022-10-26T17:47:01.151902 #5094] ERROR -- : worker=0 PID:7920 timeout (31s > 30s), killing
I, [2022-10-26T21:01:44.288907 #10359]  INFO -- : worker=1 ready
I, [2022-10-26T21:02:06.344069 #10391]  INFO -- : worker=0 ready
I, [2022-10-26T21:02:10.336134 #10410]  INFO -- : worker=2 ready
D, [2022-10-26T21:02:14.022594 #595] DEBUG -- : waiting 16.0s after suspend/hibernation
E, [2022-10-26T21:02:39.047922 #595] ERROR -- : worker=1 PID:10359 timeout (31s > 30s), killing
E, [2022-10-26T21:02:39.175049 #595] ERROR -- : reaped #<Process::Status: pid 10359 SIGKILL (signal 9)> worker=1
E, [2022-10-26T21:02:44.202628 #595] ERROR -- : worker=0 PID:10391 timeout (31s > 30s), killing
E, [2022-10-26T21:02:44.327961 #595] ERROR -- : reaped #<Process::Status: pid 10391 SIGKILL (signal 9)> worker=0
I, [2022-10-26T21:03:08.803095 #10488]  INFO -- : worker=1 ready
E, [2022-10-26T21:03:11.378475 #595] ERROR -- : worker=2 PID:10410 timeout (31s > 30s), killing
E, [2022-10-26T21:03:11.456868 #595] ERROR -- : worker=2 PID:10410 timeout (31s > 30s), killing
E, [2022-10-26T21:03:11.548047 #595] ERROR -- : reaped #<Process::Status: pid 10410 SIGKILL (signal 9)> worker=2

Falco · 10월 26, 2022, 9:03오후

또한 /var/log/nginx/ 파일들을 확인해 보세요.

pfaffman · 10월 26, 2022, 9:04오후

https://ammanvalley.foss.wales/t/welcome-to-amman-valley-makerspace/7.json이 아무것도 반환하지 않고 502 오류가 발생하는 것을 확인할 수 있습니다.

현재 설치된 플러그인이 있나요?

최근에 재부팅을 했나요? RAM과 스왑 공간은 얼마나 사용 가능한가요?

Redis에 어떤 문제가 있는 것으로 보입니다. 이 두 가지 주제(토픽)가 그 원인이 될 만한 이유는 떠오르지 않습니다.

niccokunzmann · 10월 27, 2022, 5:49오전

https://ammanvalley.foss.wales/t/plugins-to-install/14

아니요, 하지만 웹 인터페이스를 통해 앱을 재시작하고, 다시 빌드하며 업데이트했습니다.

nginx 출력이 access.log에서 타임아웃을 기다린 후 다음과 같은 내용을 생성합니다:

[27/Oct/2022:05:45:48 +0000] "ammanvalley.foss.wales" 8.8.8.8 "GET /t/wip-arduino-workshop/58 HTTP/2.0" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:105.
0) Gecko/20100101 Firefox/105.0" "-" 502 217 "https://meta.discourse.org/" 30.516 30.516 "-" "-" "-" "-" "-" "-" "-"

error.log에서는 다음 내용을 볼 수 있습니다:

2022/10/26 21:03:48 [error] 578#578: *509 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 46.59.192.44, se
rver: _, request: "GET /t/wip-arduino-workshop/58 HTTP/2.0", upstream: "http://127.0.0.1:3000/t/wip-arduino-workshop/58", host: "ammanvalley.foss.wales", refe
rrer: "https://meta.discourse.org/"
2022/10/26 21:05:58 [error] 575#575: *591 upstream prematurely closed connection while reading response header from upstream, client: 189.79.134.158, server:
_, request: "GET /t/wip-arduino-workshop/58 HTTP/2.0", upstream: "http://127.0.0.1:3000/t/wip-arduino-workshop/58", host: "ammanvalley.foss.wales", referrer:
"https://meta.discourse.org/"
2022/10/26 21:52:07 [error] 578#578: *1117 upstream prematurely closed connection while reading response header from upstream, client: 216.68.248.7, server: _
, request: "GET /t/welcome-to-amman-valley-makerspace/7 HTTP/1.1", upstream: "http://127.0.0.1:3000/t/welcome-to-amman-valley-makerspace/7", host: "ammanvalle
y.foss.wales"
2022/10/26 22:33:23 [error] 578#578: *1534 upstream prematurely closed connection while reading response header from upstream, client: 64.222.212.196, server:
 _, request: "GET /t/wip-arduino-workshop/58 HTTP/1.1", upstream: "http://127.0.0.1:3000/t/wip-arduino-workshop/58", host: "ammanvalley.foss.wales"
2022/10/27 05:21:21 [crit] 578#578: *2066 SSL_do_handshake() failed (SSL: error:141CF06C:SSL routines:tls_parse_ctos_key_share:bad key share) while SSL handsh
aking, client: 64.62.197.155, server: 0.0.0.0:443
2022/10/27 05:44:54 [error] 578#578: *2073 upstream prematurely closed connection while reading response header from upstream, client: 92.40.170.157, server:
_, request: "GET /t/wip-arduino-workshop/58 HTTP/2.0", upstream: "http://127.0.0.1:3000/t/wip-arduino-workshop/58", host: "ammanvalley.foss.wales", referrer:
"https://meta.discourse.org/"
2022/10/27 05:45:48 [error] 578#578: *2073 upstream prematurely closed connection while reading response header from upstream, client: 92.40.170.157, server:
_, request: "GET /t/wip-arduino-workshop/58 HTTP/2.0", upstream: "http://127.0.0.1:3000/t/wip-arduino-workshop/58", host: "ammanvalley.foss.wales", referrer:
"https://meta.discourse.org/"
2022/10/27 05:48:48 [error] 575#575: *2161 upstream prematurely closed connection while reading response header from upstream, client: 92.40.170.156, server:
_, request: "GET /t/wip-arduino-workshop/58 HTTP/2.0", upstream: "http://127.0.0.1:3000/t/wip-arduino-workshop/58", host: "ammanvalley.foss.wales", referrer:
"https://meta.discourse.org/"

SWAP: 311M/2.97G RAM: 1.13G/1.93G

Benjamin_D · 10월 27, 2022, 6:07오전

하지만 원본 데이터는 있고, 무해해 보입니다.
레일즈 콘솔에서는 어떤 결과가 나올지 궁금하네요…

rails c
Topic.where(id: 58)

niccokunzmann · 10월 28, 2022, 8:23오전

레일즈 콘솔로 들어갔습니다.

pry(main)> Topic.where(id: 58)
=> [#<Topic:0x0000aaab01b22260
  id: 58,
  title: "[WIP] Arduino Workshop",
  last_posted_at: Sun, 23 Oct 2022 22:13:09.895053000 UTC +00:00,
  created_at: Fri, 23 Sep 2022 10:30:18.827248000 UTC +00:00,
  updated_at: Sun, 23 Oct 2022 22:13:10.291385000 UTC +00:00,
  views: 26,
  posts_count: 6,
  user_id: 1,
  last_post_user_id: 1,
  reply_count: 4,
  featured_user1_id: 2,
  featured_user2_id: nil,
  featured_user3_id: nil,
  deleted_at: nil,
  highest_post_number: 6,
  like_count: 1,
  incoming_link_count: 16,
  category_id: 6,
  visible: true,
  moderator_posts_count: 0,
  closed: false,
  archived: false,
  bumped_at: Sun, 23 Oct 2022 22:13:09.895053000 UTC +00:00,
  has_summary: false,
  archetype: "regular",
  featured_user4_id: nil,
  notify_moderators_count: 0,
  spam_count: 0,
  pinned_at: nil,
  score: 11.233333333333333,
  percent_rank: 1.0,
  subtype: nil,
  slug: "wip-arduino-workshop",
  deleted_by_id: nil,
  participant_count: 2,
  word_count: 184,
  excerpt:
   "I can run a workshop to introduce into programming arduinos. I post this here, so that we can organize \n\nOrganization\n\n\n make a poster @DomAman made one\n\n get material down from the community center\n\n make a list of what&hellip;",
  pinned_globally: false,
  pinned_until: nil,
  fancy_title: "[WIP] Arduino Workshop",
  highest_staff_post_number: 6,
  featured_link: nil,
  reviewable_score: 0.0,
  image_upload_id: nil,
  slow_mode_seconds: 0,
  bannered_until: nil,
  external_id: nil>]

RGJ · 10월 30, 2022, 5:17오후

https://ammanvalley.foss.wales/raw/58 은 정상적으로 작동하는 것 같습니다.
이 문제가 언제부터 시작되었나요?

niccokunzmann · 11월 5, 2022, 12:03오전

로우 링크를 공유해 주셔서 정말 감사합니다! 덕분에 주제에 대한 근거를 마련할 수 있었어요.

새 링크가 이제 502 오류를 반환하고 있습니다: https://ammanvalley.foss.wales/t/hovercraft-project/93

음. 업데이트를 수행하고 플러그인을 비활성화해 보았어요. 그런데 새 링크는 방금 생성된 것이었습니다. 업데이트도, 비활성화도, 기타 관리 작업도 하지 않았는데 말이죠.

아니! 이제 작동하네요! 음. 이해가 안 됩니다.
이제 해결된 것 같습니다: 세 개의 게시물이 모두 오류 없이 HTML로 표시되고 있어요. 이제 이 문제를 닫아야 할까요? 이 문제를 계속 추적해 보고 싶으신 분이 있다면 댓글을 남겨 주세요. 저는 닫는 시점을 두고 기다리겠습니다.

niccokunzmann · 11월 6, 2022, 5:42오후

아니요! 아직 해결되지 않았습니다. 포럼이 30분 전에 재시작되었고 다른 게시물에서도 502 오류가 나타납니다:

https://ammanvalley.foss.wales/t/hovercraft-project/93
https://ammanvalley.foss.wales/t/welcome-to-amman-valley-makerspace/7
https://ammanvalley.foss.wales/t/wip-arduino-workshop/58

포럼은 가끔 느리게 로드됩니다. 타임아웃을 늘리는 방법이 있을까요? 원본 링크는 정상적으로 작동합니다!

이 문제를 어떻게 처리해야 할지 정말로 모르겠습니다.

RGJ · 11월 6, 2022, 6:48오후

즉, 502 오류는 실제로 타임아웃 때문에 발생하는 것이며, 정확히 30초를 조금 넘어서는 시점에 발생한다는 뜻입니다.

Sidekiq에서는 어떤 일이 일어나고 있나요? 대기열이 크게 쌓여 있나요?

pfaffman · 11월 6, 2022, 6:53오후

그렇다면 그 문제를 해결해야겠네요.

RAM을 두 배로 늘리거나 더 빠른 CPU를 사용하는 것을 권장합니다. 공유 호스팅(디지털 오션의 드롭렛 대부분과 같은)을 사용하고 있다면, 사용자의 환경을 느리게 만드는 "시끄러운 이웃(noisy neighbor)"이 있을 수 있습니다.

또한 아직 OS를 재부팅하지 않았다면, RAM을 많이 차지하는 라이브러리가 업그레이드되었을 수도 있습니다.

Stephen · 11월 6, 2022, 6:54오후

정말 ARM CPU에서 실행 중인가요, 아니면 그 태그가 잘못된 건가요?

niccokunzmann · 11월 6, 2022, 9:31오후

네, ARM CPU입니다. 실수로 더 자세히 알아보니 502 페이지를 열면 이 프로세스가 급증하는 것을 확인했습니다:

unicorn worker[1] -E production -c config/unicorn.conf.rb

CPU 사용률이 99.0%입니다:

그리고 502 페이지가 브라우저로 반환된 후에도 몇 초간 이 상태가 유지됩니다. 이것이 무엇을 하고 있는 건가요?

이것을 어떻게 확인할 수 있나요?

음. 이것은 최소 요구 사항입니다. 또한 RAM은 절대 급증하지도 않고 움직이는 것조차 아닌 것 같습니다.

RGJ · 11월 6, 2022, 9:50오후

즉, 유니콘(unicorn) 프로세스가 페이지를 렌더링하고 nginx로 넘기는 책임을 맡고 있습니다. 30초가 지나면 nginx는 더 이상 기다리지 않지만, 유니콘 프로세스는 그 사실을 알지 못하므로 페이지 렌더링을 계속합니다.

질문은 이렇습니다. 유니콘이 특정 토픽을 렌더링하는 데 왜 이렇게 오래 걸리는 것일까요?

RAM은 문제없을 것 같습니다.

무엇이 일어나고 있는지 확인하기 위해 미니 프로파일러(mini-profiler)를 사용해 볼 수 있습니다. 여기에서 그 방법을 확인할 수 있습니다(상관없는 다른 문제의 맥락에서 설명되어 있습니다).

관리자로 /sidekiq에 접속하여 사이드킥(Sidekiq)을 확인해 볼 수도 있습니다.

niccokunzmann · 11월 6, 2022, 9:57오후

Sidekiq에서 실제로 일어나는 일이 거의 없어 보입니다: 프로세스 스파이크 한 번은 브라우저가 GET을 보낼 때 발생했습니다.

답변이 있는 주제에서 이 문제가 발생하는 것을 확인했습니다. 이 주제들을 더 자주 방문하기 때문일 수도 있습니다.

niccokunzmann · 11월 9, 2022, 3:57오후

열려고 하면 문제가 발생합니다. 단축키가 작동하지 않습니다. 502 오류를 발생시키는 게시글을 클릭했을 때, 클릭한 페이지에서 열렸다가 사라졌습니다. 지금은 어떻게 열어야 할지 모르겠습니다. 다만, 일부 JavaScript를 실행할 수는 있습니다.

niccokunzmann · 11월 15, 2022, 11:19오후

지금 내 질문은 다음과 같습니다. nginx의 타임아웃을 예를 들어 30초에서 약 2분으로 늘릴 수 있는 방법이 있을까요? 이렇게 하면 문제가 해결될 가능성이 높습니다.

https://stackoverflow.com/questions/63801913/nginx-timeout-20s-causes-502-bad-gateway
templates/web.letsencrypt.ssl.template.yml - nginx 설정에 접근합니다.
Module ngx_http_proxy_module

제 결론은 타임아웃을 설정하는 .conf 파일을 /etc/nginx/sites-enabled/ 디렉터리에 추가하는 템플릿을 작성할 수 있다는 것입니다.

또한, 답변이 게시될 때 502 오류가 발생할 확률이 더 높다는 것을 발견했습니다. 제 추측으로는 포럼이 페이지를 다시 빌드해야 하고, 그 과정에 시간이 걸리기 때문인 것 같습니다. 이것이 맞을까요? 궁금한 점이 있습니다: discourse 페이지의 재빌드를 수동으로 트리거할 수 있을까요?

niccokunzmann · 11월 17, 2022, 6:27오후

내 시도:

/var/discourse# cat templates/web.proxy_read_timeout.template.yml 
#
# 느린 시스템에서 nginx 프록시 읽기 시간 초과를 설정합니다.
# https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_read_timeout 참조
# https://meta.discourse.org/t/502-bad-gateway-for-only-two-topics/242981/20?u=niccokunzmann 참조
#
run:
  - file:
     path: "/etc/nginx/conf.d/proxy_read_timeout.conf"
     contents: |
       proxy_read_timeout 180s;

그 다음, containers/app.yml에 추가했습니다:

templates:
# ...
  - "templates/web.proxy_read_timeout.template.yml"

그 후, 다음 명령을 실행했습니다.

./launcher rebuild app

현재로서는 이러한 오류가 더 이상 발생하지 않는 것을 확인했습니다.

글		댓글	조회수
Bursts of 502 Service Unavailable, pointers to debug Self-hosting	15	1812	2월 18, 2018
502 bad gateway with clean rebuild and no plugins except discourse docker Self-hosting	5	508	2월 14, 2024
502 Bad Gateway - nginx/1.14.0 (Ubuntu): Unable to find image locally + Error response from daemon Self-hosting	9	9451	11월 18, 2019
Moving posts returns 502 bad gateway Self-hosting performance , server-resources	71	6279	1월 12, 2025
Nginx issue, when i am stop and starting the app getting "502 Bad Gateway" Self-hosting	15	668	4월 16, 2024

특정 두 주제에서만 502 Bad Gateway 오류 발생

HEAD 요청

GET 요청

로그

추가 조사

Related topics