유니콘 워커가 너무 많습니다 - 업그레이드를 위해 정지하지 못했나요?

Ed_S · 11월 2, 2020, 11:13오전

Docker를 방금 업그레이드했는데, 업그레이드 중에 이전 unicorn 프로세스들이 여전히 실행되고 있는 것을 확인했습니다. 지금은 두 세트의 unicorn 프로세스가 실행 중인 것 같습니다. 그래서 이전 프로세스를 중지하도록 의도된 작업이 실행되지 않았거나, 실행은 되었지만 제대로 동작하지 않은 것 같습니다.

업그레이드 중 (10월 19일은 이전 업데이트 날짜입니다):

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
discour+  6992 93.5 18.9 13543864 376908 ?     Rl   11:05   0:55 /var/www/discourse/vendor/bundle/ruby/2.6.0/bin/rake assets:precompile
discour+  6611  0.1 13.4 8799684 268616 ?      Sl   11:04   0:00 unicorn worker[0] -E production -c config/unicorn.conf.rb
discour+  6608  0.0 13.2 8805772 264008 ?      S    11:04   0:00 unicorn worker[0] -E production -c config/unicorn.conf.rb
discour+  6609  0.0 13.2 8797576 263936 ?      Ss   11:04   0:00 unicorn worker[0] -E production -c config/unicorn.conf.rb
discour+  6834  5.6 11.9 8733980 238900 ?      Sl   11:04   0:06 unicorn worker[0] -E production -c config/unicorn.conf.rb
discour+  8868  0.0 11.6 8816880 231564 ?      S    Oct19   0:01 unicorn worker[0] -E production -c config/unicorn.conf.rb
discour+  8867  0.0 11.4 8763632 227900 ?      Ss   Oct19   0:00 unicorn worker[0] -E production -c config/unicorn.conf.rb
discour+  8866  0.0 11.4 8755436 227892 ?      S    Oct19   0:00 unicorn worker[0] -E production -c config/unicorn.conf.rb
discour+  6661  8.0  9.9 315668 198900 ?       Sl   11:04   0:10 unicorn master -E production -c config/unicorn.conf.rb
discour+  7081  0.0  4.2 8805768 84856 ?       S    Oct19   0:00 unicorn worker[1] -E production -c config/unicorn.conf.rb
postgres   693  0.0  3.3 214172 67016 ?        Ss   Sep28   1:48 postgres: 12/main: checkpointer   
discour+  7080  0.0  2.0 8805768 41456 ?       Ss   Oct19   0:00 unicorn worker[1] -E production -c config/unicorn.conf.rb
discour+  7079  0.0  2.0 8805768 41348 ?       S    Oct19   0:00 unicorn worker[1] -E production -c config/unicorn.conf.rb
postgres  6750  1.0  1.6 221452 33028 ?        Ss   11:04   0:01 postgres: 12/main: discourse discourse [local] idle
postgres  6427  0.0  1.0 217244 20424 ?        Ss   11:04   0:00 postgres: 12/main: discourse discourse [local] idle
postgres  6937  0.0  0.9 215828 18676 ?        Ss   11:05   0:00 postgres: 12/main: discourse discourse [local] idle
postgres  7086  1.0  0.8 215828 16332 ?        Ss   11:06   0:00 postgres: 12/main: discourse discourse [local] idle
postgres   694  0.0  0.3 214060  6404 ?        Ss   Sep28   1:25 postgres: 12/main: background writer   
postgres   672  0.0  0.3 213928  6092 ?        S    Sep28   5:52 /usr/lib/postgresql/12/bin/postmaster -D /etc/postgresql/12/main
www-data 18624  0.0  0.2  57792  5956 ?        S    Oct06   1:50 nginx: worker process
redis      669  0.3  0.2  52724  5196 ?        Sl   Sep28 184:37 /usr/bin/redis-server *:6379
postgres   695  0.0  0.2 213928  4524 ?        Ss   Sep28  15:14 postgres: 12/main: walwriter   
discour+  7088  0.0  0.1  18644  3400 ?        R    11:06   0:00 ps aux --sort -rss
discour+  7087  0.0  0.1  13708  2156 ?        S    11:06   0:00 sleep 1
discour+   673  0.0  0.0  15132  1696 ?        S    Sep28  34:54 /bin/bash config/unicorn_launcher -E production -c config/unicorn.conf.rb
postgres   696  0.0  0.0 214600  1368 ?        Ss   Sep28   3:28 postgres: 12/main: autovacuum launcher   
postgres   698  0.0  0.0 214464   980 ?        Ss   Sep28   0:11 postgres: 12/main: logical replication launcher   
discour+  6991  0.0  0.0   2388   712 ?        S    11:05   0:00 sh -c cd /var/www/discourse && RUBY_GC_MALLOC_LIMIT_MAX=20971520 RUBY_GC_OLDMALLOC_LIMIT_MAX=20971520 RUBY_GC_HEAP_GROWTH_MAX_SLOTS=50000 RUBY_GC_HEAP_OLDOBJECT_LIMIT_FACTOR=0.9  bundle exec rake assets:precompile 2>&1
postgres   697  0.0  0.0  69020   684 ?        Ss   Sep28   7:57 postgres: 12/main: stats collector   
root       674  0.0  0.0   8440   572 ?        S    Sep28   0:27 cron -f
root       671  0.0  0.0   2304   440 ?        S    Sep28   0:00 svlogd /var/log/postgres
www-data 18625  0.0  0.0  56472   420 ?        S    Oct06   0:34 nginx: cache manager process
root       670  0.0  0.0  56148   400 ?        S    Sep28   0:00 nginx: master process /usr/sbin/nginx
root       675  0.0  0.0 156188   288 ?        Sl   Sep28   0:07 rsyslogd -n
root       664  0.0  0.0   2160   240 ?        Ss   Sep28   0:00 runsv unicorn
root       666  0.0  0.0   2160   224 ?        Ss   Sep28   0:00 runsv nginx
root       668  0.0  0.0   2304   216 ?        S    Sep28   0:06 svlogd /var/log/redis
root       661  0.0  0.0   2312    24 pts/0    S+   Sep28   1:16 /usr/bin/runsvdir -P /etc/service
root       663  0.0  0.0   2160    24 ?        Ss   Sep28   0:00 runsv cron
root       665  0.0  0.0   2160    24 ?        Ss   Sep28   0:00 runsv postgres
root       667  0.0  0.0   2160    24 ?        Ss   Sep28   0:00 runsv redis
root       662  0.0  0.0   2160    20 ?        Ss   Sep28   0:00 runsv rsyslog
root         1  0.0  0.0   6664     4 pts/0    Ss+  Sep28   0:00 /bin/bash /sbin/boot

업그레이드 후:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
discour+  7778  2.6 13.1 8746268 261300 ?      Sl   11:08   0:06 unicorn worker[1] -E production -c config/unicorn.conf.rb
discour+  7764  2.5 13.0 8748324 259192 ?      Sl   11:08   0:06 unicorn worker[0] -E production -c config/unicorn.conf.rb
discour+  6608  0.0 12.9 8805772 258420 ?      S    11:04   0:00 unicorn worker[0] -E production -c config/unicorn.conf.rb
discour+  6609  0.0 12.9 8797576 258348 ?      Ss   11:04   0:00 unicorn worker[0] -E production -c config/unicorn.conf.rb
discour+  6611  0.1 12.5 8830348 250348 ?      S    11:04   0:00 unicorn worker[0] -E production -c config/unicorn.conf.rb
discour+  7733  0.9 11.4 425296 227348 ?       SNl  11:08   0:02 sidekiq 6.1.2 discourse [0 of 5 busy]
discour+  7569  4.9 11.0 327956 219724 ?       Sl   11:07   0:13 unicorn master -E production -c config/unicorn.conf.rb
discour+  8868  0.0 10.6 8816880 212128 ?      S    Oct19   0:01 unicorn worker[0] -E production -c config/unicorn.conf.rb
discour+  8867  0.0 10.2 8763632 204668 ?      Ss   Oct19   0:00 unicorn worker[0] -E production -c config/unicorn.conf.rb
discour+  8866  0.0 10.2 8755436 204660 ?      S    Oct19   0:00 unicorn worker[0] -E production -c config/unicorn.conf.rb
discour+  7081  0.0  3.7 8805768 73728 ?       S    Oct19   0:00 unicorn worker[1] -E production -c config/unicorn.conf.rb
postgres   693  0.0  3.3 214172 65752 ?        Ss   Sep28   1:48 postgres: 12/main: checkpointer   
discour+  7080  0.0  2.0 8805768 41004 ?       Ss   Oct19   0:00 unicorn worker[1] -E production -c config/unicorn.conf.rb
discour+  7079  0.0  2.0 8805768 40896 ?       S    Oct19   0:00 unicorn worker[1] -E production -c config/unicorn.conf.rb
postgres  7679  0.3  1.6 221464 33004 ?        Ss   11:07   0:01 postgres: 12/main: discourse discourse [local] idle
postgres  8070  0.0  0.9 215828 17960 ?        Ss   11:11   0:00 postgres: 12/main: discourse discourse [local] idle
postgres  8124  0.0  0.8 215828 16272 ?        Ss   11:12   0:00 postgres: 12/main: discourse discourse [local] idle
postgres  8122  0.0  0.8 215828 16224 ?        Ss   11:12   0:00 postgres: 12/main: discourse discourse [local] idle
postgres  8114  0.1  0.8 215828 16200 ?        Ss   11:12   0:00 postgres: 12/main: discourse discourse [local] idle
postgres   694  0.0  0.3 214060  6768 ?        Ss   Sep28   1:25 postgres: 12/main: background writer   
postgres   672  0.0  0.3 213928  6464 ?        S    Sep28   5:52 /usr/lib/postgresql/12/bin/postmaster -D /etc/postgresql/12/main
redis      669  0.3  0.2  52724  5292 ?        Sl   Sep28 184:39 /usr/bin/redis-server *:6379
www-data 18624  0.0  0.2  57792  4792 ?        S    Oct06   1:50 nginx: worker process
postgres   695  0.0  0.2 213928  4524 ?        Ss   Sep28  15:14 postgres: 12/main: walwriter   
discour+  8125  0.0  0.1  18644  3392 ?        R    11:12   0:00 ps aux --sort -rss
postgres   696  0.0  0.1 214600  2416 ?        Ss   Sep28   3:28 postgres: 12/main: autovacuum launcher   
postgres   698  0.0  0.1 214464  2240 ?        Ss   Sep28   0:11 postgres: 12/main: logical replication launcher   
discour+  8123  0.0  0.1  13708  2168 ?        S    11:12   0:00 sleep 1
discour+   673  0.0  0.0  15132  1444 ?        S    Sep28  34:54 /bin/bash config/unicorn_launcher -E production -c config/unicorn.conf.rb
postgres   697  0.0  0.0  69020  1416 ?        Ss   Sep28   7:57 postgres: 12/main: stats collector   
www-data 18625  0.0  0.0  56472   544 ?        S    Oct06   0:34 nginx: cache manager process
root       670  0.0  0.0  56148   400 ?        S    Sep28   0:00 nginx: master process /usr/sbin/nginx
root       674  0.0  0.0   8440   340 ?        S    Sep28   0:27 cron -f
root       671  0.0  0.0   2304   288 ?        S    Sep28   0:00 svlogd /var/log/postgres
root       675  0.0  0.0 156188   288 ?        Sl   Sep28   0:07 rsyslogd -n
root       664  0.0  0.0   2160   160 ?        Ss   Sep28   0:00 runsv unicorn
root       666  0.0  0.0   2160   152 ?        Ss   Sep28   0:00 runsv nginx
root       668  0.0  0.0   2304    72 ?        S    Sep28   0:06 svlogd /var/log/redis
root       661  0.0  0.0   2312    24 pts/0    S+   Sep28   1:16 /usr/bin/runsvdir -P /etc/service
root       663  0.0  0.0   2160    24 ?        Ss   Sep28   0:00 runsv cron
root       665  0.0  0.0   2160    24 ?        Ss   Sep28   0:00 runsv postgres
root       667  0.0  0.0   2160    24 ?        Ss   Sep28   0:00 runsv redis
root       662  0.0  0.0   2160    20 ?        Ss   Sep28   0:00 runsv rsyslog
root         1  0.0  0.0   6664     4 pts/0    Ss+  Sep28   0:00 /bin/bash /sbin/boot

Ed_S · 11월 2, 2020, 1:23오후

이제 discourse도 업데이트를 마쳤는데, 10월에 생성된 3개 워커 2세트, docker 업데이트 시점에 생성된 3개 워커, 그리고 지금 시점에 생성된 워커 2개와 마스터 유니콘 1개가 보입니다.

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
discour+ 18038  9.4 13.0 8742172 260704 ?      Sl   13:17   0:06 unicorn worker[1] -E production -c config/unicorn.conf.rb
discour+ 18024  9.1 13.0 8748324 260412 ?      Sl   13:17   0:06 unicorn worker[0] -E production -c config/unicorn.conf.rb
discour+ 17821 16.5 11.4 327956 227912 ?       Sl   13:17   0:14 unicorn master -E production -c config/unicorn.conf.rb
discour+ 17992  1.1 11.2 384336 224148 ?       SNl  13:17   0:00 sidekiq 6.1.2 discourse [0 of 5 busy]
discour+  8868  0.0  6.3 8816880 127064 ?      S    Oct19   0:01 unicorn worker[0] -E production -c config/unicorn.conf.rb
discour+  8866  0.0  4.5 8755436 91520 ?       S    Oct19   0:00 unicorn worker[0] -E production -c config/unicorn.conf.rb
discour+  8867  0.0  4.5 8763632 91520 ?       Ss   Oct19   0:00 unicorn worker[0] -E production -c config/unicorn.conf.rb
discour+  6608  0.0  3.9 8805772 78544 ?       S    11:04   0:00 unicorn worker[0] -E production -c config/unicorn.conf.rb
discour+  6609  0.0  3.9 8797576 78512 ?       Ss   11:04   0:00 unicorn worker[0] -E production -c config/unicorn.conf.rb
discour+  6611  0.0  3.2 8830348 64732 ?       S    11:04   0:00 unicorn worker[0] -E production -c config/unicorn.conf.rb
postgres 17932  1.6  1.6 221444 32344 ?        Ss   13:17   0:01 postgres: 12/main: discourse discourse [local] idle
postgres 18168  0.3  0.9 216036 19512 ?        Ss   13:18   0:00 postgres: 12/main: discourse discourse [local] idle
postgres   693  0.0  0.9 214172 19008 ?        Ss   Sep28   1:48 postgres: 12/main: checkpointer   
postgres 18068  0.0  0.9 215828 18580 ?        Ss   13:18   0:00 postgres: 12/main: discourse discourse [local] idle
postgres 18129  0.0  0.8 215828 16652 ?        Ss   13:18   0:00 postgres: 12/main: discourse discourse [local] idle
postgres   672  0.0  0.3 213928  6364 ?        S    Sep28   5:53 /usr/lib/postgresql/12/bin/postmaster -D /etc/postgresql/12/main
redis      669  0.3  0.2  52724  4704 ?        Sl   Sep28 185:06 /usr/bin/redis-server *:6379
www-data 18624  0.0  0.2  57792  4360 ?        S    Oct06   1:51 nginx: worker process
postgres   694  0.0  0.1 214060  3512 ?        Ss   Sep28   1:25 postgres: 12/main: background writer   
discour+ 18175  0.0  0.1  18644  3448 ?        R    13:18   0:00 ps aux --sort -rss
postgres   695  0.0  0.1 213928  3292 ?        Ss   Sep28  15:17 postgres: 12/main: walwriter   
postgres   698  0.0  0.1 214464  2120 ?        Ss   Sep28   0:11 postgres: 12/main: logical replication launcher   
discour+ 18174  0.0  0.1  13708  2092 ?        S    13:18   0:00 sleep 1
discour+   673  0.0  0.1  15132  2052 ?        S    Sep28  34:59 /bin/bash config/unicorn_launcher -E production -c config/unicorn.conf.rb
postgres   696  0.0  0.0 214600  1340 ?        Ss   Sep28   3:28 postgres: 12/main: autovacuum launcher   
postgres   697  0.0  0.0  69020   684 ?        Ss   Sep28   7:58 postgres: 12/main: stats collector   
www-data 18625  0.0  0.0  56472   428 ?        S    Oct06   0:35 nginx: cache manager process
root       670  0.0  0.0  56148   348 ?        S    Sep28   0:00 nginx: master process /usr/sbin/nginx
root       675  0.0  0.0 156188   224 ?        Sl   Sep28   0:07 rsyslogd -n
root       674  0.0  0.0   8440   108 ?        S    Sep28   0:27 cron -f
root       671  0.0  0.0   2304    32 ?        S    Sep28   0:00 svlogd /var/log/postgres
root       661  0.0  0.0   2312    24 pts/0    S+   Sep28   1:16 /usr/bin/runsvdir -P /etc/service
root       663  0.0  0.0   2160    24 ?        Ss   Sep28   0:00 runsv cron
root       664  0.0  0.0   2160    24 ?        Ss   Sep28   0:00 runsv unicorn
root       665  0.0  0.0   2160    24 ?        Ss   Sep28   0:00 runsv postgres
root       666  0.0  0.0   2160    24 ?        Ss   Sep28   0:00 runsv nginx
root       667  0.0  0.0   2160    24 ?        Ss   Sep28   0:00 runsv redis
root       668  0.0  0.0   2304    24 ?        S    Sep28   0:06 svlogd /var/log/redis
root       662  0.0  0.0   2160    20 ?        Ss   Sep28   0:00 runsv rsyslog
root         1  0.0  0.0   6664     4 pts/0    Ss+  Sep28   0:00 /bin/bash /sbin/boot
discour+  7079  0.0  0.0 8805768    4 ?        S    Oct19   0:00 unicorn worker[1] -E production -c config/unicorn.conf.rb
discour+  7080  0.0  0.0 8805768    4 ?        Ss   Oct19   0:00 unicorn worker[1] -E production -c config/unicorn.conf.rb
discour+  7081  0.0  0.0 8805768    4 ?        S    Oct19   0:00 unicorn worker[1] -E production -c config/unicorn.conf.rb

sam · 11월 3, 2020, 12:42오전

docker manager가 최신 버전인지 확인해 주시겠어요?

@Osama 생성된 업그레이드 프로세스에 대해 몇 가지 안전장치를 추가해야 할 것 같습니다… 잘 모르겠네요.

Ed_S · 11월 3, 2020, 9:09오전

네, 모든 것이 최신 상태였습니다. 10월 19일에 업데이트를 한 뒤, 11월 2일에 다시 업데이트했습니다. 보통 알림을 받은 후 3~4일 이내에 공지된 모든 업데이트를 적용하고 있습니다.

Osama · 11월 3, 2020, 7:05오후

@Ed_S 지난주(10월 29일) 이 문제를 해결할 패시를 적용했습니다. 문제는 웹 UI를 통해 업그레이드를 수행할 때마다 unicorn 워커가 3번 포크(fork)되고, 이 포크된 프로세스들이 종료되지 않는 것이었습니다. 이 버그와 여기에 보고된 문제는 관련이 있으며, 이제 수정되었어야 합니다.

귀하의 인스턴스에는 이제 수정 사항이 적용되었으므로, 관리자 UI를 통해 업그레이드한 후에도 추가적인 unicorn 워커가 표시되지 않아야 합니다. 기존에 남아 있는 워커를 제거하려면 kill -9 명령을 사용하거나 컨테이너를 재시작하면 충분합니다. 향후 업그레이드 시에도 이 문제가 지속되면 알려 주시기 바랍니다.

글		댓글	조회수
Browser upgrade fails when discourse is configured to use one unicorn worker Bug	5	535	10월 18, 2023
Upgrade abort - not enough Unicorn workers running Self-hosting	4	201	1월 8, 2026
"Waiting for unicorn to reload", then update stops Product Support	30	5099	6월 20, 2021
"Start upgrading" button after successfully upgraded Bug	13	1441	10월 29, 2020
Problem after update to 1.8.0.beta10 (today) Self-hosting	9	2867	4월 11, 2017

유니콘 워커가 너무 많습니다 - 업그레이드를 위해 정지하지 못했나요?

Related topics