Too many unicorn workers - failed to stop for upgrade?

Installation
1

Just upgraded Docker and I see the old unicorns were still running during the upgrade, and now I seem to have two sets of unicorns. So I wonder if something which is meant to stop the old ones has not run, or not worked.

During the upgrade (Oct 19 would be the date of the previous update):

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
discour+  6992 93.5 18.9 13543864 376908 ?     Rl   11:05   0:55 /var/www/discourse/vendor/bundle/ruby/2.6.0/bin/rake assets:precompile
discour+  6611  0.1 13.4 8799684 268616 ?      Sl   11:04   0:00 unicorn worker[0] -E production -c config/unicorn.conf.rb
discour+  6608  0.0 13.2 8805772 264008 ?      S    11:04   0:00 unicorn worker[0] -E production -c config/unicorn.conf.rb
discour+  6609  0.0 13.2 8797576 263936 ?      Ss   11:04   0:00 unicorn worker[0] -E production -c config/unicorn.conf.rb
discour+  6834  5.6 11.9 8733980 238900 ?      Sl   11:04   0:06 unicorn worker[0] -E production -c config/unicorn.conf.rb
discour+  8868  0.0 11.6 8816880 231564 ?      S    Oct19   0:01 unicorn worker[0] -E production -c config/unicorn.conf.rb
discour+  8867  0.0 11.4 8763632 227900 ?      Ss   Oct19   0:00 unicorn worker[0] -E production -c config/unicorn.conf.rb
discour+  8866  0.0 11.4 8755436 227892 ?      S    Oct19   0:00 unicorn worker[0] -E production -c config/unicorn.conf.rb
discour+  6661  8.0  9.9 315668 198900 ?       Sl   11:04   0:10 unicorn master -E production -c config/unicorn.conf.rb
discour+  7081  0.0  4.2 8805768 84856 ?       S    Oct19   0:00 unicorn worker[1] -E production -c config/unicorn.conf.rb
postgres   693  0.0  3.3 214172 67016 ?        Ss   Sep28   1:48 postgres: 12/main: checkpointer   
discour+  7080  0.0  2.0 8805768 41456 ?       Ss   Oct19   0:00 unicorn worker[1] -E production -c config/unicorn.conf.rb
discour+  7079  0.0  2.0 8805768 41348 ?       S    Oct19   0:00 unicorn worker[1] -E production -c config/unicorn.conf.rb
postgres  6750  1.0  1.6 221452 33028 ?        Ss   11:04   0:01 postgres: 12/main: discourse discourse [local] idle
postgres  6427  0.0  1.0 217244 20424 ?        Ss   11:04   0:00 postgres: 12/main: discourse discourse [local] idle
postgres  6937  0.0  0.9 215828 18676 ?        Ss   11:05   0:00 postgres: 12/main: discourse discourse [local] idle
postgres  7086  1.0  0.8 215828 16332 ?        Ss   11:06   0:00 postgres: 12/main: discourse discourse [local] idle
postgres   694  0.0  0.3 214060  6404 ?        Ss   Sep28   1:25 postgres: 12/main: background writer   
postgres   672  0.0  0.3 213928  6092 ?        S    Sep28   5:52 /usr/lib/postgresql/12/bin/postmaster -D /etc/postgresql/12/main
www-data 18624  0.0  0.2  57792  5956 ?        S    Oct06   1:50 nginx: worker process
redis      669  0.3  0.2  52724  5196 ?        Sl   Sep28 184:37 /usr/bin/redis-server *:6379
postgres   695  0.0  0.2 213928  4524 ?        Ss   Sep28  15:14 postgres: 12/main: walwriter   
discour+  7088  0.0  0.1  18644  3400 ?        R    11:06   0:00 ps aux --sort -rss
discour+  7087  0.0  0.1  13708  2156 ?        S    11:06   0:00 sleep 1
discour+   673  0.0  0.0  15132  1696 ?        S    Sep28  34:54 /bin/bash config/unicorn_launcher -E production -c config/unicorn.conf.rb
postgres   696  0.0  0.0 214600  1368 ?        Ss   Sep28   3:28 postgres: 12/main: autovacuum launcher   
postgres   698  0.0  0.0 214464   980 ?        Ss   Sep28   0:11 postgres: 12/main: logical replication launcher   
discour+  6991  0.0  0.0   2388   712 ?        S    11:05   0:00 sh -c cd /var/www/discourse && RUBY_GC_MALLOC_LIMIT_MAX=20971520 RUBY_GC_OLDMALLOC_LIMIT_MAX=20971520 RUBY_GC_HEAP_GROWTH_MAX_SLOTS=50000 RUBY_GC_HEAP_OLDOBJECT_LIMIT_FACTOR=0.9  bundle exec rake assets:precompile 2>&1
postgres   697  0.0  0.0  69020   684 ?        Ss   Sep28   7:57 postgres: 12/main: stats collector   
root       674  0.0  0.0   8440   572 ?        S    Sep28   0:27 cron -f
root       671  0.0  0.0   2304   440 ?        S    Sep28   0:00 svlogd /var/log/postgres
www-data 18625  0.0  0.0  56472   420 ?        S    Oct06   0:34 nginx: cache manager process
root       670  0.0  0.0  56148   400 ?        S    Sep28   0:00 nginx: master process /usr/sbin/nginx
root       675  0.0  0.0 156188   288 ?        Sl   Sep28   0:07 rsyslogd -n
root       664  0.0  0.0   2160   240 ?        Ss   Sep28   0:00 runsv unicorn
root       666  0.0  0.0   2160   224 ?        Ss   Sep28   0:00 runsv nginx
root       668  0.0  0.0   2304   216 ?        S    Sep28   0:06 svlogd /var/log/redis
root       661  0.0  0.0   2312    24 pts/0    S+   Sep28   1:16 /usr/bin/runsvdir -P /etc/service
root       663  0.0  0.0   2160    24 ?        Ss   Sep28   0:00 runsv cron
root       665  0.0  0.0   2160    24 ?        Ss   Sep28   0:00 runsv postgres
root       667  0.0  0.0   2160    24 ?        Ss   Sep28   0:00 runsv redis
root       662  0.0  0.0   2160    20 ?        Ss   Sep28   0:00 runsv rsyslog
root         1  0.0  0.0   6664     4 pts/0    Ss+  Sep28   0:00 /bin/bash /sbin/boot

After the upgrade:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
discour+  7778  2.6 13.1 8746268 261300 ?      Sl   11:08   0:06 unicorn worker[1] -E production -c config/unicorn.conf.rb
discour+  7764  2.5 13.0 8748324 259192 ?      Sl   11:08   0:06 unicorn worker[0] -E production -c config/unicorn.conf.rb
discour+  6608  0.0 12.9 8805772 258420 ?      S    11:04   0:00 unicorn worker[0] -E production -c config/unicorn.conf.rb
discour+  6609  0.0 12.9 8797576 258348 ?      Ss   11:04   0:00 unicorn worker[0] -E production -c config/unicorn.conf.rb
discour+  6611  0.1 12.5 8830348 250348 ?      S    11:04   0:00 unicorn worker[0] -E production -c config/unicorn.conf.rb
discour+  7733  0.9 11.4 425296 227348 ?       SNl  11:08   0:02 sidekiq 6.1.2 discourse [0 of 5 busy]
discour+  7569  4.9 11.0 327956 219724 ?       Sl   11:07   0:13 unicorn master -E production -c config/unicorn.conf.rb
discour+  8868  0.0 10.6 8816880 212128 ?      S    Oct19   0:01 unicorn worker[0] -E production -c config/unicorn.conf.rb
discour+  8867  0.0 10.2 8763632 204668 ?      Ss   Oct19   0:00 unicorn worker[0] -E production -c config/unicorn.conf.rb
discour+  8866  0.0 10.2 8755436 204660 ?      S    Oct19   0:00 unicorn worker[0] -E production -c config/unicorn.conf.rb
discour+  7081  0.0  3.7 8805768 73728 ?       S    Oct19   0:00 unicorn worker[1] -E production -c config/unicorn.conf.rb
postgres   693  0.0  3.3 214172 65752 ?        Ss   Sep28   1:48 postgres: 12/main: checkpointer   
discour+  7080  0.0  2.0 8805768 41004 ?       Ss   Oct19   0:00 unicorn worker[1] -E production -c config/unicorn.conf.rb
discour+  7079  0.0  2.0 8805768 40896 ?       S    Oct19   0:00 unicorn worker[1] -E production -c config/unicorn.conf.rb
postgres  7679  0.3  1.6 221464 33004 ?        Ss   11:07   0:01 postgres: 12/main: discourse discourse [local] idle
postgres  8070  0.0  0.9 215828 17960 ?        Ss   11:11   0:00 postgres: 12/main: discourse discourse [local] idle
postgres  8124  0.0  0.8 215828 16272 ?        Ss   11:12   0:00 postgres: 12/main: discourse discourse [local] idle
postgres  8122  0.0  0.8 215828 16224 ?        Ss   11:12   0:00 postgres: 12/main: discourse discourse [local] idle
postgres  8114  0.1  0.8 215828 16200 ?        Ss   11:12   0:00 postgres: 12/main: discourse discourse [local] idle
postgres   694  0.0  0.3 214060  6768 ?        Ss   Sep28   1:25 postgres: 12/main: background writer   
postgres   672  0.0  0.3 213928  6464 ?        S    Sep28   5:52 /usr/lib/postgresql/12/bin/postmaster -D /etc/postgresql/12/main
redis      669  0.3  0.2  52724  5292 ?        Sl   Sep28 184:39 /usr/bin/redis-server *:6379
www-data 18624  0.0  0.2  57792  4792 ?        S    Oct06   1:50 nginx: worker process
postgres   695  0.0  0.2 213928  4524 ?        Ss   Sep28  15:14 postgres: 12/main: walwriter   
discour+  8125  0.0  0.1  18644  3392 ?        R    11:12   0:00 ps aux --sort -rss
postgres   696  0.0  0.1 214600  2416 ?        Ss   Sep28   3:28 postgres: 12/main: autovacuum launcher   
postgres   698  0.0  0.1 214464  2240 ?        Ss   Sep28   0:11 postgres: 12/main: logical replication launcher   
discour+  8123  0.0  0.1  13708  2168 ?        S    11:12   0:00 sleep 1
discour+   673  0.0  0.0  15132  1444 ?        S    Sep28  34:54 /bin/bash config/unicorn_launcher -E production -c config/unicorn.conf.rb
postgres   697  0.0  0.0  69020  1416 ?        Ss   Sep28   7:57 postgres: 12/main: stats collector   
www-data 18625  0.0  0.0  56472   544 ?        S    Oct06   0:34 nginx: cache manager process
root       670  0.0  0.0  56148   400 ?        S    Sep28   0:00 nginx: master process /usr/sbin/nginx
root       674  0.0  0.0   8440   340 ?        S    Sep28   0:27 cron -f
root       671  0.0  0.0   2304   288 ?        S    Sep28   0:00 svlogd /var/log/postgres
root       675  0.0  0.0 156188   288 ?        Sl   Sep28   0:07 rsyslogd -n
root       664  0.0  0.0   2160   160 ?        Ss   Sep28   0:00 runsv unicorn
root       666  0.0  0.0   2160   152 ?        Ss   Sep28   0:00 runsv nginx
root       668  0.0  0.0   2304    72 ?        S    Sep28   0:06 svlogd /var/log/redis
root       661  0.0  0.0   2312    24 pts/0    S+   Sep28   1:16 /usr/bin/runsvdir -P /etc/service
root       663  0.0  0.0   2160    24 ?        Ss   Sep28   0:00 runsv cron
root       665  0.0  0.0   2160    24 ?        Ss   Sep28   0:00 runsv postgres
root       667  0.0  0.0   2160    24 ?        Ss   Sep28   0:00 runsv redis
root       662  0.0  0.0   2160    20 ?        Ss   Sep28   0:00 runsv rsyslog
root         1  0.0  0.0   6664     4 pts/0    Ss+  Sep28   0:00 /bin/bash /sbin/boot
2 Likes
2

Having now also updated discourse, I have 2 sets of 3 workers from October, 3 workers dated from the docker update, and now two workers and one master unicorn dated about now.

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
discour+ 18038  9.4 13.0 8742172 260704 ?      Sl   13:17   0:06 unicorn worker[1] -E production -c config/unicorn.conf.rb
discour+ 18024  9.1 13.0 8748324 260412 ?      Sl   13:17   0:06 unicorn worker[0] -E production -c config/unicorn.conf.rb
discour+ 17821 16.5 11.4 327956 227912 ?       Sl   13:17   0:14 unicorn master -E production -c config/unicorn.conf.rb
discour+ 17992  1.1 11.2 384336 224148 ?       SNl  13:17   0:00 sidekiq 6.1.2 discourse [0 of 5 busy]
discour+  8868  0.0  6.3 8816880 127064 ?      S    Oct19   0:01 unicorn worker[0] -E production -c config/unicorn.conf.rb
discour+  8866  0.0  4.5 8755436 91520 ?       S    Oct19   0:00 unicorn worker[0] -E production -c config/unicorn.conf.rb
discour+  8867  0.0  4.5 8763632 91520 ?       Ss   Oct19   0:00 unicorn worker[0] -E production -c config/unicorn.conf.rb
discour+  6608  0.0  3.9 8805772 78544 ?       S    11:04   0:00 unicorn worker[0] -E production -c config/unicorn.conf.rb
discour+  6609  0.0  3.9 8797576 78512 ?       Ss   11:04   0:00 unicorn worker[0] -E production -c config/unicorn.conf.rb
discour+  6611  0.0  3.2 8830348 64732 ?       S    11:04   0:00 unicorn worker[0] -E production -c config/unicorn.conf.rb
postgres 17932  1.6  1.6 221444 32344 ?        Ss   13:17   0:01 postgres: 12/main: discourse discourse [local] idle
postgres 18168  0.3  0.9 216036 19512 ?        Ss   13:18   0:00 postgres: 12/main: discourse discourse [local] idle
postgres   693  0.0  0.9 214172 19008 ?        Ss   Sep28   1:48 postgres: 12/main: checkpointer   
postgres 18068  0.0  0.9 215828 18580 ?        Ss   13:18   0:00 postgres: 12/main: discourse discourse [local] idle
postgres 18129  0.0  0.8 215828 16652 ?        Ss   13:18   0:00 postgres: 12/main: discourse discourse [local] idle
postgres   672  0.0  0.3 213928  6364 ?        S    Sep28   5:53 /usr/lib/postgresql/12/bin/postmaster -D /etc/postgresql/12/main
redis      669  0.3  0.2  52724  4704 ?        Sl   Sep28 185:06 /usr/bin/redis-server *:6379
www-data 18624  0.0  0.2  57792  4360 ?        S    Oct06   1:51 nginx: worker process
postgres   694  0.0  0.1 214060  3512 ?        Ss   Sep28   1:25 postgres: 12/main: background writer   
discour+ 18175  0.0  0.1  18644  3448 ?        R    13:18   0:00 ps aux --sort -rss
postgres   695  0.0  0.1 213928  3292 ?        Ss   Sep28  15:17 postgres: 12/main: walwriter   
postgres   698  0.0  0.1 214464  2120 ?        Ss   Sep28   0:11 postgres: 12/main: logical replication launcher   
discour+ 18174  0.0  0.1  13708  2092 ?        S    13:18   0:00 sleep 1
discour+   673  0.0  0.1  15132  2052 ?        S    Sep28  34:59 /bin/bash config/unicorn_launcher -E production -c config/unicorn.conf.rb
postgres   696  0.0  0.0 214600  1340 ?        Ss   Sep28   3:28 postgres: 12/main: autovacuum launcher   
postgres   697  0.0  0.0  69020   684 ?        Ss   Sep28   7:58 postgres: 12/main: stats collector   
www-data 18625  0.0  0.0  56472   428 ?        S    Oct06   0:35 nginx: cache manager process
root       670  0.0  0.0  56148   348 ?        S    Sep28   0:00 nginx: master process /usr/sbin/nginx
root       675  0.0  0.0 156188   224 ?        Sl   Sep28   0:07 rsyslogd -n
root       674  0.0  0.0   8440   108 ?        S    Sep28   0:27 cron -f
root       671  0.0  0.0   2304    32 ?        S    Sep28   0:00 svlogd /var/log/postgres
root       661  0.0  0.0   2312    24 pts/0    S+   Sep28   1:16 /usr/bin/runsvdir -P /etc/service
root       663  0.0  0.0   2160    24 ?        Ss   Sep28   0:00 runsv cron
root       664  0.0  0.0   2160    24 ?        Ss   Sep28   0:00 runsv unicorn
root       665  0.0  0.0   2160    24 ?        Ss   Sep28   0:00 runsv postgres
root       666  0.0  0.0   2160    24 ?        Ss   Sep28   0:00 runsv nginx
root       667  0.0  0.0   2160    24 ?        Ss   Sep28   0:00 runsv redis
root       668  0.0  0.0   2304    24 ?        S    Sep28   0:06 svlogd /var/log/redis
root       662  0.0  0.0   2160    20 ?        Ss   Sep28   0:00 runsv rsyslog
root         1  0.0  0.0   6664     4 pts/0    Ss+  Sep28   0:00 /bin/bash /sbin/boot
discour+  7079  0.0  0.0 8805768    4 ?        S    Oct19   0:00 unicorn worker[1] -E production -c config/unicorn.conf.rb
discour+  7080  0.0  0.0 8805768    4 ?        Ss   Oct19   0:00 unicorn worker[1] -E production -c config/unicorn.conf.rb
discour+  7081  0.0  0.0 8805768    4 ?        S    Oct19   0:00 unicorn worker[1] -E production -c config/unicorn.conf.rb
1 Like
3

Can you confirm docker manager is on latest.

@Osama we may need to add some failsafes here with the spawned upgrade process … I don’t know.

2 Likes
4

Yes, everything was up to date. Updated on Oct 19th then again on Nov 2nd. I’ve taken every announced update, usually 3 or 4 days after I get the message.

1 Like
5

@Ed_S I pushed a patch last week (Oct 29th) that should fix this problem. The problem was that every time you do an upgrade via the web UI, a unicorn worker forked itself 3 times and these forked processes never died. This bug and the one reported here are connected and should be fixed now.

Your instance should have the fix now, so you shouldn’t see any extra unicorn workers after you upgrade via the admin UI. To get rid of the existing dangling workers, you can kill -9 them or a container restart should be enough. Please let me know if you still see this problem in future upgrades.

4 Likes
6

This topic was automatically closed after 6 days. New replies are no longer allowed.