"Waiting for unicorn to reload", then update stops

support
1

I’m getting these messages when updating:

image
image.png1782×1196 230 KB

and then the update stops with the error message in the header.

Any ideas? :slight_smile:

1 Like
2

Last time when this happened, I needed to

cd /var/discourse
./launcher rebuild app

https://meta.discourse.org/t/error-when-updating-discourse-to-2-2-0-beta3/99422/2

7 Likes
3

Thanks! Will give that a try :slight_smile:

4

Rebuilding worked, but I still get the same error afterwards when updating using the web UI.

5

I’ve had this but did not need to rebuild. It had apparently updated though this was not reflected on the web UI progress. Waiting a while and then opening the upgrade screen again confirmed that despite the odd visual behaviour, Discourse had updated.

1 Like
6

This is odd, do you see anything if you do ./launcher logs app?

1 Like
7

This is the last I see in the logs:

[Wed Nov  7 21:34:09 UTC 2018] Run reload cmd: sv reload nginx
warning: nginx: unable to open supervise/ok: file does not exist
[Wed Nov  7 21:34:09 UTC 2018] Reload error for :
Started runsvdir, PID is 326
ok: run: redis: (pid 337) 0s
ok: run: postgres: (pid 342) 0s
rsyslogd: command 'KLogPermitNonKernelFacility' is currently not permitted - did you already set it via a RainerScript command (v6+ config)? [v8.16.0 try http://www.rsyslog.com/e/2222 ]
rsyslogd: imklog: cannot open kernel log (/proc/kmsg): Operation not permitted.
rsyslogd: activation of module imklog failed [v8.16.0 try http://www.rsyslog.com/e/2145 ]
rsyslogd: Could not open output pipe '/dev/xconsole':: No such file or directory [v8.16.0 try http://www.rsyslog.com/e/2039 ]
supervisor pid: 338 unicorn pid: 378
(338) Reloading unicorn (378)
(338) Waiting for new unicorn master pid... 18109
(338) Waiting for new unicorn workers under 18109 to start up...
(338) Waiting for new unicorn workers under 18109 to start up...
(338) Waiting for new unicorn workers under 18109 to start up...
(338) Waiting for new unicorn workers under 18109 to start up...
(338) Waiting for new unicorn workers under 18109 to start up...
(338) Waiting for new unicorn workers under 18109 to start up...
(338) Waiting for new unicorn workers under 18109 to start up...
(338) Waiting for new unicorn workers under 18109 to start up...
(338) Waiting for new unicorn workers under 18109 to start up...
(338) Waiting for new unicorn workers under 18109 to start up...
(338) Waiting for new unicorn workers under 18109 to start up...
(338) Old pid is: 378 New pid is: 18109
8

Ah I found the problem, when the script encounters an error it logs to STDERR which goes to nowhere right now. I fixed it so that we publish the errors to the client too.

https://github.com/discourse/docker_manager/commit/fe454cb3a908e515ea5bfb59f9766bc0bc1088b5

7 Likes
9

Thanks @tgxworld!

After the latest rebuild I now get this error when updating via the UI:

********************************************************
*** Please be patient, next steps might take a while ***
********************************************************
Cycling Unicorn, to free up memory
Restarting unicorn pid: 333
Waiting for Unicorn to reload.
Waiting for Unicorn to reload..
Waiting for Unicorn to reload...
Waiting for Unicorn to reload....
Waiting for Unicorn to reload.....
Waiting for Unicorn to reload......
Waiting for Unicorn to reload.......
Waiting for Unicorn to reload........
Waiting for Unicorn to reload.........
Waiting for Unicorn to reload..........
Waiting for Unicorn to reload...........
Waiting for Unicorn to reload............
Waiting for Unicorn to reload.............
Waiting for Unicorn to reload..............
Waiting for Unicorn to reload...............
Stopping 15 Unicorn worker(s), to free up memory
Stopping job queue to reclaim memory, master pid is 24565
Docker Manager: FAILED TO UPGRADE
#<Errno::ESRCH: No such process>
/var/www/discourse/lib/demon/base.rb:71:in `kill'
/var/www/discourse/lib/demon/base.rb:71:in `stop'
/var/www/discourse/lib/demon/base.rb:20:in `block in stop'
/var/www/discourse/lib/demon/base.rb:19:in `each'
/var/www/discourse/lib/demon/base.rb:19:in `stop'
config/unicorn.conf.rb:136:in `block (2 levels) in reload'
/var/www/discourse/plugins/docker_manager/lib/docker_manager/upgrader.rb:59:in `sleep'
/var/www/discourse/plugins/docker_manager/lib/docker_manager/upgrader.rb:59:in `upgrade'
/var/www/discourse/plugins/docker_manager/app/controllers/docker_manager/admin_controller.rb:91:in `block in upgrade'
/var/www/discourse/plugins/docker_manager/app/controllers/docker_manager/admin_controller.rb:87:in `fork'
/var/www/discourse/plugins/docker_manager/app/controllers/docker_manager/admin_controller.rb:87:in `upgrade'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/abstract_controller/base.rb:194:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_controller/metal/rendering.rb:30:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.0/lib/active_support/callbacks.rb:132:in `run_callbacks'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/abstract_controller/callbacks.rb:41:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_controller/metal/rescue.rb:22:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.0/lib/active_support/notifications.rb:168:in `block in instrument'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.0/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.0/lib/active_support/notifications.rb:168:in `instrument'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_controller/metal/params_wrapper.rb:256:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/railties/controller_runtime.rb:24:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/abstract_controller/base.rb:134:in `process'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionview-5.2.0/lib/action_view/rendering.rb:32:in `process'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rack-mini-profiler-1.0.0/lib/mini_profiler/profiling_methods.rb:78:in `block in profile_method'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_controller/metal.rb:191:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_controller/metal.rb:252:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_dispatch/routing/route_set.rb:52:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_dispatch/routing/route_set.rb:34:in `serve'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_dispatch/routing/mapper.rb:18:in `block in <class:Constraints>'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_dispatch/routing/mapper.rb:48:in `serve'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_dispatch/journey/router.rb:52:in `block in serve'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_dispatch/journey/router.rb:35:in `each'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_dispatch/journey/router.rb:35:in `serve'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_dispatch/routing/route_set.rb:840:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/railties-5.2.0/lib/rails/engine.rb:524:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/railties-5.2.0/lib/rails/railtie.rb:190:in `public_send'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/railties-5.2.0/lib/rails/railtie.rb:190:in `method_missing'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_dispatch/routing/mapper.rb:19:in `block in <class:Constraints>'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_dispatch/routing/mapper.rb:48:in `serve'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_dispatch/journey/router.rb:52:in `block in serve'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_dispatch/journey/router.rb:35:in `each'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_dispatch/journey/router.rb:35:in `serve'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_dispatch/routing/route_set.rb:840:in `call'
/var/www/discourse/plugins/new_relic-discourse/gems/2.5.2/gems/newrelic_rpm-5.4.0.347/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rack-protection-2.0.3/lib/rack/protection/frame_options.rb:31:in `call'
/var/www/discourse/plugins/new_relic-discourse/gems/2.5.2/gems/newrelic_rpm-5.4.0.347/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
/var/www/discourse/lib/middleware/omniauth_bypass_middleware.rb:24:in `call'
/var/www/discourse/plugins/new_relic-discourse/gems/2.5.2/gems/newrelic_rpm-5.4.0.347/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
/var/www/discourse/plugins/new_relic-discourse/gems/2.5.2/gems/newrelic_rpm-5.4.0.347/lib/new_relic/rack/agent_hooks.rb:30:in `traced_call'
/var/www/discourse/plugins/new_relic-discourse/gems/2.5.2/gems/newrelic_rpm-5.4.0.347/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
/var/www/discourse/plugins/new_relic-discourse/gems/2.5.2/gems/newrelic_rpm-5.4.0.347/lib/new_relic/rack/browser_monitoring.rb:32:in `traced_call'
/var/www/discourse/plugins/new_relic-discourse/gems/2.5.2/gems/newrelic_rpm-5.4.0.347/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rack-2.0.6/lib/rack/tempfile_reaper.rb:15:in `call'
/var/www/discourse/plugins/new_relic-discourse/gems/2.5.2/gems/newrelic_rpm-5.4.0.347/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rack-2.0.6/lib/rack/conditional_get.rb:38:in `call'
/var/www/discourse/plugins/new_relic-discourse/gems/2.5.2/gems/newrelic_rpm-5.4.0.347/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rack-2.0.6/lib/rack/head.rb:12:in `call'
/var/www/discourse/plugins/new_relic-discourse/gems/2.5.2/gems/newrelic_rpm-5.4.0.347/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
/var/www/discourse/lib/content_security_policy.rb:14:in `call'
/var/www/discourse/plugins/new_relic-discourse/gems/2.5.2/gems/newrelic_rpm-5.4.0.347/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
/var/www/discourse/lib/middleware/anonymous_cache.rb:216:in `call'
/var/www/discourse/plugins/new_relic-discourse/gems/2.5.2/gems/newrelic_rpm-5.4.0.347/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rack-2.0.6/lib/rack/session/abstract/id.rb:232:in `context'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rack-2.0.6/lib/rack/session/abstract/id.rb:226:in `call'
/var/www/discourse/plugins/new_relic-discourse/gems/2.5.2/gems/newrelic_rpm-5.4.0.347/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_dispatch/middleware/cookies.rb:670:in `call'
/var/www/discourse/plugins/new_relic-discourse/gems/2.5.2/gems/newrelic_rpm-5.4.0.347/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.0/lib/active_support/callbacks.rb:98:in `run_callbacks'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
/var/www/discourse/plugins/new_relic-discourse/gems/2.5.2/gems/newrelic_rpm-5.4.0.347/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call'
/var/www/discourse/plugins/new_relic-discourse/gems/2.5.2/gems/newrelic_rpm-5.4.0.347/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
/var/www/discourse/plugins/new_relic-discourse/gems/2.5.2/gems/newrelic_rpm-5.4.0.347/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/logster-1.3.1/lib/logster/middleware/reporter.rb:31:in `call'
/var/www/discourse/plugins/new_relic-discourse/gems/2.5.2/gems/newrelic_rpm-5.4.0.347/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/railties-5.2.0/lib/rails/rack/logger.rb:38:in `call_app'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/railties-5.2.0/lib/rails/rack/logger.rb:28:in `call'
/var/www/discourse/config/initializers/100-quiet_logger.rb:16:in `call'
/var/www/discourse/config/initializers/100-silence_logger.rb:29:in `call'
/var/www/discourse/plugins/new_relic-discourse/gems/2.5.2/gems/newrelic_rpm-5.4.0.347/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
/var/www/discourse/plugins/new_relic-discourse/gems/2.5.2/gems/newrelic_rpm-5.4.0.347/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_dispatch/middleware/request_id.rb:27:in `call'
/var/www/discourse/plugins/new_relic-discourse/gems/2.5.2/gems/newrelic_rpm-5.4.0.347/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rack-2.0.6/lib/rack/method_override.rb:22:in `call'
/var/www/discourse/plugins/new_relic-discourse/gems/2.5.2/gems/newrelic_rpm-5.4.0.347/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_dispatch/middleware/executor.rb:14:in `call'
/var/www/discourse/plugins/new_relic-discourse/gems/2.5.2/gems/newrelic_rpm-5.4.0.347/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rack-2.0.6/lib/rack/sendfile.rb:111:in `call'
/var/www/discourse/plugins/new_relic-discourse/gems/2.5.2/gems/newrelic_rpm-5.4.0.347/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rack-mini-profiler-1.0.0/lib/mini_profiler/profiler.rb:174:in `call'
/var/www/discourse/plugins/new_relic-discourse/gems/2.5.2/gems/newrelic_rpm-5.4.0.347/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/message_bus-2.1.6/lib/message_bus/rack/middleware.rb:63:in `call'
/var/www/discourse/plugins/new_relic-discourse/gems/2.5.2/gems/newrelic_rpm-5.4.0.347/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
/var/www/discourse/lib/middleware/request_tracker.rb:180:in `call'
/var/www/discourse/plugins/new_relic-discourse/gems/2.5.2/gems/newrelic_rpm-5.4.0.347/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/railties-5.2.0/lib/rails/engine.rb:524:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/railties-5.2.0/lib/rails/railtie.rb:190:in `public_send'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/railties-5.2.0/lib/rails/railtie.rb:190:in `method_missing'
/var/www/discourse/plugins/new_relic-discourse/gems/2.5.2/gems/newrelic_rpm-5.4.0.347/lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rack-2.0.6/lib/rack/urlmap.rb:68:in `block in call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rack-2.0.6/lib/rack/urlmap.rb:53:in `each'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rack-2.0.6/lib/rack/urlmap.rb:53:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/unicorn-5.4.0/lib/unicorn/http_server.rb:606:in `process_client'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/unicorn-5.4.0/lib/unicorn/http_server.rb:701:in `worker_loop'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/unicorn-5.4.0/lib/unicorn/http_server.rb:549:in `spawn_missing_workers'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/unicorn-5.4.0/lib/unicorn/http_server.rb:563:in `maintain_worker_count'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/unicorn-5.4.0/lib/unicorn/http_server.rb:293:in `join'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/unicorn-5.4.0/bin/unicorn:126:in `<top (required)>'
/var/www/discourse/vendor/bundle/ruby/2.5.0/bin/unicorn:23:in `load'
/var/www/discourse/vendor/bundle/ruby/2.5.0/bin/unicorn:23:in `<main>'
3 Likes
10

I’m working in a solution for automated updating discourse. Currently I have this cronjob script:

@reboot sleep 300 && /opt/discourse/launcher rebuild app > /root/discord-update.log

Change your path to wherever your launcher file is. It seems to work for me, but I’ll try to improve it later later today.

With this script all you have to do is to reboot the server, and it will update discourse too

12

Anyone have any ideas of what the problem is and how to fix it? :slight_smile:

13

does this issue repeat after you do a console upgrade via ./launcher rebuild app

14

Yes, unfortunately :frowning:

15

Does that mean ./launcher rebuild app works and the problem only occurs when trying to upgrade via the UI?

Curious, do you have UNICORN_SIDEKIQS configured?

3 Likes
16

The problem only appears when using the UI to update. Rebuilding via the launcher works fine :slight_smile:

UNICORN_SIDEKIQS is not configured. Should it be?

17

@tgxworld I am also experiencing this error on my self hosted Discourse

Docker Manager: FAILED TO UPGRADE #<Errno::ESRCH: No such process>

Full trace

********************************************************
*** Please be patient, next steps might take a while ***
********************************************************
Cycling Unicorn, to free up memory
Restarting unicorn pid: 363
Waiting for Unicorn to reload.
Waiting for Unicorn to reload..
Waiting for Unicorn to reload...
Waiting for Unicorn to reload....
Waiting for Unicorn to reload.....
Waiting for Unicorn to reload......
Waiting for Unicorn to reload.......
Waiting for Unicorn to reload........
Waiting for Unicorn to reload.........
Stopping 3 Unicorn worker(s), to free up memory
Stopping job queue to reclaim memory, master pid is 28195
Docker Manager: FAILED TO UPGRADE
#<Errno::ESRCH: No such process>
/var/www/discourse/lib/demon/base.rb:71:in `kill'
/var/www/discourse/lib/demon/base.rb:71:in `stop'
/var/www/discourse/lib/demon/base.rb:20:in `block in stop'
/var/www/discourse/lib/demon/base.rb:19:in `each'
/var/www/discourse/lib/demon/base.rb:19:in `stop'
config/unicorn.conf.rb:136:in `block (2 levels) in reload'
/var/www/discourse/plugins/docker_manager/lib/docker_manager/upgrader.rb:59:in `sleep'
/var/www/discourse/plugins/docker_manager/lib/docker_manager/upgrader.rb:59:in `upgrade'
/var/www/discourse/plugins/docker_manager/app/controllers/docker_manager/admin_controller.rb:91:in `block in upgrade'
/var/www/discourse/plugins/docker_manager/app/controllers/docker_manager/admin_controller.rb:87:in `fork'
/var/www/discourse/plugins/docker_manager/app/controllers/docker_manager/admin_controller.rb:87:in `upgrade'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/abstract_controller/base.rb:194:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_controller/metal/rendering.rb:30:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.0/lib/active_support/callbacks.rb:132:in `run_callbacks'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/abstract_controller/callbacks.rb:41:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_controller/metal/rescue.rb:22:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.0/lib/active_support/notifications.rb:168:in `block in instrument'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.0/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.0/lib/active_support/notifications.rb:168:in `instrument'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_controller/metal/params_wrapper.rb:256:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/railties/controller_runtime.rb:24:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/abstract_controller/base.rb:134:in `process'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionview-5.2.0/lib/action_view/rendering.rb:32:in `process'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rack-mini-profiler-1.0.0/lib/mini_profiler/profiling_methods.rb:78:in `block in profile_method'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_controller/metal.rb:191:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_controller/metal.rb:252:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_dispatch/routing/route_set.rb:52:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_dispatch/routing/route_set.rb:34:in `serve'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_dispatch/routing/mapper.rb:18:in `block in <class:Constraints>'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_dispatch/routing/mapper.rb:48:in `serve'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_dispatch/journey/router.rb:52:in `block in serve'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_dispatch/journey/router.rb:35:in `each'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_dispatch/journey/router.rb:35:in `serve'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_dispatch/routing/route_set.rb:840:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/railties-5.2.0/lib/rails/engine.rb:524:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/railties-5.2.0/lib/rails/railtie.rb:190:in `public_send'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/railties-5.2.0/lib/rails/railtie.rb:190:in `method_missing'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_dispatch/routing/mapper.rb:19:in `block in <class:Constraints>'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_dispatch/routing/mapper.rb:48:in `serve'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_dispatch/journey/router.rb:52:in `block in serve'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_dispatch/journey/router.rb:35:in `each'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_dispatch/journey/router.rb:35:in `serve'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_dispatch/routing/route_set.rb:840:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rack-protection-2.0.3/lib/rack/protection/frame_options.rb:31:in `call'
/var/www/discourse/lib/middleware/omniauth_bypass_middleware.rb:24:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rack-2.0.6/lib/rack/tempfile_reaper.rb:15:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rack-2.0.6/lib/rack/conditional_get.rb:38:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rack-2.0.6/lib/rack/head.rb:12:in `call'
/var/www/discourse/lib/content_security_policy.rb:14:in `call'
/var/www/discourse/lib/middleware/anonymous_cache.rb:216:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rack-2.0.6/lib/rack/session/abstract/id.rb:232:in `context'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rack-2.0.6/lib/rack/session/abstract/id.rb:226:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_dispatch/middleware/cookies.rb:670:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.0/lib/active_support/callbacks.rb:98:in `run_callbacks'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/logster-1.3.1/lib/logster/middleware/reporter.rb:31:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/railties-5.2.0/lib/rails/rack/logger.rb:38:in `call_app'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/railties-5.2.0/lib/rails/rack/logger.rb:28:in `call'
/var/www/discourse/config/initializers/100-quiet_logger.rb:16:in `call'
/var/www/discourse/config/initializers/100-silence_logger.rb:29:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_dispatch/middleware/request_id.rb:27:in `call'
/var/www/discourse/lib/middleware/enforce_hostname.rb:17:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rack-2.0.6/lib/rack/method_override.rb:22:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.0/lib/action_dispatch/middleware/executor.rb:14:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rack-2.0.6/lib/rack/sendfile.rb:111:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rack-mini-profiler-1.0.0/lib/mini_profiler/profiler.rb:174:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/message_bus-2.1.6/lib/message_bus/rack/middleware.rb:63:in `call'
/var/www/discourse/lib/middleware/request_tracker.rb:180:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/railties-5.2.0/lib/rails/engine.rb:524:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/railties-5.2.0/lib/rails/railtie.rb:190:in `public_send'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/railties-5.2.0/lib/rails/railtie.rb:190:in `method_missing'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rack-2.0.6/lib/rack/urlmap.rb:68:in `block in call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rack-2.0.6/lib/rack/urlmap.rb:53:in `each'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rack-2.0.6/lib/rack/urlmap.rb:53:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/unicorn-5.4.0/lib/unicorn/http_server.rb:606:in `process_client'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/unicorn-5.4.0/lib/unicorn/http_server.rb:701:in `worker_loop'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/unicorn-5.4.0/lib/unicorn/http_server.rb:549:in `spawn_missing_workers'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/unicorn-5.4.0/lib/unicorn/http_server.rb:142:in `start'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/unicorn-5.4.0/bin/unicorn:126:in `<top (required)>'
/var/www/discourse/vendor/bundle/ruby/2.5.0/bin/unicorn:23:in `load'
/var/www/discourse/vendor/bundle/ruby/2.5.0/bin/unicorn:23:in `<main>'
3 Likes
18

Will have a look. This is a bug in our app where it might lose tracking of Sidekiq’s pid after a reload.

1 Like
19

I monitored the process tree while reloading Unicorn and noticed that the Sidekiq process was flapping around while the reloading is going on. That may lead to the master process not tracking the right Sidekiq pid and causing the error when we try to kill a pid which has already been killed. I fixed it in

https://github.com/discourse/discourse/commit/598ac69773101bf791ad9fee8b648c43f570a3f0

but will keep this topic open in case the problem happens again.

4 Likes
20

Unfortunately I still see the exact same issue as described above. CLI rebuild works fine, UI upgrade consistently fails. I am currently on commit 577af81 from December 17.

Stack trace 
********************************************************
*** Please be patient, next steps might take a while ***
********************************************************
Cycling Unicorn, to free up memory
Restarting unicorn pid: 43
Waiting for Unicorn to reload.
Waiting for Unicorn to reload..
Waiting for Unicorn to reload...
Waiting for Unicorn to reload....
Waiting for Unicorn to reload.....
Waiting for Unicorn to reload......
Waiting for Unicorn to reload.......
Waiting for Unicorn to reload........
Waiting for Unicorn to reload.........
Waiting for Unicorn to reload..........
Waiting for Unicorn to reload...........
Waiting for Unicorn to reload............
Waiting for Unicorn to reload.............
Waiting for Unicorn to reload..............
Waiting for Unicorn to reload...............
Waiting for Unicorn to reload................
Waiting for Unicorn to reload.................
Waiting for Unicorn to reload..................
Waiting for Unicorn to reload...................
Waiting for Unicorn to reload....................
Waiting for Unicorn to reload.....................
Waiting for Unicorn to reload......................
Waiting for Unicorn to reload.......................
Waiting for Unicorn to reload........................
Waiting for Unicorn to reload.........................
Waiting for Unicorn to reload..........................
Waiting for Unicorn to reload...........................
Waiting for Unicorn to reload............................
Waiting for Unicorn to reload.............................
Waiting for Unicorn to reload..............................
Waiting for Unicorn to reload...............................
Waiting for Unicorn to reload................................
Waiting for Unicorn to reload.................................
Waiting for Unicorn to reload..................................
Waiting for Unicorn to reload...................................
Waiting for Unicorn to reload....................................
Waiting for Unicorn to reload.....................................
Waiting for Unicorn to reload......................................
Waiting for Unicorn to reload.......................................
Waiting for Unicorn to reload........................................
Waiting for Unicorn to reload.........................................
Waiting for Unicorn to reload..........................................
Waiting for Unicorn to reload...........................................
Waiting for Unicorn to reload............................................
Waiting for Unicorn to reload.............................................
Waiting for Unicorn to reload..............................................
Waiting for Unicorn to reload...............................................
Waiting for Unicorn to reload................................................
Waiting for Unicorn to reload.................................................
Waiting for Unicorn to reload..................................................
Waiting for Unicorn to reload...................................................
Waiting for Unicorn to reload....................................................
Waiting for Unicorn to reload.....................................................
Waiting for Unicorn to reload......................................................
Waiting for Unicorn to reload.......................................................
Waiting for Unicorn to reload........................................................
Waiting for Unicorn to reload.........................................................
Waiting for Unicorn to reload..........................................................
Waiting for Unicorn to reload...........................................................
Stopping 1 Unicorn worker(s), to free up memory
Stopping job queue to reclaim memory, master pid is 28696
Docker Manager: FAILED TO UPGRADE
#<Errno::ESRCH: No such process>
/var/www/discourse/lib/demon/base.rb:71:in `kill'
/var/www/discourse/lib/demon/base.rb:71:in `stop'
/var/www/discourse/lib/demon/base.rb:20:in `block in stop'
/var/www/discourse/lib/demon/base.rb:19:in `each'
/var/www/discourse/lib/demon/base.rb:19:in `stop'
config/unicorn.conf.rb:136:in `block (2 levels) in reload'
/var/www/discourse/plugins/docker_manager/lib/docker_manager/upgrader.rb:60:in `kill'
/var/www/discourse/plugins/docker_manager/lib/docker_manager/upgrader.rb:60:in `upgrade'
/var/www/discourse/plugins/docker_manager/app/controllers/docker_manager/admin_controller.rb:91:in `block in upgrade'
/var/www/discourse/plugins/docker_manager/app/controllers/docker_manager/admin_controller.rb:87:in `fork'
/var/www/discourse/plugins/docker_manager/app/controllers/docker_manager/admin_controller.rb:87:in `upgrade'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.2/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.2/lib/abstract_controller/base.rb:194:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.2/lib/action_controller/metal/rendering.rb:30:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.2/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.2/lib/active_support/callbacks.rb:132:in `run_callbacks'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.2/lib/abstract_controller/callbacks.rb:41:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.2/lib/action_controller/metal/rescue.rb:22:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.2/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.2/lib/active_support/notifications.rb:168:in `block in instrument'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.2/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.2/lib/active_support/notifications.rb:168:in `instrument'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.2/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.2/lib/action_controller/metal/params_wrapper.rb:256:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.2/lib/active_record/railties/controller_runtime.rb:24:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.2/lib/abstract_controller/base.rb:134:in `process'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionview-5.2.2/lib/action_view/rendering.rb:32:in `process'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rack-mini-profiler-1.0.1/lib/mini_profiler/profiling_methods.rb:78:in `block in profile_method'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.2/lib/action_controller/metal.rb:191:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.2/lib/action_controller/metal.rb:252:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.2/lib/action_dispatch/routing/route_set.rb:52:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.2/lib/action_dispatch/routing/route_set.rb:34:in `serve'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.2/lib/action_dispatch/routing/mapper.rb:18:in `block in <class:Constraints>'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.2/lib/action_dispatch/routing/mapper.rb:48:in `serve'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.2/lib/action_dispatch/journey/router.rb:52:in `block in serve'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.2/lib/action_dispatch/journey/router.rb:35:in `each'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.2/lib/action_dispatch/journey/router.rb:35:in `serve'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.2/lib/action_dispatch/routing/route_set.rb:840:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/railties-5.2.2/lib/rails/engine.rb:524:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/railties-5.2.2/lib/rails/railtie.rb:190:in `public_send'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/railties-5.2.2/lib/rails/railtie.rb:190:in `method_missing'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.2/lib/action_dispatch/routing/mapper.rb:19:in `block in <class:Constraints>'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.2/lib/action_dispatch/routing/mapper.rb:48:in `serve'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.2/lib/action_dispatch/journey/router.rb:52:in `block in serve'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.2/lib/action_dispatch/journey/router.rb:35:in `each'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.2/lib/action_dispatch/journey/router.rb:35:in `serve'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.2/lib/action_dispatch/routing/route_set.rb:840:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rack-protection-2.0.3/lib/rack/protection/frame_options.rb:31:in `call'
/var/www/discourse/lib/middleware/omniauth_bypass_middleware.rb:32:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rack-2.0.6/lib/rack/tempfile_reaper.rb:15:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rack-2.0.6/lib/rack/conditional_get.rb:38:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rack-2.0.6/lib/rack/head.rb:12:in `call'
/var/www/discourse/lib/content_security_policy/middleware.rb:12:in `call'
/var/www/discourse/lib/middleware/anonymous_cache.rb:216:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rack-2.0.6/lib/rack/session/abstract/id.rb:232:in `context'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rack-2.0.6/lib/rack/session/abstract/id.rb:226:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.2/lib/action_dispatch/middleware/cookies.rb:670:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.2/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.2/lib/active_support/callbacks.rb:98:in `run_callbacks'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.2/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.2/lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.2/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/logster-1.3.1/lib/logster/middleware/reporter.rb:31:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/railties-5.2.2/lib/rails/rack/logger.rb:38:in `call_app'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/railties-5.2.2/lib/rails/rack/logger.rb:28:in `call'
/var/www/discourse/config/initializers/100-quiet_logger.rb:16:in `call'
/var/www/discourse/config/initializers/100-silence_logger.rb:29:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.2/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.2/lib/action_dispatch/middleware/request_id.rb:27:in `call'
/var/www/discourse/lib/middleware/enforce_hostname.rb:17:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rack-2.0.6/lib/rack/method_override.rb:22:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.2/lib/action_dispatch/middleware/executor.rb:14:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rack-2.0.6/lib/rack/sendfile.rb:111:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rack-mini-profiler-1.0.1/lib/mini_profiler/profiler.rb:171:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/message_bus-2.2.0.pre.1/lib/message_bus/rack/middleware.rb:57:in `call'
/var/www/discourse/lib/middleware/request_tracker.rb:180:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/railties-5.2.2/lib/rails/engine.rb:524:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/railties-5.2.2/lib/rails/railtie.rb:190:in `public_send'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/railties-5.2.2/lib/rails/railtie.rb:190:in `method_missing'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rack-2.0.6/lib/rack/urlmap.rb:68:in `block in call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rack-2.0.6/lib/rack/urlmap.rb:53:in `each'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rack-2.0.6/lib/rack/urlmap.rb:53:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/unicorn-5.4.0/lib/unicorn/http_server.rb:606:in `process_client'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/unicorn-5.4.0/lib/unicorn/http_server.rb:701:in `worker_loop'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/unicorn-5.4.0/lib/unicorn/http_server.rb:549:in `spawn_missing_workers'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/unicorn-5.4.0/lib/unicorn/http_server.rb:142:in `start'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/unicorn-5.4.0/bin/unicorn:126:in `<top (required)>'
/var/www/discourse/vendor/bundle/ruby/2.5.0/bin/unicorn:23:in `load'
/var/www/discourse/vendor/bundle/ruby/2.5.0/bin/unicorn:23:in `<main>'
Spinning up 1 Unicorn worker(s) that were stopped initially
21

I’ve been battling the problem @danekhollas details above since December. I usually attempt to upgrade multiple times after each failure and usually on the third or fourth attempt the upgrade will finally succeed. Unfortunately this equates to quite a lot of downtime as the site is completely down for ~5 minutes after each failure. I have also rebuilt the container a few times since, the last time being a few weeks ago and that has not solved the problem.

The situation was worse again this evening, as the site never restarted and I had no choice but to give up and reboot, which got discourse back online. Presently, on /admin/upgrade, the docker_manager claims it is “Currently Upgrading…” but otherwise everything is back to normal.

This is a self-hosted, standard docker “by the book” setup as per current install docs.

2 Likes