Error for logged in users and Git hang

Support
1

Without me changing anything, my Discourse forum https://lilypond.community suddenly started returning an error page for any request with login cookies:

Oops

The software powering this discussion forum encountered an unexpected problem. We apologize for the inconvenience.

Detailed information about the error was logged, and an automatic notification generated. We’ll take a look at it.

No further action is necessary. However, if the error condition persists, you can provide additional detail, including steps to reproduce the error, by posting a discussion topic in the site’s feedback category.

The site appears to work fine when you’re not logged in.

I looked into /shared/log/rails/production.log (after ./launcher enter app) and it contains errors like this:

Pitchfork worker is about to timeout, dumping backtrace for main thread
/var/www/discourse/lib/git_utils.rb:33:in 'Kernel#`'
/var/www/discourse/lib/git_utils.rb:33:in 'GitUtils.try_git'
/var/www/discourse/lib/git_utils.rb:21:in 'GitUtils.has_commit?'
/var/www/discourse/lib/discourse_updates.rb:167:in 'block in DiscourseUpdates.new_features'
/var/www/discourse/lib/discourse_updates.rb:163:in 'Array#select!'
/var/www/discourse/lib/discourse_updates.rb:163:in 'DiscourseUpdates.new_features'
/var/www/discourse/lib/discourse_updates.rb:269:in 'DiscourseUpdates.latest_new_feature_created_at'
/var/www/discourse/lib/discourse_updates.rb:254:in 'DiscourseUpdates.has_unseen_features?'
/var/www/discourse/app/serializers/current_user_serializer.rb:193:in 'CurrentUserSerializer#has_unseen_features'
(eval at /var/www/discourse/vendor/bundle/ruby/3.4.0/gems/active_model_serializers-0.8.4/lib/active_model/serializer.rb:467):80:in 'CurrentUserSerializer#_fast_attributes'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/active_model_serializers-0.8.4/lib/active_model/serializer.rb:456:in 'ActiveModel::Serializer#attributes'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/active_model_serializers-0.8.4/lib/active_model/serializer.rb:480:in 'ActiveModel::Serializer#_serializable_hash'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/active_model_serializers-0.8.4/lib/active_model/serializer.rb:359:in 'ActiveModel::Serializer#serializable_hash'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/active_model_serializers-0.8.4/lib/active_model/serializer.rb:347:in 'ActiveModel::Serializer#as_json'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/activesupport-8.0.5/lib/active_support/json/encoding.rb:41:in 'Oj::Rails::Encoder#encode'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/activesupport-8.0.5/lib/active_support/json/encoding.rb:41:in 'ActiveSupport::JSON.encode'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/activesupport-8.0.5/lib/active_support/core_ext/object/json.rb:42:in 'ActiveSupport::ToJsonWithActiveSupportEncoder#to_json'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/active_model_serializers-0.8.4/lib/active_model/serializer.rb:331:in 'ActiveModel::Serializer#to_json'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/multi_json-1.20.1/lib/multi_json/adapters/oj.rb:67:in 'Oj.dump'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/multi_json-1.20.1/lib/multi_json/adapters/oj.rb:67:in 'MultiJson::Adapters::Oj#dump'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/multi_json-1.20.1/lib/multi_json/adapter.rb:92:in 'MultiJson::Adapter.dump'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/multi_json-1.20.1/lib/multi_json.rb:214:in 'MultiJson.dump'
/var/www/discourse/lib/application_layout_preloader.rb:76:in 'ApplicationLayoutPreloader#preload_current_user_data'
/var/www/discourse/lib/application_layout_preloader.rb:32:in 'ApplicationLayoutPreloader#preloaded_data'
/var/www/discourse/app/helpers/application_helper.rb:1001:in 'ApplicationHelper#preloaded_json'
/var/www/discourse/app/views/layouts/application.html.erb:101:in '_app_views_layouts_application_html_erb___243682310287886530_152408'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionview-8.0.5/lib/action_view/base.rb:278:in 'Kernel#public_send'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionview-8.0.5/lib/action_view/base.rb:278:in 'ActionView::Base#_run'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionview-8.0.5/lib/action_view/template.rb:284:in 'block in ActionView::Template#render'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/activesupport-8.0.5/lib/active_support/notifications.rb:212:in 'ActiveSupport::Notifications.instrument'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionview-8.0.5/lib/action_view/template.rb:589:in 'ActionView::Template#instrument_render_template'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionview-8.0.5/lib/action_view/template.rb:272:in 'ActionView::Template#render'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/rack-mini-profiler-4.0.1/lib/mini_profiler/profiling_methods.rb:116:in 'block in ActionView::Template#profile_method'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionview-8.0.5/lib/action_view/renderer/template_renderer.rb:77:in 'block in ActionView::TemplateRenderer#render_with_layout'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/activesupport-8.0.5/lib/active_support/notifications.rb:210:in 'block in ActiveSupport::Notifications.instrument'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/activesupport-8.0.5/lib/active_support/notifications/instrumenter.rb:58:in 'ActiveSupport::Notifications::Instrumenter#instrument'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/activesupport-8.0.5/lib/active_support/notifications.rb:210:in 'ActiveSupport::Notifications.instrument'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionview-8.0.5/lib/action_view/renderer/template_renderer.rb:75:in 'ActionView::TemplateRenderer#render_with_layout'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionview-8.0.5/lib/action_view/renderer/template_renderer.rb:59:in 'ActionView::TemplateRenderer#render_template'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionview-8.0.5/lib/action_view/renderer/template_renderer.rb:11:in 'ActionView::TemplateRenderer#render'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionview-8.0.5/lib/action_view/renderer/renderer.rb:58:in 'ActionView::Renderer#render_template_to_object'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionview-8.0.5/lib/action_view/renderer/renderer.rb:31:in 'ActionView::Renderer#render_to_object'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionview-8.0.5/lib/action_view/rendering.rb:136:in 'block in ActionView::Rendering#_render_template'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionview-8.0.5/lib/action_view/base.rb:305:in 'ActionView::Base#in_rendering_context'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionview-8.0.5/lib/action_view/rendering.rb:135:in 'ActionView::Rendering#_render_template'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_controller/metal/streaming.rb:179:in 'ActionController::Streaming#_render_template'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionview-8.0.5/lib/action_view/rendering.rb:122:in 'ActionView::Rendering#render_to_body'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_controller/metal/rendering.rb:186:in 'ActionController::Rendering#render_to_body'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_controller/metal/renderers.rb:140:in 'ActionController::Renderers#render_to_body'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/abstract_controller/rendering.rb:28:in 'AbstractController::Rendering#render'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_controller/metal/rendering.rb:167:in 'ActionController::Rendering#render'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_controller/metal/instrumentation.rb:31:in 'block (2 levels) in ActionController::Instrumentation#render'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/activesupport-8.0.5/lib/active_support/benchmark.rb:17:in 'ActiveSupport::Benchmark.realtime'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_controller/metal/instrumentation.rb:31:in 'block in ActionController::Instrumentation#render'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_controller/metal/instrumentation.rb:100:in 'ActionController::Instrumentation#cleanup_view_runtime'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.5/lib/active_record/railties/controller_runtime.rb:46:in 'ActiveRecord::Railties::ControllerRuntime#cleanup_view_runtime'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_controller/metal/instrumentation.rb:30:in 'ActionController::Instrumentation#render'
/var/www/discourse/lib/topic_list_responder.rb:16:in 'block (2 levels) in TopicListResponder#respond_with_list'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_controller/metal/mime_responds.rb:224:in 'ActionController::MimeResponds#respond_to'
/var/www/discourse/lib/topic_list_responder.rb:7:in 'TopicListResponder#respond_with_list'
/var/www/discourse/app/controllers/list_controller.rb:117:in 'block (2 levels) in <class:ListController>'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_controller/metal/basic_implicit_render.rb:8:in 'ActionController::BasicImplicitRender#send_action'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/abstract_controller/base.rb:215:in 'AbstractController::Base#process_action'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_controller/metal/rendering.rb:193:in 'ActionController::Rendering#process_action'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/abstract_controller/callbacks.rb:261:in 'block in AbstractController::Callbacks#process_action'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/activesupport-8.0.5/lib/active_support/callbacks.rb:120:in 'block in ActiveSupport::Callbacks#run_callbacks'
/var/www/discourse/app/controllers/application_controller.rb:443:in 'block in ApplicationController#with_resolved_locale'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/i18n-1.14.8/lib/i18n.rb:354:in 'I18n::Base#with_locale'
/var/www/discourse/app/controllers/application_controller.rb:443:in 'ApplicationController#with_resolved_locale'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/activesupport-8.0.5/lib/active_support/callbacks.rb:129:in 'block in ActiveSupport::Callbacks#run_callbacks'
/var/www/discourse/app/controllers/application_controller.rb:1125:in 'ApplicationController#ensure_dont_cache_page'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/activesupport-8.0.5/lib/active_support/callbacks.rb:129:in 'block in ActiveSupport::Callbacks#run_callbacks'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/activesupport-8.0.5/lib/active_support/callbacks.rb:140:in 'ActiveSupport::Callbacks#run_callbacks'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/abstract_controller/callbacks.rb:260:in 'AbstractController::Callbacks#process_action'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_controller/metal/rescue.rb:27:in 'ActionController::Rescue#process_action'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_controller/metal/instrumentation.rb:76:in 'block in ActionController::Instrumentation#process_action'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/activesupport-8.0.5/lib/active_support/notifications.rb:210:in 'block in ActiveSupport::Notifications.instrument'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/activesupport-8.0.5/lib/active_support/notifications/instrumenter.rb:58:in 'ActiveSupport::Notifications::Instrumenter#instrument'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/activesupport-8.0.5/lib/active_support/notifications.rb:210:in 'ActiveSupport::Notifications.instrument'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_controller/metal/instrumentation.rb:75:in 'ActionController::Instrumentation#process_action'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_controller/metal/params_wrapper.rb:259:in 'ActionController::ParamsWrapper#process_action'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.5/lib/active_record/railties/controller_runtime.rb:39:in 'ActiveRecord::Railties::ControllerRuntime#process_action'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/abstract_controller/base.rb:152:in 'AbstractController::Base#process'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionview-8.0.5/lib/action_view/rendering.rb:40:in 'ActionView::Rendering#process'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/rack-mini-profiler-4.0.1/lib/mini_profiler/profiling_methods.rb:116:in 'block in ActionController::Base#profile_method'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_controller/metal.rb:252:in 'ActionController::Metal#dispatch'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_controller/metal.rb:335:in 'ActionController::Metal.dispatch'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_dispatch/routing/route_set.rb:67:in 'ActionDispatch::Routing::RouteSet::Dispatcher#dispatch'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_dispatch/routing/route_set.rb:50:in 'ActionDispatch::Routing::RouteSet::Dispatcher#serve'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_dispatch/routing/mapper.rb:32:in 'block in <class:Constraints>'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_dispatch/routing/mapper.rb:62:in 'ActionDispatch::Routing::Mapper::Constraints#serve'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_dispatch/journey/router.rb:53:in 'block in ActionDispatch::Journey::Router#serve'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_dispatch/journey/router.rb:133:in 'block in ActionDispatch::Journey::Router#find_routes'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_dispatch/journey/router.rb:126:in 'Array#each'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_dispatch/journey/router.rb:126:in 'ActionDispatch::Journey::Router#find_routes'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_dispatch/journey/router.rb:34:in 'ActionDispatch::Journey::Router#serve'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_dispatch/routing/route_set.rb:908:in 'ActionDispatch::Routing::RouteSet#call'
/var/www/discourse/lib/middleware/omniauth_bypass_middleware.rb:35:in 'Middleware::OmniauthBypassMiddleware#call'
/var/www/discourse/lib/middleware/crawler_hooks.rb:13:in 'Middleware::CrawlerHooks#call'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/rack-2.2.23/lib/rack/tempfile_reaper.rb:15:in 'Rack::TempfileReaper#call'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/rack-2.2.23/lib/rack/conditional_get.rb:27:in 'Rack::ConditionalGet#call'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/rack-2.2.23/lib/rack/head.rb:12:in 'Rack::Head#call'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_dispatch/http/permissions_policy.rb:38:in 'ActionDispatch::PermissionsPolicy::Middleware#call'
/var/www/discourse/lib/content_security_policy/middleware.rb:12:in 'ContentSecurityPolicy::Middleware#call'
/var/www/discourse/lib/middleware/anonymous_cache.rb:425:in 'Middleware::AnonymousCache#call'
/var/www/discourse/lib/middleware/csp_script_nonce_injector.rb:13:in 'Middleware::CspScriptNonceInjector#call'
/var/www/discourse/lib/middleware/track_view_session_id_injector.rb:12:in 'Middleware::TrackViewSessionIdInjector#call'
/var/www/discourse/config/initializers/008-rack-cors.rb:14:in 'Discourse::Cors#call'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/rack-2.2.23/lib/rack/session/abstract/id.rb:266:in 'Rack::Session::Abstract::Persisted#context'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/rack-2.2.23/lib/rack/session/abstract/id.rb:260:in 'Rack::Session::Abstract::Persisted#call'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_dispatch/middleware/cookies.rb:706:in 'ActionDispatch::Cookies#call'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_dispatch/middleware/callbacks.rb:31:in 'block in ActionDispatch::Callbacks#call'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/activesupport-8.0.5/lib/active_support/callbacks.rb:100:in 'ActiveSupport::Callbacks#run_callbacks'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_dispatch/middleware/callbacks.rb:30:in 'ActionDispatch::Callbacks#call'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_dispatch/middleware/debug_exceptions.rb:31:in 'ActionDispatch::DebugExceptions#call'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_dispatch/middleware/show_exceptions.rb:32:in 'ActionDispatch::ShowExceptions#call'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/logster-2.21.0/lib/logster/middleware/reporter.rb:40:in 'Logster::Middleware::Reporter#call'
/var/www/discourse/lib/middleware/default_headers.rb:13:in 'Middleware::DefaultHeaders#call'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/railties-8.0.5/lib/rails/rack/logger.rb:41:in 'Rails::Rack::Logger#call_app'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/railties-8.0.5/lib/rails/rack/logger.rb:29:in 'Rails::Rack::Logger#call'
/var/www/discourse/config/initializers/100-quiet_logger.rb:20:in 'DiscourseRackQuietAssetsLogger#call'
/var/www/discourse/config/initializers/100-silence_logger.rb:29:in 'SilenceLogger#call'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_dispatch/middleware/request_id.rb:34:in 'ActionDispatch::RequestId#call'
/var/www/discourse/lib/middleware/enforce_hostname.rb:23:in 'Middleware::EnforceHostname#call'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/rack-2.2.23/lib/rack/method_override.rb:24:in 'Rack::MethodOverride#call'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_dispatch/middleware/executor.rb:16:in 'ActionDispatch::Executor#call'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/rack-2.2.23/lib/rack/sendfile.rb:127:in 'Rack::Sendfile#call'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/rack-mini-profiler-4.0.1/lib/mini_profiler.rb:334:in 'Rack::MiniProfiler#call'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/message_bus-4.5.2/lib/message_bus/rack/middleware.rb:60:in 'MessageBus::Rack::Middleware#call'
/var/www/discourse/lib/middleware/request_tracker.rb:380:in 'Middleware::RequestTracker#call'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_dispatch/middleware/remote_ip.rb:96:in 'ActionDispatch::RemoteIp#call'
/var/www/discourse/lib/middleware/overload_protections.rb:18:in 'Middleware::OverloadProtections#call'
/var/www/discourse/lib/middleware/processing_request.rb:14:in 'Middleware::ProcessingRequest#call'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/railties-8.0.5/lib/rails/engine.rb:535:in 'Rails::Engine#call'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/railties-8.0.5/lib/rails/railtie.rb:226:in 'Kernel#public_send'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/railties-8.0.5/lib/rails/railtie.rb:226:in 'Rails::Railtie.method_missing'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/rack-2.2.23/lib/rack/urlmap.rb:74:in 'block in Rack::URLMap#call'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/rack-2.2.23/lib/rack/urlmap.rb:58:in 'Array#each'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/rack-2.2.23/lib/rack/urlmap.rb:58:in 'Rack::URLMap#call'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/rack-2.2.23/lib/rack/tempfile_reaper.rb:15:in 'Rack::TempfileReaper#call'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/pitchfork-0.18.2/lib/pitchfork/chunked.rb:105:in 'Pitchfork::Chunked#call'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/rack-2.2.23/lib/rack/content_length.rb:17:in 'Rack::ContentLength#call'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/pitchfork-0.18.2/lib/pitchfork/http_server.rb:868:in 'Pitchfork::HttpServer#process_client'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/pitchfork-0.18.2/lib/pitchfork/http_server.rb:1017:in 'Pitchfork::HttpServer#worker_loop'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/pitchfork-0.18.2/lib/pitchfork/http_server.rb:616:in 'block in Pitchfork::HttpServer#spawn_worker'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/pitchfork-0.18.2/lib/pitchfork/http_server.rb:1222:in 'block in Pitchfork::HttpServer#fork_sibling'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/pitchfork-0.18.2/lib/pitchfork.rb:170:in 'block in Pitchfork.clean_fork'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/pitchfork-0.18.2/lib/pitchfork.rb:168:in 'Kernel#catch'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/pitchfork-0.18.2/lib/pitchfork.rb:168:in 'Pitchfork.clean_fork'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/pitchfork-0.18.2/lib/pitchfork/http_server.rb:690:in 'Pitchfork::HttpServer#spawn_initial_mold'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/pitchfork-0.18.2/lib/pitchfork/http_server.rb:176:in 'Pitchfork::HttpServer#start'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/pitchfork-0.18.2/exe/pitchfork:110:in '<top (required)>'
/var/www/discourse/vendor/bundle/ruby/3.4.0/bin/pitchfork:25:in 'Kernel#load'
/var/www/discourse/vendor/bundle/ruby/3.4.0/bin/pitchfork:25:in '<main>'

  Rendered layout layouts/application.html.erb (Duration: 29736.7ms | GC: 60.9ms)
Completed 500 Internal Server Error in 29972ms (ActiveRecord: 0.0ms (0 queries, 0 cached) | GC: 98.5ms)

Also, top -c sometimes shows git merge-base processes taking 100% CPU and substantial RAM.

What might be going on?

2

Same error than this:

3

Oh, actually, if I log in with an unprivileged user account, it works, but not with my admin account. So I suspect it’s the upgrade part of the admin interface that’s hanging in these Git operations?

4

Seen here too. Admins and mods are blocked while things time out, I think.

Process tree:

1000     2169778  0.0  0.3 497160 13392 ?        Sl   Aug17   0:15          |       \_ pitchfork monitor - 
1000     2169783  0.0  5.4 6554324 212356 ?      Sl   Aug17   4:55          |       |   \_ pitchfork (gen:0) mold - ready
1000     2170466  0.0  2.3 6567316 91468 ?       Sl   Aug17   0:52          |       |   \_ pitchfork (gen:0) service - ready
1000     2170475  1.3  9.9 6958868 386872 ?      SNl  Aug17 310:37          |       |   |   \_ sidekiq 7.3.10 discourse [0 of 5 busy]
1000     2170499  2.6 11.3 6650900 444392 ?      Sl   Aug17 597:55          |       |   \_ pitchfork (gen:0) worker[0] - requests: 1767782, processing: /
1000     3914556  0.0  0.0      0     0 ?        Z    17:20   0:00          |       |   |   \_ [sh] <defunct>
1000     3925648  0.0  0.0      0     0 ?        Z    19:39   0:00          |       |   |   \_ [sh] <defunct>
1000     3925710  0.0  0.0  10552  2312 ?        S    19:40   0:00          |       |   |   \_ sh -c git merge-base --is-ancestor e20c10c056d2ed11533c3428e33e93cb31fa22d8 HEAD 2> /dev/null; echo $?
1000     3925711 92.4 15.1 710404 590864 ?       R    19:40   0:24          |       |   |       \_ git merge-base --is-ancestor e20c10c056d2ed11533c3428e33e93cb31fa22d8 HEAD
1000     2170558  0.1 10.3 6573012 402832 ?      Sl   Aug17  35:21          |       |   \_ pitchfork (gen:0) worker[1] - requests: 75111, processing: /
1000     3925725  0.0  0.0  10552  2316 ?        S    19:40   0:00          |       |   |   \_ sh -c git merge-base --is-ancestor e20c10c056d2ed11533c3428e33e93cb31fa22d8 HEAD 2> /dev/null; echo $?
1000     3925726 90.1 14.4 710404 562628 ?       R    19:40   0:15          |       |   |       \_ git merge-base --is-ancestor e20c10c056d2ed11533c3428e33e93cb31fa22d8 HEAD
1000     2170585  0.0  8.3 6578772 327676 ?      Rl   Aug17  10:28          |       |   \_ pitchfork (gen:0) worker[2] - requests: 8139, processing: /t/two-upcoming-books-about-videogame-history/4406
1000     2170608  0.0  7.8 6589908 305532 ?      Sl   Aug17   8:08          |       |   \_ pitchfork (gen:0) worker[3] - requests: 4055, waiting
1000     3925781  0.0  0.0  15992  2684 ?        S    19:40   0:00          |       \_ sleep 1

First seen about 2 hours ago I think.

5

So
DiscourseUpdates.has_unseen_features
seems to be a feature which is bringing in a realtime dependency on github working correctly? Even when we’re not trying to upgrade?

6

I’ve unchecked this feature:

admin->site admin → version checks → clear the checkbox → save

image
image1368×584 40.7 KB

Site admin
Configure various settings relating to the management of your site
version_checks
Ping the Discourse Hub for version updates and show new version messages on the admin dashboard

7

How did you manage get the admin interface to open without error in the first place?

8

I had to wait until the git nonsense timed out, and/or could get my request served to a pitchfork which wasn’t blocked.

My forum/logs area is presently showing 4 cases of “pitchfork worker timed out” and it’s not a very busy forum at all.

9

One could also use rails from an ssh access, I believe.
This might be a terrible idea. Proceed at your own risk.

# docker exec -i app rails runner 'puts SiteSetting.version_checks'
true
# docker exec -i app rails runner 'SiteSetting.set("version_checks", false)'
# docker exec -i app rails runner 'puts SiteSetting.version_checks'
false
10

It seems that Github continue with issues, that were the previous ones: