Error for logged in users and Git hang

Support
1

Without me changing anything, my Discourse forum https://lilypond.community suddenly started returning an error page for any request with login cookies:

Oops

The software powering this discussion forum encountered an unexpected problem. We apologize for the inconvenience.

Detailed information about the error was logged, and an automatic notification generated. We’ll take a look at it.

No further action is necessary. However, if the error condition persists, you can provide additional detail, including steps to reproduce the error, by posting a discussion topic in the site’s feedback category.

The site appears to work fine when you’re not logged in.

I looked into /shared/log/rails/production.log (after ./launcher enter app) and it contains errors like this:

Pitchfork worker is about to timeout, dumping backtrace for main thread
/var/www/discourse/lib/git_utils.rb:33:in 'Kernel#`'
/var/www/discourse/lib/git_utils.rb:33:in 'GitUtils.try_git'
/var/www/discourse/lib/git_utils.rb:21:in 'GitUtils.has_commit?'
/var/www/discourse/lib/discourse_updates.rb:167:in 'block in DiscourseUpdates.new_features'
/var/www/discourse/lib/discourse_updates.rb:163:in 'Array#select!'
/var/www/discourse/lib/discourse_updates.rb:163:in 'DiscourseUpdates.new_features'
/var/www/discourse/lib/discourse_updates.rb:269:in 'DiscourseUpdates.latest_new_feature_created_at'
/var/www/discourse/lib/discourse_updates.rb:254:in 'DiscourseUpdates.has_unseen_features?'
/var/www/discourse/app/serializers/current_user_serializer.rb:193:in 'CurrentUserSerializer#has_unseen_features'
(eval at /var/www/discourse/vendor/bundle/ruby/3.4.0/gems/active_model_serializers-0.8.4/lib/active_model/serializer.rb:467):80:in 'CurrentUserSerializer#_fast_attributes'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/active_model_serializers-0.8.4/lib/active_model/serializer.rb:456:in 'ActiveModel::Serializer#attributes'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/active_model_serializers-0.8.4/lib/active_model/serializer.rb:480:in 'ActiveModel::Serializer#_serializable_hash'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/active_model_serializers-0.8.4/lib/active_model/serializer.rb:359:in 'ActiveModel::Serializer#serializable_hash'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/active_model_serializers-0.8.4/lib/active_model/serializer.rb:347:in 'ActiveModel::Serializer#as_json'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/activesupport-8.0.5/lib/active_support/json/encoding.rb:41:in 'Oj::Rails::Encoder#encode'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/activesupport-8.0.5/lib/active_support/json/encoding.rb:41:in 'ActiveSupport::JSON.encode'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/activesupport-8.0.5/lib/active_support/core_ext/object/json.rb:42:in 'ActiveSupport::ToJsonWithActiveSupportEncoder#to_json'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/active_model_serializers-0.8.4/lib/active_model/serializer.rb:331:in 'ActiveModel::Serializer#to_json'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/multi_json-1.20.1/lib/multi_json/adapters/oj.rb:67:in 'Oj.dump'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/multi_json-1.20.1/lib/multi_json/adapters/oj.rb:67:in 'MultiJson::Adapters::Oj#dump'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/multi_json-1.20.1/lib/multi_json/adapter.rb:92:in 'MultiJson::Adapter.dump'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/multi_json-1.20.1/lib/multi_json.rb:214:in 'MultiJson.dump'
/var/www/discourse/lib/application_layout_preloader.rb:76:in 'ApplicationLayoutPreloader#preload_current_user_data'
/var/www/discourse/lib/application_layout_preloader.rb:32:in 'ApplicationLayoutPreloader#preloaded_data'
/var/www/discourse/app/helpers/application_helper.rb:1001:in 'ApplicationHelper#preloaded_json'
/var/www/discourse/app/views/layouts/application.html.erb:101:in '_app_views_layouts_application_html_erb___243682310287886530_152408'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionview-8.0.5/lib/action_view/base.rb:278:in 'Kernel#public_send'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionview-8.0.5/lib/action_view/base.rb:278:in 'ActionView::Base#_run'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionview-8.0.5/lib/action_view/template.rb:284:in 'block in ActionView::Template#render'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/activesupport-8.0.5/lib/active_support/notifications.rb:212:in 'ActiveSupport::Notifications.instrument'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionview-8.0.5/lib/action_view/template.rb:589:in 'ActionView::Template#instrument_render_template'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionview-8.0.5/lib/action_view/template.rb:272:in 'ActionView::Template#render'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/rack-mini-profiler-4.0.1/lib/mini_profiler/profiling_methods.rb:116:in 'block in ActionView::Template#profile_method'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionview-8.0.5/lib/action_view/renderer/template_renderer.rb:77:in 'block in ActionView::TemplateRenderer#render_with_layout'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/activesupport-8.0.5/lib/active_support/notifications.rb:210:in 'block in ActiveSupport::Notifications.instrument'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/activesupport-8.0.5/lib/active_support/notifications/instrumenter.rb:58:in 'ActiveSupport::Notifications::Instrumenter#instrument'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/activesupport-8.0.5/lib/active_support/notifications.rb:210:in 'ActiveSupport::Notifications.instrument'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionview-8.0.5/lib/action_view/renderer/template_renderer.rb:75:in 'ActionView::TemplateRenderer#render_with_layout'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionview-8.0.5/lib/action_view/renderer/template_renderer.rb:59:in 'ActionView::TemplateRenderer#render_template'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionview-8.0.5/lib/action_view/renderer/template_renderer.rb:11:in 'ActionView::TemplateRenderer#render'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionview-8.0.5/lib/action_view/renderer/renderer.rb:58:in 'ActionView::Renderer#render_template_to_object'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionview-8.0.5/lib/action_view/renderer/renderer.rb:31:in 'ActionView::Renderer#render_to_object'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionview-8.0.5/lib/action_view/rendering.rb:136:in 'block in ActionView::Rendering#_render_template'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionview-8.0.5/lib/action_view/base.rb:305:in 'ActionView::Base#in_rendering_context'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionview-8.0.5/lib/action_view/rendering.rb:135:in 'ActionView::Rendering#_render_template'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_controller/metal/streaming.rb:179:in 'ActionController::Streaming#_render_template'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionview-8.0.5/lib/action_view/rendering.rb:122:in 'ActionView::Rendering#render_to_body'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_controller/metal/rendering.rb:186:in 'ActionController::Rendering#render_to_body'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_controller/metal/renderers.rb:140:in 'ActionController::Renderers#render_to_body'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/abstract_controller/rendering.rb:28:in 'AbstractController::Rendering#render'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_controller/metal/rendering.rb:167:in 'ActionController::Rendering#render'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_controller/metal/instrumentation.rb:31:in 'block (2 levels) in ActionController::Instrumentation#render'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/activesupport-8.0.5/lib/active_support/benchmark.rb:17:in 'ActiveSupport::Benchmark.realtime'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_controller/metal/instrumentation.rb:31:in 'block in ActionController::Instrumentation#render'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_controller/metal/instrumentation.rb:100:in 'ActionController::Instrumentation#cleanup_view_runtime'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.5/lib/active_record/railties/controller_runtime.rb:46:in 'ActiveRecord::Railties::ControllerRuntime#cleanup_view_runtime'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_controller/metal/instrumentation.rb:30:in 'ActionController::Instrumentation#render'
/var/www/discourse/lib/topic_list_responder.rb:16:in 'block (2 levels) in TopicListResponder#respond_with_list'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_controller/metal/mime_responds.rb:224:in 'ActionController::MimeResponds#respond_to'
/var/www/discourse/lib/topic_list_responder.rb:7:in 'TopicListResponder#respond_with_list'
/var/www/discourse/app/controllers/list_controller.rb:117:in 'block (2 levels) in <class:ListController>'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_controller/metal/basic_implicit_render.rb:8:in 'ActionController::BasicImplicitRender#send_action'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/abstract_controller/base.rb:215:in 'AbstractController::Base#process_action'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_controller/metal/rendering.rb:193:in 'ActionController::Rendering#process_action'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/abstract_controller/callbacks.rb:261:in 'block in AbstractController::Callbacks#process_action'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/activesupport-8.0.5/lib/active_support/callbacks.rb:120:in 'block in ActiveSupport::Callbacks#run_callbacks'
/var/www/discourse/app/controllers/application_controller.rb:443:in 'block in ApplicationController#with_resolved_locale'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/i18n-1.14.8/lib/i18n.rb:354:in 'I18n::Base#with_locale'
/var/www/discourse/app/controllers/application_controller.rb:443:in 'ApplicationController#with_resolved_locale'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/activesupport-8.0.5/lib/active_support/callbacks.rb:129:in 'block in ActiveSupport::Callbacks#run_callbacks'
/var/www/discourse/app/controllers/application_controller.rb:1125:in 'ApplicationController#ensure_dont_cache_page'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/activesupport-8.0.5/lib/active_support/callbacks.rb:129:in 'block in ActiveSupport::Callbacks#run_callbacks'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/activesupport-8.0.5/lib/active_support/callbacks.rb:140:in 'ActiveSupport::Callbacks#run_callbacks'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/abstract_controller/callbacks.rb:260:in 'AbstractController::Callbacks#process_action'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_controller/metal/rescue.rb:27:in 'ActionController::Rescue#process_action'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_controller/metal/instrumentation.rb:76:in 'block in ActionController::Instrumentation#process_action'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/activesupport-8.0.5/lib/active_support/notifications.rb:210:in 'block in ActiveSupport::Notifications.instrument'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/activesupport-8.0.5/lib/active_support/notifications/instrumenter.rb:58:in 'ActiveSupport::Notifications::Instrumenter#instrument'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/activesupport-8.0.5/lib/active_support/notifications.rb:210:in 'ActiveSupport::Notifications.instrument'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_controller/metal/instrumentation.rb:75:in 'ActionController::Instrumentation#process_action'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_controller/metal/params_wrapper.rb:259:in 'ActionController::ParamsWrapper#process_action'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/activerecord-8.0.5/lib/active_record/railties/controller_runtime.rb:39:in 'ActiveRecord::Railties::ControllerRuntime#process_action'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/abstract_controller/base.rb:152:in 'AbstractController::Base#process'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionview-8.0.5/lib/action_view/rendering.rb:40:in 'ActionView::Rendering#process'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/rack-mini-profiler-4.0.1/lib/mini_profiler/profiling_methods.rb:116:in 'block in ActionController::Base#profile_method'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_controller/metal.rb:252:in 'ActionController::Metal#dispatch'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_controller/metal.rb:335:in 'ActionController::Metal.dispatch'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_dispatch/routing/route_set.rb:67:in 'ActionDispatch::Routing::RouteSet::Dispatcher#dispatch'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_dispatch/routing/route_set.rb:50:in 'ActionDispatch::Routing::RouteSet::Dispatcher#serve'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_dispatch/routing/mapper.rb:32:in 'block in <class:Constraints>'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_dispatch/routing/mapper.rb:62:in 'ActionDispatch::Routing::Mapper::Constraints#serve'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_dispatch/journey/router.rb:53:in 'block in ActionDispatch::Journey::Router#serve'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_dispatch/journey/router.rb:133:in 'block in ActionDispatch::Journey::Router#find_routes'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_dispatch/journey/router.rb:126:in 'Array#each'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_dispatch/journey/router.rb:126:in 'ActionDispatch::Journey::Router#find_routes'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_dispatch/journey/router.rb:34:in 'ActionDispatch::Journey::Router#serve'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_dispatch/routing/route_set.rb:908:in 'ActionDispatch::Routing::RouteSet#call'
/var/www/discourse/lib/middleware/omniauth_bypass_middleware.rb:35:in 'Middleware::OmniauthBypassMiddleware#call'
/var/www/discourse/lib/middleware/crawler_hooks.rb:13:in 'Middleware::CrawlerHooks#call'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/rack-2.2.23/lib/rack/tempfile_reaper.rb:15:in 'Rack::TempfileReaper#call'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/rack-2.2.23/lib/rack/conditional_get.rb:27:in 'Rack::ConditionalGet#call'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/rack-2.2.23/lib/rack/head.rb:12:in 'Rack::Head#call'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_dispatch/http/permissions_policy.rb:38:in 'ActionDispatch::PermissionsPolicy::Middleware#call'
/var/www/discourse/lib/content_security_policy/middleware.rb:12:in 'ContentSecurityPolicy::Middleware#call'
/var/www/discourse/lib/middleware/anonymous_cache.rb:425:in 'Middleware::AnonymousCache#call'
/var/www/discourse/lib/middleware/csp_script_nonce_injector.rb:13:in 'Middleware::CspScriptNonceInjector#call'
/var/www/discourse/lib/middleware/track_view_session_id_injector.rb:12:in 'Middleware::TrackViewSessionIdInjector#call'
/var/www/discourse/config/initializers/008-rack-cors.rb:14:in 'Discourse::Cors#call'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/rack-2.2.23/lib/rack/session/abstract/id.rb:266:in 'Rack::Session::Abstract::Persisted#context'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/rack-2.2.23/lib/rack/session/abstract/id.rb:260:in 'Rack::Session::Abstract::Persisted#call'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_dispatch/middleware/cookies.rb:706:in 'ActionDispatch::Cookies#call'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_dispatch/middleware/callbacks.rb:31:in 'block in ActionDispatch::Callbacks#call'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/activesupport-8.0.5/lib/active_support/callbacks.rb:100:in 'ActiveSupport::Callbacks#run_callbacks'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_dispatch/middleware/callbacks.rb:30:in 'ActionDispatch::Callbacks#call'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_dispatch/middleware/debug_exceptions.rb:31:in 'ActionDispatch::DebugExceptions#call'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_dispatch/middleware/show_exceptions.rb:32:in 'ActionDispatch::ShowExceptions#call'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/logster-2.21.0/lib/logster/middleware/reporter.rb:40:in 'Logster::Middleware::Reporter#call'
/var/www/discourse/lib/middleware/default_headers.rb:13:in 'Middleware::DefaultHeaders#call'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/railties-8.0.5/lib/rails/rack/logger.rb:41:in 'Rails::Rack::Logger#call_app'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/railties-8.0.5/lib/rails/rack/logger.rb:29:in 'Rails::Rack::Logger#call'
/var/www/discourse/config/initializers/100-quiet_logger.rb:20:in 'DiscourseRackQuietAssetsLogger#call'
/var/www/discourse/config/initializers/100-silence_logger.rb:29:in 'SilenceLogger#call'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_dispatch/middleware/request_id.rb:34:in 'ActionDispatch::RequestId#call'
/var/www/discourse/lib/middleware/enforce_hostname.rb:23:in 'Middleware::EnforceHostname#call'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/rack-2.2.23/lib/rack/method_override.rb:24:in 'Rack::MethodOverride#call'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_dispatch/middleware/executor.rb:16:in 'ActionDispatch::Executor#call'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/rack-2.2.23/lib/rack/sendfile.rb:127:in 'Rack::Sendfile#call'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/rack-mini-profiler-4.0.1/lib/mini_profiler.rb:334:in 'Rack::MiniProfiler#call'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/message_bus-4.5.2/lib/message_bus/rack/middleware.rb:60:in 'MessageBus::Rack::Middleware#call'
/var/www/discourse/lib/middleware/request_tracker.rb:380:in 'Middleware::RequestTracker#call'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.5/lib/action_dispatch/middleware/remote_ip.rb:96:in 'ActionDispatch::RemoteIp#call'
/var/www/discourse/lib/middleware/overload_protections.rb:18:in 'Middleware::OverloadProtections#call'
/var/www/discourse/lib/middleware/processing_request.rb:14:in 'Middleware::ProcessingRequest#call'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/railties-8.0.5/lib/rails/engine.rb:535:in 'Rails::Engine#call'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/railties-8.0.5/lib/rails/railtie.rb:226:in 'Kernel#public_send'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/railties-8.0.5/lib/rails/railtie.rb:226:in 'Rails::Railtie.method_missing'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/rack-2.2.23/lib/rack/urlmap.rb:74:in 'block in Rack::URLMap#call'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/rack-2.2.23/lib/rack/urlmap.rb:58:in 'Array#each'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/rack-2.2.23/lib/rack/urlmap.rb:58:in 'Rack::URLMap#call'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/rack-2.2.23/lib/rack/tempfile_reaper.rb:15:in 'Rack::TempfileReaper#call'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/pitchfork-0.18.2/lib/pitchfork/chunked.rb:105:in 'Pitchfork::Chunked#call'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/rack-2.2.23/lib/rack/content_length.rb:17:in 'Rack::ContentLength#call'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/pitchfork-0.18.2/lib/pitchfork/http_server.rb:868:in 'Pitchfork::HttpServer#process_client'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/pitchfork-0.18.2/lib/pitchfork/http_server.rb:1017:in 'Pitchfork::HttpServer#worker_loop'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/pitchfork-0.18.2/lib/pitchfork/http_server.rb:616:in 'block in Pitchfork::HttpServer#spawn_worker'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/pitchfork-0.18.2/lib/pitchfork/http_server.rb:1222:in 'block in Pitchfork::HttpServer#fork_sibling'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/pitchfork-0.18.2/lib/pitchfork.rb:170:in 'block in Pitchfork.clean_fork'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/pitchfork-0.18.2/lib/pitchfork.rb:168:in 'Kernel#catch'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/pitchfork-0.18.2/lib/pitchfork.rb:168:in 'Pitchfork.clean_fork'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/pitchfork-0.18.2/lib/pitchfork/http_server.rb:690:in 'Pitchfork::HttpServer#spawn_initial_mold'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/pitchfork-0.18.2/lib/pitchfork/http_server.rb:176:in 'Pitchfork::HttpServer#start'
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/pitchfork-0.18.2/exe/pitchfork:110:in '<top (required)>'
/var/www/discourse/vendor/bundle/ruby/3.4.0/bin/pitchfork:25:in 'Kernel#load'
/var/www/discourse/vendor/bundle/ruby/3.4.0/bin/pitchfork:25:in '<main>'

  Rendered layout layouts/application.html.erb (Duration: 29736.7ms | GC: 60.9ms)
Completed 500 Internal Server Error in 29972ms (ActiveRecord: 0.0ms (0 queries, 0 cached) | GC: 98.5ms)

Also, top -c sometimes shows git merge-base processes taking 100% CPU and substantial RAM.

What might be going on?

2

Same error than this:

3

Oh, actually, if I log in with an unprivileged user account, it works, but not with my admin account. So I suspect it’s the upgrade part of the admin interface that’s hanging in these Git operations?

4

Seen here too. Admins and mods are blocked while things time out, I think.

Process tree:

1000     2169778  0.0  0.3 497160 13392 ?        Sl   Aug17   0:15          |       \_ pitchfork monitor - 
1000     2169783  0.0  5.4 6554324 212356 ?      Sl   Aug17   4:55          |       |   \_ pitchfork (gen:0) mold - ready
1000     2170466  0.0  2.3 6567316 91468 ?       Sl   Aug17   0:52          |       |   \_ pitchfork (gen:0) service - ready
1000     2170475  1.3  9.9 6958868 386872 ?      SNl  Aug17 310:37          |       |   |   \_ sidekiq 7.3.10 discourse [0 of 5 busy]
1000     2170499  2.6 11.3 6650900 444392 ?      Sl   Aug17 597:55          |       |   \_ pitchfork (gen:0) worker[0] - requests: 1767782, processing: /
1000     3914556  0.0  0.0      0     0 ?        Z    17:20   0:00          |       |   |   \_ [sh] <defunct>
1000     3925648  0.0  0.0      0     0 ?        Z    19:39   0:00          |       |   |   \_ [sh] <defunct>
1000     3925710  0.0  0.0  10552  2312 ?        S    19:40   0:00          |       |   |   \_ sh -c git merge-base --is-ancestor e20c10c056d2ed11533c3428e33e93cb31fa22d8 HEAD 2> /dev/null; echo $?
1000     3925711 92.4 15.1 710404 590864 ?       R    19:40   0:24          |       |   |       \_ git merge-base --is-ancestor e20c10c056d2ed11533c3428e33e93cb31fa22d8 HEAD
1000     2170558  0.1 10.3 6573012 402832 ?      Sl   Aug17  35:21          |       |   \_ pitchfork (gen:0) worker[1] - requests: 75111, processing: /
1000     3925725  0.0  0.0  10552  2316 ?        S    19:40   0:00          |       |   |   \_ sh -c git merge-base --is-ancestor e20c10c056d2ed11533c3428e33e93cb31fa22d8 HEAD 2> /dev/null; echo $?
1000     3925726 90.1 14.4 710404 562628 ?       R    19:40   0:15          |       |   |       \_ git merge-base --is-ancestor e20c10c056d2ed11533c3428e33e93cb31fa22d8 HEAD
1000     2170585  0.0  8.3 6578772 327676 ?      Rl   Aug17  10:28          |       |   \_ pitchfork (gen:0) worker[2] - requests: 8139, processing: /t/two-upcoming-books-about-videogame-history/4406
1000     2170608  0.0  7.8 6589908 305532 ?      Sl   Aug17   8:08          |       |   \_ pitchfork (gen:0) worker[3] - requests: 4055, waiting
1000     3925781  0.0  0.0  15992  2684 ?        S    19:40   0:00          |       \_ sleep 1

First seen about 2 hours ago I think.

5

So
DiscourseUpdates.has_unseen_features
seems to be a feature which is bringing in a realtime dependency on github working correctly? Even when we’re not trying to upgrade?

6

Edit: this did not solve my problem, as I discover after several hours.

I’ve unchecked this feature:

admin->site admin → version checks → clear the checkbox → save

image
image1368×584 40.7 KB

Site admin
Configure various settings relating to the management of your site
version_checks
Ping the Discourse Hub for version updates and show new version messages on the admin dashboard

7

How did you manage get the admin interface to open without error in the first place?

8

I had to wait until the git nonsense timed out, and/or could get my request served to a pitchfork which wasn’t blocked.

My forum/logs area is presently showing 4 cases of “pitchfork worker timed out” and it’s not a very busy forum at all.

9

Edit: this did not solve my problem, as I discover after several hours.

One could also use rails from an ssh access, I believe.
This might be a terrible idea. Proceed at your own risk.

# docker exec -i app rails runner 'puts SiteSetting.version_checks'
true
# docker exec -i app rails runner 'SiteSetting.set("version_checks", false)'
# docker exec -i app rails runner 'puts SiteSetting.version_checks'
false
10

It seems that Github continue with issues, that were the previous ones:

11

I’m now seeing 15 cases of pitchfork worker timed out, with two different backtraces. On the linux side pitchfork continues to accrue unreaped children and sometimes I see a git process which is - I think - the culprit.

My change is not a fix and might or might not be helpful. I will revert it.

github themselves are still broken. The change to HTTP/1.1 for git connections might be a good workaround.

12

Confirmed, they are still having issues in their systems:

fatal: remote error: GitHub is temporarily limiting some unauthenticated downloads to protect the stability of the platform. Please retry later or authenticate.

could not read Username for '``https://github.com/``': No such device or address

This time not published on https://www.githubstatus.com/ but with the same impact as the August incident.

Perhaps it is time to move to Gitlab.

13

On the github issue it seems people (at least in Europe) are seeing the problem is resolved. It seems OK for me (my servers are in Finland)

Anyone confirm they still have a problem?

14

My own forum is working again as well. However, the issue is certainly not solved in general: an outage of GitHub shouldn’t cause an outage of Discourse forums in the first place.

15

Yet it does, because any reason why rebuilding of a container fails, will kill Discourse (and that’s why we have an option: 2-container setup).

But sure. It would be nice if the app-container would survive.

16

I completely agree. And in my case, there was no rebuild or other intentional git action. These were I think ‘helpful’ git actions which turned out not to be helpful.

I would like my self-hosted forum to run unattended even if github (and discourse.org) fell into a very deep hole.

17

:100:

It is completely unacceptable to be any other way.

19

And it will — unless an admin says .launcher rebuild app. Or what am I missing now?

20

@Ed_S I think I found the issue and I have a PR for it, can you check to see if it touches the area you saw with issues?

21

oh, nice, that does seem to me to have the right shape! (Edit: I read the commentary, not the code!)

For interest, when you say per process, is that per pitchfork?