On my forum for example, I became suddenly unable to log in without any admin action on my part. The problem isn’t that you can’t upgrade Discourse when GitHub is unreachable, it’s that GitHub being unreachable takes down your forum because Discourse tries to check for upgrades that you could perform.
Yes, exactly.
I think this was caused by a way way too eager want of telling people about new features, in the current user serializer of all places, paired with a 3 years old change that moved us away from the normal shallow clones:
@Ed_S and @jeanas you both admins on the affected instances right? This should only affect admins while GitHub is down.
Yes, exactly, I could log in with another account but not with my admin account.
Cool I have merged the PR. Update when it hits the release channel you run, then wait a couple of days for GitHub to go down again and let me know how it goes during it.
great, thanks. Just to confirm, I’m an admin, I have two mods, and one of those was affected. So, either it affects all staff, or it affects all logged-in users. I was OK with incognito access. And in my case, I could log in some of the time with much delay - I wasn’t entirely locked out.
Awesome, thanks a lot for the quick fix!
Think I have encountered this exact issue as the OP.
Server was under increasing load. Very strange patchy performance then the Oops screen became permanent. Over the last 3/4 days.
Now many lets-encrypt rate limits trigged and rebuilds out the door later before I found this topic, I have since tested with a non-admin user today and I can log in!
Yet admins still cannot login (and probably mods by virtue, but not sure), since this app was rebuilt many times today it has to be latest version.
What am I missing?
Are you able to log in as an admin?
Some feedback - I’m sure I hit this issue approx 19th Sep onwards (initially misdiagnosing as was unaware of this topic/report).
I have no idea of GitHub status over this period, but maybe someone can check that, as I see this issue goes back to August uptime/downtime events.
On the latest rebuild as of yesterday but existing admins can not log in.
Which is a weird state to be in, a ship without a captain but passengers and maybe no crew.
I was thinking as I was writing this reply, maybe if I create a new admin using rake command it might work, and guess what, it did!
So what is occurring here now is a historical admin is trying to log in and is getting the “Oops…” screen every single time, but a fresh admin user has no login access issues.
Yes. The issue looks fixed in my case.
So @Ed_S you eventually got back in right, but indicating maybe having to wait a period, whereas I still can’t get back in (original Admin accs affected), what I am wondering is. maybe it is wider than “git hang” issue and what has so far been addressed with commits, and so do you or anyone have any tips/insights on a diagnostics to run that might reveal something more?
Appreciate any help in advance.
Yes, my forum was affected by Github’s systems being too defensive, and it returned to normal as they rolled out some kind of fix, some hours later.
It might be that they have done something again which breaks Discourse forums. It might be geography specific, or specific to certain providers. I’m using Hetzner, in Helsinki.
Ok, this instance is US based DO droplet. 
Ok, ran a rebuild today, to see if it might shake things up but nothing.
Currently pre-break admins still cannot log into the discourse instance, while admins created post-break can.
Maybe it’s just me but a supposed bug that only affects the admins and mod accounts would surely raise a security eyebrow?
If it were a targeted disruption, it woudl be sneaky and insidious don’t ya think?
- The users remain intact and untouched, oblivious that the Captain and Crew are tied up in the hold while the ship sails toward the horizon.
I’ve read this topic, and have a basic grasp of the alleged git dependency root casue that shouldn’t have been , but if you were a devious rouge AI, isn’t this a perfect way to hack discourse?
Add to that, in this instance it is still not fixed. I have no idea why.
Beyond my meager musings, I checked the logs right after rebuild to see what errors occurred and below is one error, I have no idea if it is relevant or not but post it in case it holds some value.
activesupport (8.1.4) lib/active_support/values/time_zone.rb:461:in 'Date._parse'
activesupport (8.1.4) lib/active_support/values/time_zone.rb:461:in 'ActiveSupport::TimeZone#parse'
app/serializers/current_user_serializer.rb:216:in 'CurrentUserSerializer#has_new_upcoming_changes'
(eval at active_model_serializers-0.8.4/lib/active_model/serializer.rb:467):81:in 'CurrentUserSerializer#_fast_attributes'
active_model_serializers (0.8.4) lib/active_model/serializer.rb:456:in 'ActiveModel::Serializer#attributes'
active_model_serializers (0.8.4) lib/active_model/serializer.rb:480:in 'ActiveModel::Serializer#_serializable_hash'
active_model_serializers (0.8.4) lib/active_model/serializer.rb:359:in 'ActiveModel::Serializer#serializable_hash'
active_model_serializers (0.8.4) lib/active_model/serializer.rb:347:in 'ActiveModel::Serializer#as_json'
activesupport (8.1.4) lib/active_support/json/encoding.rb:53:in 'Oj::Rails::Encoder#encode'
activesupport (8.1.4) lib/active_support/json/encoding.rb:53:in 'ActiveSupport::JSON.encode'
activesupport (8.1.4) lib/active_support/core_ext/object/json.rb:42:in 'ActiveSupport::ToJsonWithActiveSupportEncoder#to_json'
active_model_serializers (0.8.4) lib/active_model/serializer.rb:331:in 'ActiveModel::Serializer#to_json'
multi_json (1.21.1) lib/multi_json/adapters/oj.rb:67:in 'Oj.dump'
multi_json (1.21.1) lib/multi_json/adapters/oj.rb:67:in 'MultiJSON::Adapters::Oj#dump'
multi_json (1.21.1) lib/multi_json/adapter.rb:110:in 'MultiJSON::Adapter.dump'
multi_json (1.21.1) lib/multi_json.rb:215:in 'MultiJSON.generate'
multi_json (1.21.1) lib/multi_json/deprecated.rb:34:in 'Kernel#public_send'
multi_json (1.21.1) lib/multi_json/deprecated.rb:34:in 'block in MultiJSON.deprecate_alias'
multi_json (1.21.1) lib/multi_json.rb:298:in 'Kernel#public_send'
multi_json (1.21.1) lib/multi_json.rb:298:in 'block (2 levels) in <module:MultiJson>'
lib/application_layout_preloader.rb:81:in 'ApplicationLayoutPreloader#preload_current_user_data'
lib/application_layout_preloader.rb:32:in 'ApplicationLayoutPreloader#preloaded_data'
app/helpers/application_helper.rb:987:in 'ApplicationHelper#preloaded_json'
app/views/layouts/application.html.erb:111
actionview (8.1.4) lib/action_view/base.rb:281:in 'Kernel#public_send'
actionview (8.1.4) lib/action_view/base.rb:281:in 'ActionView::Base#_run'
actionview (8.1.4) lib/action_view/template.rb:284:in 'block in ActionView::Template#render'
activesupport (8.1.4) lib/active_support/notifications.rb:212:in 'ActiveSupport::Notifications.instrument'
actionview (8.1.4) lib/action_view/template.rb:590:in 'ActionView::Template#instrument_render_template'
actionview (8.1.4) lib/action_view/template.rb:272:in 'ActionView::Template#render'
rack-mini-profiler (5.0.0) lib/mini_profiler/profiling_methods.rb:90:in 'block in ActionView::Template#profile_method'
actionview (8.1.4) lib/action_view/renderer/template_renderer.rb:77:in 'block in ActionView::TemplateRenderer#render_with_layout'
activesupport (8.1.4) lib/active_support/notifications.rb:210:in 'block in ActiveSupport::Notifications.instrument'
activesupport (8.1.4) lib/active_support/notifications/instrumenter.rb:58:in 'ActiveSupport::Notifications::Instrumenter#instrument'
activesupport (8.1.4) lib/active_support/notifications.rb:210:in 'ActiveSupport::Notifications.instrument'
actionview (8.1.4) lib/action_view/renderer/template_renderer.rb:75:in 'ActionView::TemplateRenderer#render_with_layout'
actionview (8.1.4) lib/action_view/renderer/template_renderer.rb:59:in 'ActionView::TemplateRenderer#render_template'
actionview (8.1.4) lib/action_view/renderer/template_renderer.rb:11:in 'ActionView::TemplateRenderer#render'
actionview (8.1.4) lib/action_view/renderer/renderer.rb:58:in 'ActionView::Renderer#render_template_to_object'
actionview (8.1.4) lib/action_view/renderer/renderer.rb:31:in 'ActionView::Renderer#render_to_object'
actionview (8.1.4) lib/action_view/rendering.rb:136:in 'block in ActionView::Rendering#_render_template'
actionview (8.1.4) lib/action_view/base.rb:308:in 'ActionView::Base#in_rendering_context'
actionview (8.1.4) lib/action_view/rendering.rb:135:in 'ActionView::Rendering#_render_template'
actionpack (8.1.4) lib/action_controller/metal/streaming.rb:179:in 'ActionController::Streaming#_render_template'
actionview (8.1.4) lib/action_view/rendering.rb:122:in 'ActionView::Rendering#render_to_body'
actionpack (8.1.4) lib/action_controller/metal/rendering.rb:192:in 'ActionController::Rendering#render_to_body'
actionpack (8.1.4) lib/action_controller/metal/renderers.rb:155:in 'ActionController::Renderers#render_to_body'
actionpack (8.1.4) lib/abstract_controller/rendering.rb:28:in 'AbstractController::Rendering#render'
actionpack (8.1.4) lib/action_controller/metal/rendering.rb:173:in 'ActionController::Rendering#render'
actionpack (8.1.4) lib/action_controller/metal/instrumentation.rb:31:in 'block (2 levels) in ActionController::Instrumentation#render'
activesupport (8.1.4) lib/active_support/benchmark.rb:17:in 'ActiveSupport::Benchmark.realtime'
actionpack (8.1.4) lib/action_controller/metal/instrumentation.rb:31:in 'block in ActionController::Instrumentation#render'
actionpack (8.1.4) lib/action_controller/metal/instrumentation.rb:100:in 'ActionController::Instrumentation#cleanup_view_runtime'
activerecord (8.1.4) lib/active_record/railties/controller_runtime.rb:48:in 'ActiveRecord::Railties::ControllerRuntime#cleanup_view_runtime'
actionpack (8.1.4) lib/action_controller/metal/instrumentation.rb:30:in 'ActionController::Instrumentation#render'
lib/topic_list_responder.rb:16:in 'block (2 levels) in TopicListResponder#respond_with_list'
actionpack (8.1.4) lib/action_controller/metal/mime_responds.rb:224:in 'ActionController::MimeResponds#respond_to'
lib/topic_list_responder.rb:7:in 'TopicListResponder#respond_with_list'
app/controllers/list_controller.rb:119:in 'block (2 levels) in <class:ListController>'
actionpack (8.1.4) lib/action_controller/metal/basic_implicit_render.rb:8:in 'ActionController::BasicImplicitRender#send_action'
actionpack (8.1.4) lib/abstract_controller/base.rb:221:in 'AbstractController::Base#process_action'
actionpack (8.1.4) lib/action_controller/metal/rendering.rb:199:in 'ActionController::Rendering#process_action'
actionpack (8.1.4) lib/abstract_controller/callbacks.rb:267:in 'block in AbstractController::Callbacks#process_action'
activesupport (8.1.4) lib/active_support/callbacks.rb:121:in 'block in ActiveSupport::Callbacks#run_callbacks'
app/controllers/application_controller.rb:458:in 'block in ApplicationController#with_resolved_locale'
i18n (1.14.8) lib/i18n.rb:354:in 'I18n::Base#with_locale'
app/controllers/application_controller.rb:458:in 'ApplicationController#with_resolved_locale'
activesupport (8.1.4) lib/active_support/callbacks.rb:130:in 'block in ActiveSupport::Callbacks#run_callbacks'
app/controllers/application_controller.rb:1140:in 'ApplicationController#ensure_dont_cache_page'
activesupport (8.1.4) lib/active_support/callbacks.rb:130:in 'block in ActiveSupport::Callbacks#run_callbacks'
activesupport (8.1.4) lib/active_support/callbacks.rb:141:in 'ActiveSupport::Callbacks#run_callbacks'
actionpack (8.1.4) lib/abstract_controller/callbacks.rb:266:in 'AbstractController::Callbacks#process_action'
actionpack (8.1.4) lib/action_controller/metal/rescue.rb:36:in 'ActionController::Rescue#process_action'
actionpack (8.1.4) lib/action_controller/metal/instrumentation.rb:76:in 'block in ActionController::Instrumentation#process_action'
activesupport (8.1.4) lib/active_support/notifications.rb:210:in 'block in ActiveSupport::Notifications.instrument'
activesupport (8.1.4) lib/ac
I can also say I found similar if not identical errors logged a bit back, identical to the OP posted ones. While also lots of those pitchforks I think timeouts.
None of which now appear since updating the build a few days ago, yet the situation remains the same.
I have reverted to own topic for my own instance experience since it remains unresolved and there may be genuine differentiation in this instance - https://meta.discourse.org/t/the-git-hang-and-the-admin-mod-user-accs-oops-error-page-error/