I’m loathe to try and recreate it, but I think we found/made a bug: Built and tested a chat webhook manually with curl, deleted the key, and sent more requests on the old key. Existing chat channel with threading previously disabled, worked fine previously, then corrupted.
Rough steps I took:
-
Added webhook where bot account posts,
-
works great using public curl utility to test the api
2.1 Deploy webhook to our back end software, works
2.2 don’t trust curl utility with secrit key, burn the webhook
2.3 may or may not have updated our back end to use a new key, might have sent a curl with the old key
2.4 all heck breaks loose. -
Possibly use old webhook after it was deleted
-
Chat channel totally breaks. I suspect something in the database but I’m not technical enough to know what or how to even guess. Stops rendering, gets a bunch of errors on front end. ‘unread’ status won’t go away. Sending messages in channel seems to work (to local cache I presume), but on leaving channel, posting elsewhere, and returning message is there. Refresh page, all messages gone again. 4.1 Webhook starts returning 500 internal server error.
-
Built new webhook with similar parameters posting in various channels. Other channels work, broken channel borked.
-
Deleted borked channel entirely
6.1 rebuilt channel with identical settings/name -
Pointed new working (5.) webhook at new channel - Success.
Observation - sending a bad key intentionally generates a 404 as it should. No errors logged.
38 copies in this instance - every time someone hits anything regarding chat it goes up.
Error log/trace
### Message (38 copies reported)
Module::DelegationError (Chat::WebhookEvent#username delegated to incoming_chat_webhook.username, but incoming_chat_webhook is nil: #<Chat::WebhookEvent id: 1, chat_message_id: 804, incoming_chat_webhook_id: 1, created_at: "2023-07-28 22:54:54.832175000 +0000", updated_at: "2023-07-28 22:54:54.832175000 +0000">) (eval):3:in `_fast_attributes' lib/freedom_patches/ams_include_without_root.rb:57:in `include!' app/controllers/application_controller.rb:551:in `render_json_dump' app/controllers/application_controller.rb:538:in `render_serialized' app/controllers/application_controller.rb:420:in `block in with_resolved_locale' app/controllers/application_controller.rb:420:in `with_resolved_locale' lib/middleware/omniauth_bypass_middleware.rb:74:in `call' lib/content_security_policy/middleware.rb:12:in `call' lib/middleware/anonymous_cache.rb:389:in `call' lib/middleware/gtm_script_nonce_injector.rb:10:in `call' config/initializers/100-quiet_logger.rb:20:in `call' config/initializers/100-silence_logger.rb:29:in `call' lib/middleware/enforce_hostname.rb:24:in `call' lib/middleware/request_tracker.rb:228:in `call'
### Backtrace
plugins/chat/app/models/chat/webhook_event.rb:10:in `rescue in username'
plugins/chat/app/models/chat/webhook_event.rb:10:in `username'
active_model_serializers (0.8.4) lib/active_model/serializer.rb:99:in `block in attribute'
(eval):3:in `_fast_attributes'
active_model_serializers (0.8.4) lib/active_model/serializer.rb:468:in `rescue in attributes'
active_model_serializers (0.8.4) lib/active_model/serializer.rb:455:in `attributes'
active_model_serializers (0.8.4) lib/active_model/serializer.rb:480:in `_serializable_hash'
active_model_serializers (0.8.4) lib/active_model/serializer.rb:359:in `serializable_hash'
active_model_serializers (0.8.4) lib/active_model/serializer/associations.rb:200:in `serialize'
lib/freedom_patches/ams_include_without_root.rb:57:in `include!'
active_model_serializers (0.8.4) lib/active_model/serializer.rb:368:in `block in include_associations!'
active_model_serializers (0.8.4) lib/active_model/serializer.rb:367:in `each_key'
active_model_serializers (0.8.4) lib/active_model/serializer.rb:367:in `include_associations!'
active_model_serializers (0.8.4) lib/active_model/serializer.rb:362:in `serializable_hash'
active_model_serializers (0.8.4) lib/active_model/serializer.rb:347:in `as_json'
activesupport (7.0.5.1) lib/active_support/json/encoding.rb:22:in `encode'
activesupport (7.0.5.1) lib/active_support/json/encoding.rb:22:in `encode'
activesupport (7.0.5.1) lib/active_support/core_ext/object/json.rb:42:in `to_json'
active_model_serializers (0.8.4) lib/active_model/serializer.rb:331:in `to_json'
multi_json (1.15.0) lib/multi_json/adapters/oj.rb:56:in `dump'
multi_json (1.15.0) lib/multi_json/adapters/oj.rb:56:in `dump'
multi_json (1.15.0) lib/multi_json/adapter.rb:25:in `dump'
multi_json (1.15.0) lib/multi_json.rb:139:in `dump'
app/controllers/application_controller.rb:551:in `render_json_dump'
app/controllers/application_controller.rb:538:in `render_serialized'
plugins/chat/app/controllers/chat/api/channel_messages_controller.rb:6:in `block (2 levels) in index'
plugins/chat/lib/service_runner.rb:144:in `instance_exec'
plugins/chat/lib/service_runner.rb:144:in `block in add_action'
plugins/chat/lib/service_runner.rb:128:in `call'
plugins/chat/lib/service_runner.rb:115:in `call'
plugins/chat/app/helpers/chat/with_service_helper.rb:18:in `with_service'
plugins/chat/app/controllers/chat/api/channel_messages_controller.rb:5:in `index'
actionpack (7.0.5.1) lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
actionpack (7.0.5.1) lib/abstract_controller/base.rb:215:in `process_action'
actionpack (7.0.5.1) lib/action_controller/metal/rendering.rb:165:in `process_action'
actionpack (7.0.5.1) lib/abstract_controller/callbacks.rb:234:in `block in process_action'
activesupport (7.0.5.1) lib/active_support/callbacks.rb:118:in `block in run_callbacks'
app/controllers/application_controller.rb:420:in `block in with_resolved_locale'
i18n (1.14.1) lib/i18n.rb:322:in `with_locale'
app/controllers/application_controller.rb:420:in `with_resolved_locale'
activesupport (7.0.5.1) lib/active_support/callbacks.rb:127:in `block in run_callbacks'
activesupport (7.0.5.1) lib/active_support/callbacks.rb:138:in `run_callbacks'
actionpack (7.0.5.1) lib/abstract_controller/callbacks.rb:233:in `process_action'
actionpack (7.0.5.1) lib/action_controller/metal/rescue.rb:22:in `process_action'
actionpack (7.0.5.1) lib/action_controller/metal/instrumentation.rb:67:in `block in process_action'
activesupport (7.0.5.1) lib/active_support/notifications.rb:206:in `block in instrument'
activesupport (7.0.5.1) lib/active_support/notifications/instrumenter.rb:24:in `instrument'
activesupport (7.0.5.1) lib/active_support/notifications.rb:206:in `instrument'
actionpack (7.0.5.1) lib/action_controller/metal/instrumentation.rb:66:in `process_action'
actionpack (7.0.5.1) lib/action_controller/metal/params_wrapper.rb:259:in `process_action'
activerecord (7.0.5.1) lib/active_record/railties/controller_runtime.rb:27:in `process_action'
actionpack (7.0.5.1) lib/abstract_controller/base.rb:151:in `process'
actionview (7.0.5.1) lib/action_view/rendering.rb:39:in `process'
rack-mini-profiler (3.1.0) lib/mini_profiler/profiling_methods.rb:85:in `block in profile_method'
actionpack (7.0.5.1) lib/action_controller/metal.rb:188:in `dispatch'
actionpack (7.0.5.1) lib/action_controller/metal.rb:251:in `dispatch'
actionpack (7.0.5.1) lib/action_dispatch/routing/route_set.rb:49:in `dispatch'
actionpack (7.0.5.1) lib/action_dispatch/routing/route_set.rb:32:in `serve'
actionpack (7.0.5.1) lib/action_dispatch/journey/router.rb:50:in `block in serve'
actionpack (7.0.5.1) lib/action_dispatch/journey/router.rb:32:in `each'
actionpack (7.0.5.1) lib/action_dispatch/journey/router.rb:32:in `serve'
actionpack (7.0.5.1) lib/action_dispatch/routing/route_set.rb:852:in `call'
railties (7.0.5.1) lib/rails/engine.rb:530:in `call'
railties (7.0.5.1) lib/rails/railtie.rb:226:in `public_send'
railties (7.0.5.1) lib/rails/railtie.rb:226:in `method_missing'
actionpack (7.0.5.1) lib/action_dispatch/routing/mapper.rb:19:in `block in <class:Constraints>'
actionpack (7.0.5.1) lib/action_dispatch/routing/mapper.rb:48:in `serve'
actionpack (7.0.5.1) lib/action_dispatch/journey/router.rb:50:in `block in serve'
actionpack (7.0.5.1) lib/action_dispatch/journey/router.rb:32:in `each'
actionpack (7.0.5.1) lib/action_dispatch/journey/router.rb:32:in `serve'
actionpack (7.0.5.1) lib/action_dispatch/routing/route_set.rb:852:in `call'
lib/middleware/omniauth_bypass_middleware.rb:74:in `call'
rack (2.2.7) lib/rack/tempfile_reaper.rb:15:in `call'
rack (2.2.7) lib/rack/conditional_get.rb:27:in `call'
rack (2.2.7) lib/rack/head.rb:12:in `call'
actionpack (7.0.5.1) lib/action_dispatch/http/permissions_policy.rb:38:in `call'
lib/content_security_policy/middleware.rb:12:in `call'
lib/middleware/anonymous_cache.rb:389:in `call'
lib/middleware/gtm_script_nonce_injector.rb:10:in `call'
rack (2.2.7) lib/rack/session/abstract/id.rb:266:in `context'
rack (2.2.7) lib/rack/session/abstract/id.rb:260:in `call'
actionpack (7.0.5.1) lib/action_dispatch/middleware/cookies.rb:704:in `call'
actionpack (7.0.5.1) lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
activesupport (7.0.5.1) lib/active_support/callbacks.rb:99:in `run_callbacks'
actionpack (7.0.5.1) lib/action_dispatch/middleware/callbacks.rb:26:in `call'
actionpack (7.0.5.1) lib/action_dispatch/middleware/debug_exceptions.rb:28:in `call'
actionpack (7.0.5.1) lib/action_dispatch/middleware/show_exceptions.rb:26:in `call'
logster (2.12.2) lib/logster/middleware/reporter.rb:43:in `call'
railties (7.0.5.1) lib/rails/rack/logger.rb:40:in `call_app'
railties (7.0.5.1) lib/rails/rack/logger.rb:27:in `call'
config/initializers/100-quiet_logger.rb:20:in `call'
config/initializers/100-silence_logger.rb:29:in `call'
actionpack (7.0.5.1) lib/action_dispatch/middleware/remote_ip.rb:93:in `call'
actionpack (7.0.5.1) lib/action_dispatch/middleware/request_id.rb:26:in `call'
lib/middleware/enforce_hostname.rb:24:in `call'
rack (2.2.7) lib/rack/method_override.rb:24:in `call'
actionpack (7.0.5.1) lib/action_dispatch/middleware/executor.rb:14:in `call'
rack (2.2.7) lib/rack/sendfile.rb:110:in `call'
actionpack (7.0.5.1) lib/action_dispatch/middleware/host_authorization.rb:131:in `call'
rack-mini-profiler (3.1.0) lib/mini_profiler.rb:260:in `call'
message_bus (4.3.7) lib/message_bus/rack/middleware.rb:60:in `call'
lib/middleware/request_tracker.rb:228:in `call'
railties (7.0.5.1) lib/rails/engine.rb:530:in `call'
railties (7.0.5.1) lib/rails/railtie.rb:226:in `public_send'
railties (7.0.5.1) lib/rails/railtie.rb:226:in `method_missing'
rack (2.2.7) lib/rack/urlmap.rb:74:in `block in call'
rack (2.2.7) lib/rack/urlmap.rb:58:in `each'
rack (2.2.7) lib/rack/urlmap.rb:58:in `call'
unicorn (6.1.0) lib/unicorn/http_server.rb:634:in `process_client'
unicorn (6.1.0) lib/unicorn/http_server.rb:739:in `worker_loop'
unicorn (6.1.0) lib/unicorn/http_server.rb:547:in `spawn_missing_workers'
unicorn (6.1.0) lib/unicorn/http_server.rb:143:in `start'
unicorn (6.1.0) bin/unicorn:128:in `<top (required)>'
vendor/bundle/ruby/3.2.0/bin/unicorn:25:in `load'
vendor/bundle/ruby/3.2.0/bin/unicorn:25:in `<main>'
