Chat webhook possible bug - delegated, but webhook is nil

Bug
1

I’m loathe to try and recreate it, but I think we found/made a bug: Built and tested a chat webhook manually with curl, deleted the key, and sent more requests on the old key. Existing chat channel with threading previously disabled, worked fine previously, then corrupted.

Rough steps I took:

  1. Added webhook where bot account posts,

  2. works great using public curl utility to test the api
    2.1 Deploy webhook to our back end software, works
    2.2 don’t trust curl utility with secrit key, burn the webhook
    2.3 may or may not have updated our back end to use a new key, might have sent a curl with the old key
    2.4 all heck breaks loose.

  3. Possibly use old webhook after it was deleted

  4. Chat channel totally breaks. I suspect something in the database but I’m not technical enough to know what or how to even guess. Stops rendering, gets a bunch of errors on front end. ‘unread’ status won’t go away. Sending messages in channel seems to work (to local cache I presume), but on leaving channel, posting elsewhere, and returning message is there. Refresh page, all messages gone again. 4.1 Webhook starts returning 500 internal server error.

  5. Built new webhook with similar parameters posting in various channels. Other channels work, broken channel borked.

  6. Deleted borked channel entirely
    6.1 rebuilt channel with identical settings/name

  7. Pointed new working (5.) webhook at new channel - Success.

Observation - sending a bad key intentionally generates a 404 as it should. No errors logged.

38 copies in this instance - every time someone hits anything regarding chat it goes up.

Error log/trace 
### Message (38 copies reported)

Module::DelegationError (Chat::WebhookEvent#username delegated to incoming_chat_webhook.username, but incoming_chat_webhook is nil: #<Chat::WebhookEvent id: 1, chat_message_id: 804, incoming_chat_webhook_id: 1, created_at: "2023-07-28 22:54:54.832175000 +0000", updated_at: "2023-07-28 22:54:54.832175000 +0000">) (eval):3:in `_fast_attributes' lib/freedom_patches/ams_include_without_root.rb:57:in `include!' app/controllers/application_controller.rb:551:in `render_json_dump' app/controllers/application_controller.rb:538:in `render_serialized' app/controllers/application_controller.rb:420:in `block in with_resolved_locale' app/controllers/application_controller.rb:420:in `with_resolved_locale' lib/middleware/omniauth_bypass_middleware.rb:74:in `call' lib/content_security_policy/middleware.rb:12:in `call' lib/middleware/anonymous_cache.rb:389:in `call' lib/middleware/gtm_script_nonce_injector.rb:10:in `call' config/initializers/100-quiet_logger.rb:20:in `call' config/initializers/100-silence_logger.rb:29:in `call' lib/middleware/enforce_hostname.rb:24:in `call' lib/middleware/request_tracker.rb:228:in `call'

### Backtrace

plugins/chat/app/models/chat/webhook_event.rb:10:in `rescue in username'
plugins/chat/app/models/chat/webhook_event.rb:10:in `username'
active_model_serializers (0.8.4) lib/active_model/serializer.rb:99:in `block in attribute'
(eval):3:in `_fast_attributes'
active_model_serializers (0.8.4) lib/active_model/serializer.rb:468:in `rescue in attributes'
active_model_serializers (0.8.4) lib/active_model/serializer.rb:455:in `attributes'
active_model_serializers (0.8.4) lib/active_model/serializer.rb:480:in `_serializable_hash'
active_model_serializers (0.8.4) lib/active_model/serializer.rb:359:in `serializable_hash'
active_model_serializers (0.8.4) lib/active_model/serializer/associations.rb:200:in `serialize'
lib/freedom_patches/ams_include_without_root.rb:57:in `include!'
active_model_serializers (0.8.4) lib/active_model/serializer.rb:368:in `block in include_associations!'
active_model_serializers (0.8.4) lib/active_model/serializer.rb:367:in `each_key'
active_model_serializers (0.8.4) lib/active_model/serializer.rb:367:in `include_associations!'
active_model_serializers (0.8.4) lib/active_model/serializer.rb:362:in `serializable_hash'
active_model_serializers (0.8.4) lib/active_model/serializer.rb:347:in `as_json'
activesupport (7.0.5.1) lib/active_support/json/encoding.rb:22:in `encode'
activesupport (7.0.5.1) lib/active_support/json/encoding.rb:22:in `encode'
activesupport (7.0.5.1) lib/active_support/core_ext/object/json.rb:42:in `to_json'
active_model_serializers (0.8.4) lib/active_model/serializer.rb:331:in `to_json'
multi_json (1.15.0) lib/multi_json/adapters/oj.rb:56:in `dump'
multi_json (1.15.0) lib/multi_json/adapters/oj.rb:56:in `dump'
multi_json (1.15.0) lib/multi_json/adapter.rb:25:in `dump'
multi_json (1.15.0) lib/multi_json.rb:139:in `dump'
app/controllers/application_controller.rb:551:in `render_json_dump'
app/controllers/application_controller.rb:538:in `render_serialized'
plugins/chat/app/controllers/chat/api/channel_messages_controller.rb:6:in `block (2 levels) in index'
plugins/chat/lib/service_runner.rb:144:in `instance_exec'
plugins/chat/lib/service_runner.rb:144:in `block in add_action'
plugins/chat/lib/service_runner.rb:128:in `call'
plugins/chat/lib/service_runner.rb:115:in `call'
plugins/chat/app/helpers/chat/with_service_helper.rb:18:in `with_service'
plugins/chat/app/controllers/chat/api/channel_messages_controller.rb:5:in `index'
actionpack (7.0.5.1) lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
actionpack (7.0.5.1) lib/abstract_controller/base.rb:215:in `process_action'
actionpack (7.0.5.1) lib/action_controller/metal/rendering.rb:165:in `process_action'
actionpack (7.0.5.1) lib/abstract_controller/callbacks.rb:234:in `block in process_action'
activesupport (7.0.5.1) lib/active_support/callbacks.rb:118:in `block in run_callbacks'
app/controllers/application_controller.rb:420:in `block in with_resolved_locale'
i18n (1.14.1) lib/i18n.rb:322:in `with_locale'
app/controllers/application_controller.rb:420:in `with_resolved_locale'
activesupport (7.0.5.1) lib/active_support/callbacks.rb:127:in `block in run_callbacks'
activesupport (7.0.5.1) lib/active_support/callbacks.rb:138:in `run_callbacks'
actionpack (7.0.5.1) lib/abstract_controller/callbacks.rb:233:in `process_action'
actionpack (7.0.5.1) lib/action_controller/metal/rescue.rb:22:in `process_action'
actionpack (7.0.5.1) lib/action_controller/metal/instrumentation.rb:67:in `block in process_action'
activesupport (7.0.5.1) lib/active_support/notifications.rb:206:in `block in instrument'
activesupport (7.0.5.1) lib/active_support/notifications/instrumenter.rb:24:in `instrument'
activesupport (7.0.5.1) lib/active_support/notifications.rb:206:in `instrument'
actionpack (7.0.5.1) lib/action_controller/metal/instrumentation.rb:66:in `process_action'
actionpack (7.0.5.1) lib/action_controller/metal/params_wrapper.rb:259:in `process_action'
activerecord (7.0.5.1) lib/active_record/railties/controller_runtime.rb:27:in `process_action'
actionpack (7.0.5.1) lib/abstract_controller/base.rb:151:in `process'
actionview (7.0.5.1) lib/action_view/rendering.rb:39:in `process'
rack-mini-profiler (3.1.0) lib/mini_profiler/profiling_methods.rb:85:in `block in profile_method'
actionpack (7.0.5.1) lib/action_controller/metal.rb:188:in `dispatch'
actionpack (7.0.5.1) lib/action_controller/metal.rb:251:in `dispatch'
actionpack (7.0.5.1) lib/action_dispatch/routing/route_set.rb:49:in `dispatch'
actionpack (7.0.5.1) lib/action_dispatch/routing/route_set.rb:32:in `serve'
actionpack (7.0.5.1) lib/action_dispatch/journey/router.rb:50:in `block in serve'
actionpack (7.0.5.1) lib/action_dispatch/journey/router.rb:32:in `each'
actionpack (7.0.5.1) lib/action_dispatch/journey/router.rb:32:in `serve'
actionpack (7.0.5.1) lib/action_dispatch/routing/route_set.rb:852:in `call'
railties (7.0.5.1) lib/rails/engine.rb:530:in `call'
railties (7.0.5.1) lib/rails/railtie.rb:226:in `public_send'
railties (7.0.5.1) lib/rails/railtie.rb:226:in `method_missing'
actionpack (7.0.5.1) lib/action_dispatch/routing/mapper.rb:19:in `block in <class:Constraints>'
actionpack (7.0.5.1) lib/action_dispatch/routing/mapper.rb:48:in `serve'
actionpack (7.0.5.1) lib/action_dispatch/journey/router.rb:50:in `block in serve'
actionpack (7.0.5.1) lib/action_dispatch/journey/router.rb:32:in `each'
actionpack (7.0.5.1) lib/action_dispatch/journey/router.rb:32:in `serve'
actionpack (7.0.5.1) lib/action_dispatch/routing/route_set.rb:852:in `call'
lib/middleware/omniauth_bypass_middleware.rb:74:in `call'
rack (2.2.7) lib/rack/tempfile_reaper.rb:15:in `call'
rack (2.2.7) lib/rack/conditional_get.rb:27:in `call'
rack (2.2.7) lib/rack/head.rb:12:in `call'
actionpack (7.0.5.1) lib/action_dispatch/http/permissions_policy.rb:38:in `call'
lib/content_security_policy/middleware.rb:12:in `call'
lib/middleware/anonymous_cache.rb:389:in `call'
lib/middleware/gtm_script_nonce_injector.rb:10:in `call'
rack (2.2.7) lib/rack/session/abstract/id.rb:266:in `context'
rack (2.2.7) lib/rack/session/abstract/id.rb:260:in `call'
actionpack (7.0.5.1) lib/action_dispatch/middleware/cookies.rb:704:in `call'
actionpack (7.0.5.1) lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
activesupport (7.0.5.1) lib/active_support/callbacks.rb:99:in `run_callbacks'
actionpack (7.0.5.1) lib/action_dispatch/middleware/callbacks.rb:26:in `call'
actionpack (7.0.5.1) lib/action_dispatch/middleware/debug_exceptions.rb:28:in `call'
actionpack (7.0.5.1) lib/action_dispatch/middleware/show_exceptions.rb:26:in `call'
logster (2.12.2) lib/logster/middleware/reporter.rb:43:in `call'
railties (7.0.5.1) lib/rails/rack/logger.rb:40:in `call_app'
railties (7.0.5.1) lib/rails/rack/logger.rb:27:in `call'
config/initializers/100-quiet_logger.rb:20:in `call'
config/initializers/100-silence_logger.rb:29:in `call'
actionpack (7.0.5.1) lib/action_dispatch/middleware/remote_ip.rb:93:in `call'
actionpack (7.0.5.1) lib/action_dispatch/middleware/request_id.rb:26:in `call'
lib/middleware/enforce_hostname.rb:24:in `call'
rack (2.2.7) lib/rack/method_override.rb:24:in `call'
actionpack (7.0.5.1) lib/action_dispatch/middleware/executor.rb:14:in `call'
rack (2.2.7) lib/rack/sendfile.rb:110:in `call'
actionpack (7.0.5.1) lib/action_dispatch/middleware/host_authorization.rb:131:in `call'
rack-mini-profiler (3.1.0) lib/mini_profiler.rb:260:in `call'
message_bus (4.3.7) lib/message_bus/rack/middleware.rb:60:in `call'
lib/middleware/request_tracker.rb:228:in `call'
railties (7.0.5.1) lib/rails/engine.rb:530:in `call'
railties (7.0.5.1) lib/rails/railtie.rb:226:in `public_send'
railties (7.0.5.1) lib/rails/railtie.rb:226:in `method_missing'
rack (2.2.7) lib/rack/urlmap.rb:74:in `block in call'
rack (2.2.7) lib/rack/urlmap.rb:58:in `each'
rack (2.2.7) lib/rack/urlmap.rb:58:in `call'
unicorn (6.1.0) lib/unicorn/http_server.rb:634:in `process_client'
unicorn (6.1.0) lib/unicorn/http_server.rb:739:in `worker_loop'
unicorn (6.1.0) lib/unicorn/http_server.rb:547:in `spawn_missing_workers'
unicorn (6.1.0) lib/unicorn/http_server.rb:143:in `start'
unicorn (6.1.0) bin/unicorn:128:in `<top (required)>'
vendor/bundle/ruby/3.2.0/bin/unicorn:25:in `load'
vendor/bundle/ruby/3.2.0/bin/unicorn:25:in `<main>'
1 Like
2

Maybe use read webhook_url (and use $webhook_url where you’d otherwise use the webhook URL) to keep the webhook URL out of shell history logs (assuming that that’s an issue).

2 Likes
3

Fair advice. fwiw I was playing with settings from a locked-down work computer so was using a web-based curl utility to send the commands for testing. I didn’t trust that web-service to not cache the commands and/or leak them at a further date.

Would be nice feature for chat hooks: ability to recycle/burn the key without deleting the whole hook and re-creating it.

1 Like
4

Great catch thanks :+1:

This will be fixed in:

Note, this is only preventing the error to happen, if you are already in this state you have two solutions for now:

  • create a new channel
  • manually delete the orphaned records of chat_webhook_events

Given I had only one report of this Im not providing anything to handle this, but if there are cases I will assist.

4 Likes
5

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.