signature/Content-Length incorrect in webhook headers


(Adam Warner) #1

Hi there,

Had a search around, and the closest post I could find with a similar issue is:

However, I am not certain it’s exactly the same. I’m setting up a client device to consume webhooks from our Discourse server, and noticed that some of the hooks aren’t being processed. Taking a closer look at my logs, I notice that the hooks that are not being processed are turned away due to a bad signature.

Some examples:

08/20/2017 17:23:27
Processing Incoming Hook ID:24919
Event is of Type: topic
Exp Content Length: 7601
Act Content Length: 7599
pays: {"topic":{"id":4560,"title":":robot: Greetings!","fancy_title":":robot: Greetings!","posts_count":1,"created_at":"2017-08-20T17:23:23.763Z","views":0,"reply_count":0,"participant_count":1,"like_count":0,"last_posted_at":"2017-08-20T17:23:23.865Z","visible":true,"closed":false,"archived":false,"has_summary":false,"archetype":"private_message","slug":"robot-greetings","category_id":null,"word_count":112,"deleted_at":null,"pending_posts_count":0,"user_id":-2,"pm_with_non_human_user":true,"draft":null,"draft_key":"topic_4560","draft_sequence":0,"unpinned":null,"pinned_globally":false,"pinned":false,"pinned_at":null,"pinned_until":null,"details":{"created_by":{"id":-2,"username":"discobot","avatar_template":"/user_avatar/discourse.pi-hole.net/discobot/{size}/1508_1.png"},"last_poster":{"id":-2,"username":"discobot","avatar_template":"/user_avatar/discourse.pi-hole.net/discobot/{size}/1508_1.png"},"allowed_groups":[],"allowed_users":[{"id":-2,"username":"discobot","avatar_template":"/user_avatar/discourse.pi-hole.net/discobot/{size}/1508_1.png"},{"id":3825,"username":"bengt-a","avatar_template":"/letter_avatar_proxy/v2/letter/b/dc4da7/{size}.png"}],"participants":[{"id":-2,"username":"discobot","avatar_template":"/user_avatar/discourse.pi-hole.net/discobot/{size}/1508_1.png","post_count":1,"primary_group_name":null,"primary_group_flair_url":null,"primary_group_flair_color":null,"primary_group_flair_bg_color":null}],"suggested_topics":[{"id":3011,"title":"Welcome to Pi-hole Userspace!","fancy_title":"Welcome to Pi-hole Userspace!","slug":"welcome-to-pi-hole-userspace","posts_count":2,"reply_count":0,"highest_post_number":2,"image_url":null,"created_at":"2017-05-08T09:22:29.858Z","last_posted_at":"2017-05-08T09:23:43.914Z","bumped":true,"bumped_at":"2017-05-08T09:23:43.914Z","unseen":false,"last_read_post_number":1,"unread":0,"new_posts":1,"pinned":false,"unpinned":null,"visible":true,"closed":false,"archived":false,"notification_level":3,"bookmarked":false,"liked":false,"archetype":"private_message","like_count":0,"views":1,"category_id":null,"tags":[],"featured_link":null,"posters":[{"extras":null,"description":"Original Poster","user":{"id":-1,"username":"system","avatar_template":"/user_avatar/discourse.pi-hole.net/system/{size}/1_1.png"}},{"extras":"latest","description":"Most Recent Poster","user":{"id":3028,"username":"Geffers","avatar_template":"/letter_avatar_proxy/v2/letter/g/c89c15/{size}.png"}}]},{"id":2468,"title":"Account temporarily on hold","fancy_title":"Account temporarily on hold","slug":"account-temporarily-on-hold","posts_count":2,"reply_count":0,"highest_post_number":2,"image_url":null,"created_at":"2017-04-02T01:40:45.109Z","last_posted_at":"2017-04-03T13:55:01.705Z","bumped":true,"bumped_at":"2017-04-03T13:55:01.705Z","unseen":false,"last_read_post_number":1,"unread":0,"new_posts":1,"pinned":false,"unpinned":null,"visible":true,"closed":false,"archived":false,"notification_level":3,"bookmarked":false,"liked":false,"archetype":"private_message","like_count":0,"views":1,"category_id":null,"tags":[],"featured_link":null,"posters":[{"extras":null,"description":"Original Poster","user":{"id":-1,"username":"system","avatar_template":"/user_avatar/discourse.pi-hole.net/system/{size}/1_1.png"}},{"extras":"latest","description":"Most Recent Poster","user":{"id":2738,"username":"willzim","avatar_template":"/letter_avatar_proxy/v2/letter/w/7c8e57/{size}.png"}}]},{"id":2452,"title":"Welcome to Pi-hole Userspace!","fancy_title":"Welcome to Pi-hole Userspace!","slug":"welcome-to-pi-hole-userspace","posts_count":2,"reply_count":0,"highest_post_number":2,"image_url":null,"created_at":"2017-03-31T19:11:35.926Z","last_posted_at":"2017-03-31T19:13:45.533Z","bumped":true,"bumped_at":"2017-03-31T19:13:45.533Z","unseen":false,"last_read_post_number":1,"unread":0,"new_posts":1,"pinned":false,"unpinned":null,"visible":true,"closed":false,"archived":false,"notification_level":3,"bookmarked":false,"liked":false,"archetype":"private_message","like_count":0,"views":2,"category_id":null,"tags":[],"featured_link":null,"posters":[{"extras":null,"description":"Original Poster","user":{"id":-1,"username":"system","avatar_template":"/user_avatar/discourse.pi-hole.net/system/{size}/1_1.png"}},{"extras":"latest","description":"Most Recent Poster","user":{"id":2726,"username":"teunbruijnen","avatar_template":"/letter_avatar_proxy/v2/letter/t/e79b87/{size}.png"}}]},{"id":2098,"title":"Welcome to Pi-hole Userspace!","fancy_title":"Welcome to Pi-hole Userspace!","slug":"welcome-to-pi-hole-userspace","posts_count":2,"reply_count":0,"highest_post_number":2,"image_url":null,"created_at":"2017-03-09T11:29:04.086Z","last_posted_at":"2017-03-09T11:50:35.551Z","bumped":true,"bumped_at":"2017-03-09T11:50:35.551Z","unseen":false,"last_read_post_number":1,"unread":0,"new_posts":1,"pinned":false,"unpinned":null,"visible":true,"closed":false,"archived":false,"notification_level":3,"bookmarked":false,"liked":false,"archetype":"private_message","like_count":0,"views":1,"category_id":null,"tags":[],"featured_link":null,"posters":[{"extras":null,"description":"Original Poster","user":{"id":-1,"username":"system","avatar_template":"/user_avatar/discourse.pi-hole.net/system/{size}/1_1.png"}},{"extras":"latest","description":"Most Recent Poster","user":{"id":2521,"username":"Tensai_Computers","avatar_template":"/letter_avatar_proxy/v2/letter/t/c67d28/{size}.png"}}]},{"id":2052,"title":"Welcome to Pi-hole Userspace!","fancy_title":"Welcome to Pi-hole Userspace!","slug":"welcome-to-pi-hole-userspace","posts_count":2,"reply_count":0,"highest_post_number":2,"image_url":null,"created_at":"2017-03-06T07:26:12.218Z","last_posted_at":"2017-03-06T07:47:11.118Z","bumped":true,"bumped_at":"2017-03-06T07:47:11.118Z","unseen":false,"last_read_post_number":1,"unread":0,"new_posts":1,"pinned":false,"unpinned":null,"visible":true,"closed":false,"archived":false,"notification_level":3,"bookmarked":false,"liked":false,"archetype":"private_message","like_count":0,"views":1,"category_id":null,"tags":[],"featured_link":null,"posters":[{"extras":null,"description":"Original Poster","user":{"id":-1,"username":"system","avatar_template":"/user_avatar/discourse.pi-hole.net/system/{size}/1_1.png"}},{"extras":"latest","description":"Most Recent Poster","user":{"id":2497,"username":"kebabia","avatar_template":"/letter_avatar_proxy/v2/letter/k/dc4da7/{size}.png"}}]}],"links":[{"url":"/guidelines","title":null,"fancy_title":null,"internal":true,"attachment":false,"reflection":false,"clicks":0,"user_id":-2,"domain":"discourse.pi-hole.net"},{"url":"/badges","title":null,"fancy_title":null,"internal":true,"attachment":false,"reflection":false,"clicks":0,"user_id":-2,"domain":"discourse.pi-hole.net"},{"url":"/about","title":null,"fancy_title":null,"internal":true,"attachment":false,"reflection":false,"clicks":0,"user_id":-2,"domain":"discourse.pi-hole.net"}],"notification_level":1,"can_move_posts":true,"can_edit":true,"can_delete":true,"can_remove_allowed_users":true,"can_invite_to":true,"can_invite_via_email":true,"can_create_post":true,"can_reply_as_new_topic":true,"can_flag_topic":true},"highest_post_number":1,"deleted_by":null,"has_deleted":false,"actions_summary":[{"id":4,"count":0,"hidden":false,"can_act":true},{"id":7,"count":0,"hidden":false,"can_act":true},{"id":8,"count":0,"hidden":false,"can_act":true}],"chunk_size":20,"bookmarked":null,"message_archived":false,"tags":[],"featured_link":null,"topic_timer":null,"unicode_title":"🤖 Greetings!","message_bus_last_id":2,"can_vote":false,"vote_count":null,"user_voted":false}}
sent: fb5ba4ad217d3d3fb9bd73fe971572a5e0277e079b940eb76e52e369994b09eb
calc: ba643da8460466a9b780d18fb1b56f78afe564271aac0e4e2748a4ff77ead8c0
Bad Signature!
08/20/2017 17:23:25
Processing Incoming Hook ID:24918
Event is of Type: post
Exp Content Length: 2440
Act Content Length: 2434
pays: {"post":{"id":16324,"name":"discobot","username":"discobot","avatar_template":"/user_avatar/discourse.pi-hole.net/discobot/{size}/1508_1.png","created_at":"2017-08-20T17:23:23.865Z","cooked":"\u003cp\u003eThanks for joining Pi-hole Userspace, and welcome!\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003eI’m only a robot, but \u003ca href=\"/about\"\u003eour friendly staff\u003c/a\u003e are also here to help if you need to reach a person.\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003eFor safety reasons, we temporarily limit what new users can do. You’ll gain new abilities (and \u003ca href=\"/badges\"\u003ebadges\u003c/a\u003e) as we get to know you.\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003eWe believe in \u003ca href=\"/guidelines\"\u003ecivilized community behavior\u003c/a\u003e at all times.\u003c/p\u003e\n\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eIf you’d like to learn more, select \u003cimg src=\"https://discourse.pi-hole.net/images/font-awesome-ellipsis.png\" width=\"16\" height=\"16\"\u003e below  and \u003cimg src=\"https://discourse.pi-hole.net/images/font-awesome-bookmark.png\" width=\"16\" height=\"16\"\u003e \u003cstrong\u003ebookmark this private message\u003c/strong\u003e.  If you do, there may be a \u003cimg src=\"https://discourse.pi-hole.net/images/emoji/apple/gift.png?v=5\" title=\":gift:\" class=\"emoji\" alt=\":gift:\"\u003e in your future!\u003c/p\u003e","post_number":1,"post_type":1,"updated_at":"2017-08-20T17:23:23.865Z","reply_count":0,"reply_to_post_number":null,"quote_count":0,"avg_time":null,"incoming_link_count":0,"reads":1,"score":0,"yours":false,"topic_id":4560,"topic_slug":"robot-greetings","topic_title":":robot: Greetings!","display_username":"discobot","primary_group_name":null,"primary_group_flair_url":null,"primary_group_flair_bg_color":null,"primary_group_flair_color":null,"version":1,"user_title":null,"actions_summary":[{"id":2,"can_act":true},{"id":3,"can_act":true},{"id":4,"can_act":true},{"id":5,"hidden":true,"can_act":true},{"id":6,"can_act":true},{"id":7,"can_act":true},{"id":8,"can_act":true}],"moderator":false,"admin":true,"staff":true,"user_id":-2,"hidden":false,"hidden_reason_id":null,"trust_level":4,"deleted_at":null,"user_deleted":false,"edit_reason":null,"can_view_edit_history":true,"wiki":false,"can_accept_answer":false,"can_unaccept_answer":false,"accepted_answer":false,"can_translate":false}}
sent: 2cdcc22ff4e294e11031a13c6e4b126151b85b21eebbe2e79adb96be1e59561d
calc: 682fec0df310ea73fbc7cf85adf1b77e05c477394c7ed6c38caa492d1d352e65
Bad Signature!
08/20/2017 16:58:02
Processing Incoming Hook ID:24915
Event is of Type: post
Exp Content Length: 2333
Act Content Length: 2329
pays: {"post":{"id":16323,"name":"Adam Warner","username":"PromoFaux","avatar_template":"/user_avatar/discourse.pi-hole.net/promofaux/{size}/1545_1.png","created_at":"2017-08-20T16:58:01.020Z","cooked":"\u003caside class=\"quote\" data-post=\"5\" data-topic=\"1874\"\u003e\n\u003cdiv class=\"title\"\u003e\n\u003cdiv class=\"quote-controls\"\u003e\u003c/div\u003e\n\u003cimg alt width=\"20\" height=\"20\" src=\"https://discourse.pi-hole.net/user_avatar/discourse.pi-hole.net/ramset/40/1474_1.png\" class=\"avatar\"\u003e\u003ca href=\"https://discourse.pi-hole.net/t/secondary-dns-server-for-dhcp/1874/5\"\u003eSecondary DNS Server for DHCP\u003c/a\u003e\n\u003c/div\u003e\n\u003cblockquote\u003e\n\u003cp\u003eHere’s an answer (if you’re still looking):\u003c/p\u003e\n\u003cp\u003esudo nano /etc/dnsmasq.d/02-pihole-dhcp.conf\u003c/p\u003e\n\u003cp\u003edhcp-option=6,Pi-holeIP,SecondaryDNSIP\u003c/p\u003e\n\u003cp\u003esudo /etc/init.d/dnsmasq restart\u003c/p\u003e\n\u003cp\u003eYou would have to renew release on each host after that to pull the new settings.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003c/aside\u003e\n\u003cp\u003eCan you explain what this is doing? Thanks\u003c/p\u003e","post_number":23,"post_type":1,"updated_at":"2017-08-20T16:58:01.020Z","reply_count":0,"reply_to_post_number":22,"quote_count":0,"avg_time":null,"incoming_link_count":0,"reads":1,"score":0,"yours":false,"topic_id":4357,"topic_slug":"can-dwight-hear-us-in-here","topic_title":"Can Dwight hear us in here?","display_username":"Adam Warner","primary_group_name":null,"primary_group_flair_url":null,"primary_group_flair_bg_color":null,"primary_group_flair_color":null,"version":1,"user_title":"Developer","reply_to_user":{"username":"PromoFaux","avatar_template":"/user_avatar/discourse.pi-hole.net/promofaux/{size}/1545_1.png"},"actions_summary":[{"id":2,"can_act":true},{"id":3,"can_act":true},{"id":4,"can_act":true},{"id":5,"hidden":true,"can_act":true},{"id":6,"can_act":true},{"id":7,"can_act":true},{"id":8,"can_act":true}],"moderator":true,"admin":true,"staff":true,"user_id":3,"hidden":false,"hidden_reason_id":null,"trust_level":4,"deleted_at":null,"user_deleted":false,"edit_reason":null,"can_view_edit_history":true,"wiki":false,"can_accept_answer":true,"can_unaccept_answer":false,"accepted_answer":false,"can_translate":false}}
sent: acae16a2a0f7d5bb88e2659e399ac59b18b5183b9173d7717ad35de900ddf7dd
calc: 8f59f14a080137d83a4ff3787446c37d17aa47295178b490dcaa936e21eeea4c
Bad Signature!

However, some are fine, like this one:

08/20/2017 16:57:27
Processing Incoming Hook ID:24913
Event is of Type: post
Exp Content Length: 1403
Act Content Length: 1403
pays: {"post":{"id":16322,"name":"Adam Warner","username":"PromoFaux","avatar_template":"/user_avatar/discourse.pi-hole.net/promofaux/{size}/1545_1.png","created_at":"2017-08-20T16:57:26.363Z","cooked":"\u003cp\u003eSomething I have written myself (Pasted)\u003c/p\u003e","post_number":22,"post_type":1,"updated_at":"2017-08-20T16:57:26.363Z","reply_count":0,"reply_to_post_number":21,"quote_count":0,"avg_time":null,"incoming_link_count":0,"reads":1,"score":0,"yours":false,"topic_id":4357,"topic_slug":"can-dwight-hear-us-in-here","topic_title":"Can Dwight hear us in here?","display_username":"Adam Warner","primary_group_name":null,"primary_group_flair_url":null,"primary_group_flair_bg_color":null,"primary_group_flair_color":null,"version":1,"user_title":"Developer","reply_to_user":{"username":"PromoFaux","avatar_template":"/user_avatar/discourse.pi-hole.net/promofaux/{size}/1545_1.png"},"actions_summary":[{"id":2,"can_act":true},{"id":3,"can_act":true},{"id":4,"can_act":true},{"id":5,"hidden":true,"can_act":true},{"id":6,"can_act":true},{"id":7,"can_act":true},{"id":8,"can_act":true}],"moderator":true,"admin":true,"staff":true,"user_id":3,"hidden":false,"hidden_reason_id":null,"trust_level":4,"deleted_at":null,"user_deleted":false,"edit_reason":null,"can_view_edit_history":true,"wiki":false,"can_accept_answer":true,"can_unaccept_answer":false,"accepted_answer":false,"can_translate":false}}
sent: 7dba5d12657cedad5c371daedd140d0d51ce702db4f7087da3a4c38b129f105b
calc: 7dba5d12657cedad5c371daedd140d0d51ce702db4f7087da3a4c38b129f105b

I’ve tested this on both a .NET, and a Python receiver, and am seeing the same issues with Content-Length being 4-6 bytes out.

Any ideas?

Thanks in advance!


(Eli the Bearded) #2

Good report. From the length count differences and the actual payloads, it looks to me like Discourse is returning character count for length instead of octet count. The differences come from the number of UTF-8 characters in the payload.


(Adam Warner) #3

I wouldn’t be so sure! :slight_smile:


(Eli the Bearded) #4

Well, I could have it wrong. Discourse could be returning the right number, and you are counting wrong in Python. (That actually makes more sense, since you claim to see smaller numbers.)

Edit: I pasted a couple of those messages into vim, with “set ruler” turned on. The ruler info shows both byte and character counts, and the two numbers match your expected / actual results.


(Adam Warner) #5

For sure, it’s more than possible my code is wrong!

I’m actually upgrading an old bot (as an excuse to learn a bit more about .Net Core) we are using that is a simple Python script which currently doesn’t check content-length or signatures.

Code that is receiving the “wrong” values:

        [HttpPost("")]
        public async Task<IActionResult> Receive()
        {
            Console.WriteLine("");
            Request.Headers.TryGetValue("X-Discourse-Event-Id", out StringValues eventId);
            Request.Headers.TryGetValue("X-Discourse-Event-Type", out StringValues eventType);
            Request.Headers.TryGetValue("X-Discourse-Event", out StringValues eventName);
            Request.Headers.TryGetValue("X-Discourse-Event-Signature", out StringValues signature);
            Request.Headers.TryGetValue("Content-Length", out StringValues contLength);
            Request.Headers.TryGetValue("Content-Type", out StringValues contType);

            Console.WriteLine(DateTime.Now);
            Console.WriteLine($"Processing Incoming Hook ID:{eventId}");
            Console.WriteLine($"Event is of Type: {eventType}");

            Console.WriteLine($"Content Type: {contType}");
            Console.WriteLine($"Exp Content Length: {contLength}");

            using (var reader = new StreamReader(Request.Body))
            {
                var txt = await reader.ReadToEndAsync();
                
                Console.WriteLine($"Act Content Length: {txt.Length}");

                return ProcessRequest(txt, eventName, signature);
            }

        }

Have also been playing with manually counting the payload lengths in ultraedit, too :slight_smile:


(Eli the Bearded) #6

I don’t know Python well enough to know if that is the right way to do that. I know in Ruby there are different methods for character / byte size.

To pick the UTF-8 bit out of the first one:

irb(main):001:0> u='"unicode_title":"🤖 Greetings!"'
=> "\"unicode_title\":\"🤖 Greetings!\""
irb(main):002:0> u.size
=> 30
irb(main):003:0> u.bytesize
=> 33
irb(main):004:0> 

(Adam Warner) #7

Apologies, perhaps I wasn’t clear. The new code is C#, not Python.

I will play about a bit and see if I can come up with something more solid.


(Adam Warner) #8

Oh man. Talk about the answer staring me in the face all along…

Spot the deliberate mistake…

var secret = Encoding.ASCII.GetBytes(_config.Value.DiscourseWebhookSecret);
var payloadBytes = Encoding.ASCII.GetBytes(payload);

Good old rubber duck debugging. Thanks @elijah , and sorry for wasting your time :slight_smile:


(Alan Tan) #9