# Handling Bingbot

**URL:** <https://meta.discourse.org/t/handling-bingbot/84659>\
**Category:** Feature\
**Created:** [April 5, 2018, 5:51pm UTC](https://meta.discourse.org/t/handling-bingbot/84659 "2018-04-05T17:51:37Z")\
**Posts on this page:** 1\
**Showing post:** 22

<div class="post-metadata">

**Author:** ![dolkensp](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/dolkensp/32/94017_2.png) [@dolkensp](https://meta.discourse.org/u/dolkensp)\
**Post date:** [April 9, 2018, 11:29am UTC](https://meta.discourse.org/t/handling-bingbot/84659/22 "2018-04-09T11:29:05Z")

</div>

> [@sam](#):
>
> 1. Kick up a giant political fuss hoping that Microsoft will listen and correct the bad crawling algorithm. We have some weight here cause we power a lot of sites and people do not usually touch defaults.
> 
> 2. Make Discourse somehow friendlier to crawlers so crawlers can do less work to figure out the content.
> 
> 3. Add Crawl-delay to stop the bleeding (which leaves some big open questions)
> 
> 4. Use an esoteric solution here like rate limiting by user agent, pick your poison of choice.
> 
> 5. Do nothing.

Hi Sam,

Options 1, 2, 3 all seem like good options to me, though I feel you could have done #1, without blocking Bing. That just seems like the “lazy” option, as 2 & 3 would take more time to implement.

@codinghorror acknowledged the monopoly that exists in the search space, and you both acknowledge that Discourse actually has some weight to throw around. I would expect more then, from an organisation with weight to throw around, than to effectively muscle out a smaller player in the market due to behavior that you have deemed “bad”.

I agree, Bing may use a crawl profile which may not be ideal. It may even be categorically bad. But you have still chosen to muscle out the smaller player, who offers and obeys fairly well documented extensions to the robots.txt standards.

Like you said, many users don’t touch defaults - so how many users are about to deploy Discourse, only to find that they _never_ get indexed on Bing, due to those defaults. How many users are going to be harmed, should Bing establish greater market penetration in a local market. What happens when “Cortana” suddenly becomes the killer voice assistant? Microsoft already sits on 89% of desktops worldwide, if they play their cards right, it _could_ happen. Essentially, you’ve burdened your users with risk, simply because you have the weight to do so, to stomp your feet, and get your way.

So please, do #1, #2, and #3. Kick up a fuss on Twitter. Message the Bing team. Contact your Microsoft representatives. It’s not like the names “Sam Saffron” and “Jeff Atwood” are so unknown that you’re not going to get a response from direct contact, either via email, or via Twitter. But don’t just blanket block a smaller player in a market, because they’re not doing what you would like.

Indeed, you could even have blocked Bing due to it’s low market share. It’s not that unusual for companies to exclude support for various low-impact platforms (and at the end of the day, you still must have the best interests of your own company at heart). It is rather unusual to _boast_ about it though.

---

_[View the full topic](https://meta.discourse.org/t/handling-bingbot/84659)._