# 处理 Bingbot
**URL:** https://meta.discourse.org/t/handling-bingbot/84659
**Category:** Feature
**Created:** [2018年四月5日 17:51 UTC](https://meta.discourse.org/t/handling-bingbot/84659 "2018-04-05T17:51:37Z")
**Posts on this page:** 10
**Page:** 2
### Author:  [@dolkensp](https://meta.discourse.org/u/dolkensp)
#### Post date: [2018年四月9日 11:29 UTC](https://meta.discourse.org/t/handling-bingbot/84659/22 "2018-04-09T11:29:05Z")
> [@sam](#):
>
> 1. Kick up a giant political fuss hoping that Microsoft will listen and correct the bad crawling algorithm. We have some weight here cause we power a lot of sites and people do not usually touch defaults.
>
> 2. Make Discourse somehow friendlier to crawlers so crawlers can do less work to figure out the content.
>
> 3. Add Crawl-delay to stop the bleeding (which leaves some big open questions)
>
> 4. Use an esoteric solution here like rate limiting by user agent, pick your poison of choice.
>
> 5. Do nothing.
Hi Sam,
Options 1, 2, 3 all seem like good options to me, though I feel you could have done #1, without blocking Bing. That just seems like the “lazy” option, as 2 & 3 would take more time to implement.
@codinghorror acknowledged the monopoly that exists in the search space, and you both acknowledge that Discourse actually has some weight to throw around. I would expect more then, from an organisation with weight to throw around, than to effectively muscle out a smaller player in the market due to behavior that you have deemed “bad”.
I agree, Bing may use a crawl profile which may not be ideal. It may even be categorically bad. But you have still chosen to muscle out the smaller player, who offers and obeys fairly well documented extensions to the robots.txt standards.
Like you said, many users don’t touch defaults - so how many users are about to deploy Discourse, only to find that they _never_ get indexed on Bing, due to those defaults. How many users are going to be harmed, should Bing establish greater market penetration in a local market. What happens when “Cortana” suddenly becomes the killer voice assistant? Microsoft already sits on 89% of desktops worldwide, if they play their cards right, it _could_ happen. Essentially, you’ve burdened your users with risk, simply because you have the weight to do so, to stomp your feet, and get your way.
So please, do #1, #2, and #3. Kick up a fuss on Twitter. Message the Bing team. Contact your Microsoft representatives. It’s not like the names “Sam Saffron” and “Jeff Atwood” are so unknown that you’re not going to get a response from direct contact, either via email, or via Twitter. But don’t just blanket block a smaller player in a market, because they’re not doing what you would like.
Indeed, you could even have blocked Bing due to it’s low market share. It’s not that unusual for companies to exclude support for various low-impact platforms (and at the end of the day, you still must have the best interests of your own company at heart). It is rather unusual to _boast_ about it though.
---
### Author:  [@dolkensp](https://meta.discourse.org/u/dolkensp)
#### Post date: [2018年四月9日 11:37 UTC](https://meta.discourse.org/t/handling-bingbot/84659/23 "2018-04-09T11:37:58Z")
> [@codinghorror](#):
>
> It’s trivial to change this default; just edit one setting in your site settings, takes all of 15 seconds.
but …
> [@sam](#):
>
> We have some weight here cause we power a lot of sites and people do not usually touch defaults.
So which one is it - defaults are trivial - or people don’t touch them 😉
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [2018年四月9日 11:56 UTC](https://meta.discourse.org/t/handling-bingbot/84659/24 "2018-04-09T11:56:21Z")
Just putting this out here, while discussing this with Microsoft (the engineers who build Bing and PM for Bing)
- Microsoft never told me that this blocking will cause irreparable damage to sites ranking
- Microsoft suggested a site map as a workaround
- Microsoft did not make an explicit recommendation to use Crawl delay vs blocking
- Microsoft said they want to fix the underlying issue
We are testing the site map theory.
My gut feel on this is that crawl delay vs straight out telling bot not to crawl will have almost the same effect if this only goes on for a month or so. long term crawl delay is not a proper solution cause the backlog they have is too huge and they can not work through all the urls they want to.
We are only telling Bing not to crawl, we say nothing on the pages asking Bing or any crawler not to index using meta tags
So, basically what we have here is a bunch of people who have a gut feel that due to SEO reasons blocking a crawler for N days via robots.txt while we work this out with Bing will damage sites forever in Bing. My gut feel is that the old content will remain in Bing for at least a few months if not more.
Now, if Microsoft tell me that what we are doing will have irreparable damage going forward, I think it would have more weight.
If anything brand new sites deploying Discourse at the moment have more of a chance finding this out that we messed with Bing and find this topic. Crawl delay would mask that.
---
### Author:  [@j127](https://meta.discourse.org/u/j127)
#### Post date: [2018年四月11日 02:25 UTC](https://meta.discourse.org/t/handling-bingbot/84659/25 "2018-04-11T02:25:02Z")
> [@sam](#):
>
> people who have a gut feel that due to SEO reasons blocking a crawler for N days via robots.txt while we work this out with Bing will damage sites forever in Bing.
It isn’t a completely random gut feeling – about 10 years ago part of my job was fixing large sites so that they could be crawled more easily (which can make a big difference in indexing and ranking – sometimes sites even get accidentally banned in search results due to crawling issues).
I’ve seen it happen with Google, and it was just an accidental `Disallow: /` for a few days.
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [2018年四月11日 02:31 UTC](https://meta.discourse.org/t/handling-bingbot/84659/26 "2018-04-11T02:31:54Z")
> [@j127](#):
>
> It isn’t a completely random gut feeling – about 10 years ago part of my job was fixing large sites so that they could be crawled more easily (which can make a big difference in indexing and ranking – sometimes sites even get accidentally banned in search results due to crawling issues).
>
> I’ve seen it happen with Google, and it was just an accidental Disallow: / for a few days.
My anecdotal evidence here is that this does not happen anymore, we had Google blocked from crawling meta for 20 days a while back due to a bug in our robots file, the blocking made zero difference long term.
Also, we are talking about Bing here not Google, so evidence about what Google did 10 years ago is not particularly strong when we are talking about Bing.
Besides
- The Microsoft PM for Bing is probably reading this topic
- Multiple Engineers on the Bing team are reading this topic
None of them said
Sam stop, if you block us for a few weeks you are going to destroy everything in our index for your Discourse sites. What they seem to be saying is that stuff needs fixing and they want to get to a state where we are comfortable unblocking Bing again.
We are running multiple experiments at the moment. Microsoft are working on the problem. In the meantime while this is happening “bleeding by default” is reduced to zero.
---
### Author:  [@Johan\_Hulsen](https://meta.discourse.org/u/Johan_Hulsen)
#### Post date: [2018年四月11日 04:02 UTC](https://meta.discourse.org/t/handling-bingbot/84659/27 "2018-04-11T04:02:49Z")
From my experience Bingbot always needs a little bit more help when it comes to crawling, especially caching crawled results.
Bingbot for example ignores E-Tag / I’ve never seen Bingbot asking for E-Tags.
HTTP-Headers of discourse explicitly tell NOT to cache the results (“cache-control: no-store, must-revalidate, no-cache, private”)
Google tries to apply heuristics to ignore those headers if they don’t feel they are helpful.
(Doesn’t mean that it works all the time. Have seen Google crawling a random 404 of a product page on a shop more than once every minute for months)
Bingbot sticks more with what webmasters tell them. So if webmasters tell them to ignore the crawled page they will crawl again.
---
### Author:  [@riking](https://meta.discourse.org/u/riking)
#### Post date: [2018年四月14日 00:33 UTC](https://meta.discourse.org/t/handling-bingbot/84659/28 "2018-04-14T00:33:58Z")
Update: Bing will now be heavily throttled - one request per 60 seconds - while this is being worked on.
[https://github.com/discourse/discourse/commit/6179c0ce51bc1d9d814a1baae354d68eb491e9fd](https://github.com/discourse/discourse/commit/6179c0ce51bc1d9d814a1baae354d68eb491e9fd)
There has also been an update to the original #feature:announcements topic.
---
### Author:  [@schungx](https://meta.discourse.org/u/schungx)
#### Post date: [2018年四月14日 10:48 UTC](https://meta.discourse.org/t/handling-bingbot/84659/29 "2018-04-14T10:48:23Z")
Well, Bing is my default search engine… Surprise! When I’m in China.
That’s because it is just about then only global search engine that works in China. Google, Yahoo, etc. All blocked.
If you want china sites then Baidu is fine. But if you want non Chinese non China sites with info that is not five years and two generations out of date you really need Bing…
---
### Author:  [@anon23393886](https://meta.discourse.org/u/anon23393886)
#### Post date: [2020年十一月20日 16:51 UTC](https://meta.discourse.org/t/handling-bingbot/84659/30 "2020-11-20T16:51:47Z")
> [@schungx](#):
>
> 如果你想要中国网站，百度就足够了。但如果你想要非中国地区的非中文网站，且信息不是过时五年、隔了两代人的内容，那你真的需要必应……
不必再担心了！两年后，问题已经解决。
> [@Bingbot is (no longer) default throttled](https://meta.discourse.org/t/bingbot-is-default-throttled/84620/25):
>
> Fabrice, we tested this on a few sites and crawling behavior appears a lot more reasonable. After this is merged the default throttle will be removed. [https://github.com/discourse/discourse/pull/11035](https://github.com/discourse/discourse/pull/11035)
我很高兴这件事终于有了结果。因此，我认为最合适的做法是关闭此话题，或将其与我上面链接的话题合并。
---
### Author:  [@Falco](https://meta.discourse.org/u/Falco)
#### Post date: [2020年十一月20日 17:24 UTC](https://meta.discourse.org/t/handling-bingbot/84659/31 "2020-11-20T17:24:25Z")
[上一頁](https://meta.discourse.org/t/handling-bingbot/84659.md?page=1)