Why there are lots of Disallow rule in robots.txt?

[sam (Sam Saffron)]

Well bing has a bug and is not respecting robots txt, why not raise this with Microsoft

Note, I am sure some real weird things are up with bing, looking at crawler stats here it’s going ballistic on meta

[notriddle (Michael Howell)]

Actually, it is respecting robots.txt. It is not crawling the pages that are denied. It is including the page in its index, but the robots.txt spec allows you to do that just fine.

[codinghorror (Jeff Atwood)]

@sam

some real weird things are up with bing, looking at crawler stats here it’s going ballistic on meta

I refer you to my previous statements on the matter:

@codinghorror

Bing is terrible

We saw this at Stack Overflow as well. A lot of sound and fury from bing crawlers resulting in virtually zero real traffic. Bing completely sucks, and has for a decade.

[Gulshan_Kumar]

@sam

Well bing has a bug and is not respecting robots txt, why not raise this with Microsoft

The disallow rule at robots.txt file is to prevent from crawling, never from indexing.
Raising the issue with MS? Seriously, they don’t care about webmasters.

[sam (Sam Saffron)]

I am against removing this from robots, but somehow amending discourse to carry the list and auto generate don’t index meta tags as well on those pages is something I am open to, especially if this reduces traffic from bing

[codinghorror (Jeff Atwood)]

@sam

especially if this reduces traffic from bing

Worth a try, at least that is a net benefit to every Discourse instance.

[notriddle (Michael Howell)]

Personally, @Gulshan_Kumar, I recommend:

1. Work on getting people to link to your forum’s front page from Twitter and stuff. Unindexed pages should not be ranking higher than your front page, even on Bing.

2. Don’t worry about Bing returning your user profile when searching for your login name. That’s correct behavior.

And as for @codinghorror: don’t act like Stack Overflow is representative of the internet at large. Bing, being the default search engine in Internet Explorer, gets most of its traffic from a very different demographic than SO or even Discourse Meta targets.

[codinghorror (Jeff Atwood)]

Bing search market share by country 2017| Statista

Worldwide, 9%. My main beef is that Bing is objectively very bad. Both at providing relevant results, and the crawler behavior.

I mean for a criteria of “type words in a search box and have something come back”, it works..

[Gulshan_Kumar]

About Bing, I think every bit helps. If a website is quality enough, at least for some keywords it will rank.
My purpose to keep the noindex is to avoid indexing unnecessary pages. The way Bing/Yahoo randomly show stuff, I feel helpless.

Thanks for participating in this discussion. I greatly appreciate your valuable input.

@notriddle

Also, user profiles contain excerpts of the posts, but we don’t want the search engines linking there. They should be linking to the actual posts.

Sorry, I forget to mention. How if we just remove the user-profile link for non-logged in users (anonymous/bot)? This will solve all problem.

[notriddle (Michael Howell)]

Because they’re not supposed to be secret.

[riking (Kane York)]

Also note that the crawler view user profile… doesn’t actually contain anything other than the bio. Certainly not a list of links to posts.

Now that the user page has been redefined a few times, presenting these links to crawlers might actually be a good thing - surfacing quality content is what those sections are for, and we might as well help the search engine out with them.

@sam’s profile Overview

versus noscript version of me:

      <div id="main-outlet" class="wrap">
        <!-- preload-content: -->
         <h2>riking</h2>

<p><p>Discourse is pretty great</p>
<p><a href="https://github.com/riking" class="onebox" target="_blank">https://github.com/riking</a><br>
<a href="https://twitter.com/riking27" class="onebox" target="_blank">https://twitter.com/riking27</a></p></p>



        <!-- :preload-content -->
        <footer>
          <nav itemscope itemtype='http://schema.org/SiteNavigationElement'>
            <a href='/'>Home</a>
            <a href="/categories">Categories</a>
            <a href="/guidelines">FAQ/Guidelines</a>
            <a href="/tos">Terms of Service</a>
            <a href="/privacy">Privacy Policy</a>
          </nav>
        </footer>
      </div>

      <footer id='noscript-footer'>
        <p>Powered by <a href="https://www.discourse.org">Discourse</a>, best viewed with JavaScript enabled</p>
      </footer>

wow, that’s out of date…

[codinghorror (Jeff Atwood)]

See also

U.S. search engine market share queries handled 2025| Statista

Google has a massive lead on mobile as well, so that transition heavily favors them.

[markersocial]

@notriddle

Because they’re infrequently read by humans, black hat SEO people like to spam forum user profiles. Discourse blocks user profiles in order to disincentivise it.

@notriddle - I think profiles are quite frequently read by humans, forum profiles will often have more views than any one of the user’s threads. If blocking user profiles from indexing is to disincentivize spammers from placing links, it means that we’re likely in agreement that links listed inside a profile will be valued more if the page is indexed, as opposed to not indexed.

This would mean that our internal linking will suffer, due to being on noindex profile pages. More strong internal linking is better for SEO and helps search engines crawl content better. Especially when no sitemap is being used. Simply adding nofollow to external spam/non-whitelisted profile links should be sufficient to disincentivize spammers (if it isn’t done already), many probably don’t even check the robots.txt to see if the profiles get indexed.

Here is a thread incl. video discussing how Google will not follow links on noindex pages: Google Will Eventually Stop Following Links on Noindex Pages - Google Search and SEO forum at WebmasterWorld - WebmasterWorld

@codinghorror

That’s not the only reason; the user profiles are all duplicate data, in that your posts and topics are visible on… the topics themselves, which are the focus.

@codinghorror - As for it being duplicate content, I think profiles are less duplicate content than a thread being inside parent and child categories (incl. tags) simultaneously. The links are duplicates, but on different pages/urls, with different purposes. Profiles can include unique content also like bio, the floating card that also displays the bio displays on desktop only, to view it on mobile you can only go to their full profile. Google has switched to mobile first indexing also, meaning the mobile versions of our sites have become the primary versions of our sites: How Does Mobile-First Indexing Work, and How Does It Impact SEO? - Moz

The question is, why wouldn’t we want to allow search engines to have better crawling on our sites? Reddit allows indexing of user profiles and is basically the largest forum in the world. Youtube allows indexing of channels, Twiiter, FB, Google Plus etc. allow indexing of profiles as well. The only main examples I’ve seen of using noindex for profiles is on old forum software.

I definitely think the noindex for user profile pages in the robots.txt should not be default.

[sam (Sam Saffron)]

다시 이 주제를 꺼냅니다.

1. 원하신다면 이제 robot.txt 파일을 자유롭게 수정할 수 있습니다.

2. 인덱싱되지 않아야 하는 페이지에는 항상 x-robots-tag noindex를 적용하고 있습니다.

3. robots.txt에서 엄격한 지침을 제공하지 않으면 일부 크롤러가 사이트를 과도하게 파고드는 것으로 나타났습니다. 모든 크롤러가 구글처럼 행동하지는 않으니까요. 최근에는 매우 단순한 robots.txt 파일을 사용하고 있지만, 이로 인해 상당한 비용이 발생하고 있습니다. (모든 크롤러가 구글처럼 예의 바르게 행동하기를 기대하지만, 구글처럼 되는 데는 엄청난 노력이 필요합니다.)

모든 구글봇을 제외한 크롤러에 대해서는 기본적으로 “엄격히 제한하는” robots.txt를 다시 도입하는 것이 좋다고 생각합니다.

[codinghorror (Jeff Atwood)]

@sam

기본적으로 모든 구글봇이 아닌 봇에 대해서는 “매우 제한적인” robots.txt를 다시 적용하는 것이 좋다고 생각합니다.

네, 그렇게 하셔도 좋습니다. 구글이 우리를 이런 상황에 몰아넣었거든요.