# Why there are lots of Disallow rule in robots.txt?
**URL:** https://meta.discourse.org/t/why-there-are-lots-of-disallow-rule-in-robots-txt/84251
**Category:** Support
**Created:** [3월 30, 2018, 7:23오후 UTC](https://meta.discourse.org/t/why-there-are-lots-of-disallow-rule-in-robots-txt/84251 "2018-03-30T19:23:27Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@Gulshan\_Kumar](https://meta.discourse.org/u/Gulshan_Kumar)
#### Post date: [3월 30, 2018, 7:23오후 UTC](https://meta.discourse.org/t/why-there-are-lots-of-disallow-rule-in-robots-txt/84251/1 "2018-03-30T19:23:27Z")
Does it prevent indexing in the SERPs?
- Never.
**Why does this happen?**
**What best can be done?**
Please allow editing of the robots.txt file, we will do our best. 😉
Thanks
---
### Author:  [@notriddle](https://meta.discourse.org/u/notriddle)
#### Post date: [3월 30, 2018, 7:34오후 UTC](https://meta.discourse.org/t/why-there-are-lots-of-disallow-rule-in-robots-txt/84251/2 "2018-03-30T19:34:51Z")
```plaintext
Disallow: /auth/cas
Disallow: /auth/facebook/callback
Disallow: /auth/twitter/callback
Disallow: /auth/google/callback
Disallow: /auth/yahoo/callback
Disallow: /auth/github/callback
Disallow: /auth/cas/callback
```
These pages don’t work unless you are logged in, so it makes no sense for spiders to crawl them. They’ll just get an error if it tries.
```plaintext
Disallow: /assets/browser-update*.js
```
I think this is working around a bad habit Firefox has of overeagerly re-downloading Web Workers.
```plaintext
Disallow: /users/
Disallow: /u/
```
Because they’re infrequently read by humans, [black hat SEO](https://en.wikipedia.org/wiki/Search_engine_optimization) people like to spam forum user profiles. Discourse blocks user profiles in order to disincentivise it.
Also, user profiles contain excerpts of the posts, but we don’t want the search engines linking there. They should be linking to the _actual posts_.
```plaintext
Disallow: /badges/
Disallow: /search
Disallow: /search/
Disallow: /tags
Disallow: /tags/
```
These pages don’t have a noscript version, so they just get empty responses. Try visiting [https://meta.discourse.org/badges/?\_escaped\_fragment\_=1](https://meta.discourse.org/badges/?_escaped_fragment_=1)
```plaintext
Disallow: /email/
```
I dunno. Does it even exist any more?
```plaintext
Disallow: /session
Disallow: /session/
Disallow: /admin
Disallow: /admin/
Disallow: /user-api-key
Disallow: /user-api-key/
Disallow: /*?api_key*
Disallow: /*?*api_key*
```
Again, these don’t work without being logged in anyway, and spiders can’t log in.
```plaintext
Disallow: /groups
Disallow: /groups/
```
Again, need javascript to view these pages.
```plaintext
Disallow: /uploads/
```
Don’t want spiders shoveling down huge images, or ending up hotlinked because somebody found it in an image search. This rule stops working if you turn on a CDN, by the way.
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [3월 30, 2018, 8:31오후 UTC](https://meta.discourse.org/t/why-there-are-lots-of-disallow-rule-in-robots-txt/84251/3 "2018-03-30T20:31:22Z")
> [@notriddle](#):
>
> Because they’re infrequently read by humans, black hat SEO people like to spam forum user profiles. Discourse blocks user profiles in order to disincentivise it.
That’s not the only reason; the user profiles are all duplicate data, in that your posts and topics are visible on.. the topics themselves, which _are_ the focus.
---
### Author:  [@Gulshan\_Kumar](https://meta.discourse.org/u/Gulshan_Kumar)
#### Post date: [3월 30, 2018, 8:38오후 UTC](https://meta.discourse.org/t/why-there-are-lots-of-disallow-rule-in-robots-txt/84251/4 "2018-03-30T20:38:35Z")
Maybe we could replace this way for some paths
- First, Allow Bots to see the link
`Allow: /u/`
- Then, set `X-Robots-Tag "noindex, follow"`
**Why?** This will actually prevent indexing.
Ref: \_[Robots Meta Tags Specifications | Google Search Central | Documentation | Google for Developers](https://developers.google.com/search/reference/robots_meta_tag#using-the-x-robots-tag-http-header_)
---
### Author:  [@notriddle](https://meta.discourse.org/u/notriddle)
#### Post date: [3월 30, 2018, 8:40오후 UTC](https://meta.discourse.org/t/why-there-are-lots-of-disallow-rule-in-robots-txt/84251/5 "2018-03-30T20:40:36Z")
Indexing the page isn’t a problem. If Bing thinks I’m looking for your user profile, and offers up a link to it, that’s perfectly fine. The reason they’re blocked in robots.txt is because the page contains text and links that are likely to be misleading.
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [3월 30, 2018, 8:41오후 UTC](https://meta.discourse.org/t/why-there-are-lots-of-disallow-rule-in-robots-txt/84251/6 "2018-03-30T20:41:35Z")
> [@notriddle](#):
>
> text and links that are likely to be misleading.
“duplicates” would be a much more accurate statement here.
---
### Author:  [@Gulshan\_Kumar](https://meta.discourse.org/u/Gulshan_Kumar)
#### Post date: [3월 30, 2018, 8:48오후 UTC](https://meta.discourse.org/t/why-there-are-lots-of-disallow-rule-in-robots-txt/84251/7 "2018-03-30T20:48:41Z")
> [@notriddle](#):
>
> The reason they’re blocked in robots.txt is because the page contains text and links that are likely to be misleading.
What if we …
Step 1. Allow crawling path /u/
Step 2. set noindex, follow header for path /u/
Step 3. Restrict profile access to logged-in users
_If something is restricted to logged in user means bot cannot access it. Hence, it is already prevented from duplicate text concerns. Still, why disallow for path /u/?_
---
### Author:  [@notriddle](https://meta.discourse.org/u/notriddle)
#### Post date: [3월 30, 2018, 8:52오후 UTC](https://meta.discourse.org/t/why-there-are-lots-of-disallow-rule-in-robots-txt/84251/8 "2018-03-30T20:52:29Z")
Why do you think it would be better to use a noindex header instead of robots.txt? The goal of blocking search engines is not to prevent them from returning user profiles as search results, it’s to prevent them from reading the contents of the user profiles (because the contents of the user profiles are duplicates of other pages and/or spam).
---
### Author:  [@Gulshan\_Kumar](https://meta.discourse.org/u/Gulshan_Kumar)
#### Post date: [3월 30, 2018, 8:56오후 UTC](https://meta.discourse.org/t/why-there-are-lots-of-disallow-rule-in-robots-txt/84251/9 "2018-03-30T20:56:40Z")
Because that is a right way to preventing indexing.
---
### Author:  [@Gulshan\_Kumar](https://meta.discourse.org/u/Gulshan_Kumar)
#### Post date: [3월 30, 2018, 9:06오후 UTC](https://meta.discourse.org/t/why-there-are-lots-of-disallow-rule-in-robots-txt/84251/10 "2018-03-30T21:06:11Z")
Because Google doesn’t recommend using disallow rule to handle duplicate content.
> **[How to Specify a Canonical with rel="canonical" and Other Methods | Google...](https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls?hl=en&visit_id=639158894245036236-3540454255&rd=1)**
>
> When a site has duplicate content, Google chooses the canonical URL. Learn more about canonical URLs and how to consolidate duplicate URLs.
But, two-three times they have mentioned about `noindex`.
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [3월 30, 2018, 9:18오후 UTC](https://meta.discourse.org/t/why-there-are-lots-of-disallow-rule-in-robots-txt/84251/11 "2018-03-30T21:18:20Z")
Why are you posting screenshots of text? Guess what is also very bad for searchability? Pictures of text..
So your point seems to be this – which I had to TYPE IN FROM YOUR SCREENSHOT instead of being linked like a regular link:
> **[Block Search Indexing with noindex | Google Search Central | Documentation...](https://developers.google.com/search/docs/crawling-indexing/block-indexing?visit_id=639159130610086883-2290876708&rd=1)**
>
> A noindex tag can block Google from indexing a page so that it won't appear in Search results. Learn how to implement noindex tags with this guide.
> You can prevent a page from appearing in Google Search by including a `noindex` meta tag in the page’s HTML code, or by returning a ‘noindex’ header in the HTTP request. When Googlebot next crawls that page and see the tag or header, Googlebot will drop that page entirely from Google Search results, regardless of whether other sites link to it.
>
> > ⚠ Important! For the `noindex` directive to be effective, the page **must not** be blocked by a robots.txt file. If the page is blocked by a robots.txt file, the crawler will never see the `noindex` directive, and the page can still appear in search results, for example if other pages link to it.
Which means user pages are still kinda present in Google’s indexes though **they would never appear as hits for any actual search terms**.
How is this a problem? Give me a valid search term with actual search keywords that produces a user page.
---
### Author:  [@Gulshan\_Kumar](https://meta.discourse.org/u/Gulshan_Kumar)
#### Post date: [3월 30, 2018, 9:32오후 UTC](https://meta.discourse.org/t/why-there-are-lots-of-disallow-rule-in-robots-txt/84251/13 "2018-03-30T21:32:44Z")
The first result, just for searching homepage URL.
> **[help.gulshankumar.net - Bing](https://www.bing.com/search?q=help.gulshankumar.net)**
>
> Intelligent search from Bing makes it easier to quickly find what you’re looking for and rewards you.
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [3월 30, 2018, 9:33오후 UTC](https://meta.discourse.org/t/why-there-are-lots-of-disallow-rule-in-robots-txt/84251/14 "2018-03-30T21:33:45Z")
No repro here on meta
---
### Author:  [@Gulshan\_Kumar](https://meta.discourse.org/u/Gulshan_Kumar)
#### Post date: [3월 30, 2018, 9:35오후 UTC](https://meta.discourse.org/t/why-there-are-lots-of-disallow-rule-in-robots-txt/84251/15 "2018-03-30T21:35:17Z")
I appreciate your case-study.
However, In my case, it’s different. If there were `noindex` tag, my case might be different.
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [3월 30, 2018, 9:36오후 UTC](https://meta.discourse.org/t/why-there-are-lots-of-disallow-rule-in-robots-txt/84251/16 "2018-03-30T21:36:33Z")
I’ll ask again:
> [@codinghorror](#):
>
> Give me a valid search term with actual search keywords that produces a user page.
I do not consider “type the full domain name into a search box” a valid search.
---
### Author:  [@Gulshan\_Kumar](https://meta.discourse.org/u/Gulshan_Kumar)
#### Post date: [3월 30, 2018, 9:39오후 UTC](https://meta.discourse.org/t/why-there-are-lots-of-disallow-rule-in-robots-txt/84251/17 "2018-03-30T21:39:23Z")
> [@Gulshan\_Kumar](#):
>
> [help.gulshankumar.net - Search](https://www.bing.com/search?q=help.gulshankumar.net)
Another one, please consider now.
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [3월 30, 2018, 9:39오후 UTC](https://meta.discourse.org/t/why-there-are-lots-of-disallow-rule-in-robots-txt/84251/18 "2018-03-30T21:39:50Z")
In Google – Bing is terrible, and has almost no market share.
---
### Author:  [@notriddle](https://meta.discourse.org/u/notriddle)
#### Post date: [3월 30, 2018, 9:41오후 UTC](https://meta.discourse.org/t/why-there-are-lots-of-disallow-rule-in-robots-txt/84251/19 "2018-03-30T21:41:50Z")
Also, you just searched for your own login name. Returning the user profile seems like a correct result to me.
---
### Author:  [@Gulshan\_Kumar](https://meta.discourse.org/u/Gulshan_Kumar)
#### Post date: [3월 30, 2018, 9:44오후 UTC](https://meta.discourse.org/t/why-there-are-lots-of-disallow-rule-in-robots-txt/84251/20 "2018-03-30T21:44:14Z")
Unfortunately, the Search Engine Bing powers many other small SE. So, it reflects same results there. I learned this thing from Bing, after keeping it noindex for a year. I found, other SE also stopped showing results because I had done noindex at Bing.
Maybe, Bing has a small market share, but its impact on small SE ☹
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [3월 30, 2018, 9:45오후 UTC](https://meta.discourse.org/t/why-there-are-lots-of-disallow-rule-in-robots-txt/84251/21 "2018-03-30T21:45:03Z")
If you feel strongly about it, you can write a plugin to change the behavior. I see no value in spending any effort on this, other than you typing your name into Bing.
[다음 페이지](https://meta.discourse.org/t/why-there-are-lots-of-disallow-rule-in-robots-txt/84251.md?page=2)