# Why there are lots of Disallow rule in robots.txt?

**URL:** https://meta.discourse.org/t/why-there-are-lots-of-disallow-rule-in-robots-txt/84251
**Category:** Support
**Created:** [30.Март.2018 19:23:27 UTC](https://meta.discourse.org/t/why-there-are-lots-of-disallow-rule-in-robots-txt/84251 "2018-03-30T19:23:27Z")
**Posts on this page:** 1
**Showing post:** 7

<div class="post-metadata">

### Author: ![Gulshan\_Kumar](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/gulshan_kumar/32/119562_2.png) [@Gulshan\_Kumar](https://meta.discourse.org/u/Gulshan_Kumar)
#### Post date: [30.Март.2018 20:48:41 UTC](https://meta.discourse.org/t/why-there-are-lots-of-disallow-rule-in-robots-txt/84251/7 "2018-03-30T20:48:41Z")

</div>

> [@notriddle](#):
>
> The reason they’re blocked in robots.txt is because the page contains text and links that are likely to be misleading.

What if we …

Step 1. Allow crawling path /u/  
Step 2. set noindex, follow header for path /u/  
Step 3. Restrict profile access to logged-in users  
 ![image](https://global.discourse-cdn.com/meta/original/3X/3/3/33298d1a9459181b7b126aea3659d8ee5db1f875.png)

_If something is restricted to logged in user means bot cannot access it. Hence, it is already prevented from duplicate text concerns. Still, why disallow for path /u/?_

---

_[View the full topic](https://meta.discourse.org/t/why-there-are-lots-of-disallow-rule-in-robots-txt/84251)._