# How to edit robot.txt for indexing user profiles?

**URL:** https://meta.discourse.org/t/how-to-edit-robot-txt-for-indexing-user-profiles/113644
**Category:** Support
**Created:** [April 6, 2019, 9:43pm UTC](https://meta.discourse.org/t/how-to-edit-robot-txt-for-indexing-user-profiles/113644 "2019-04-06T21:43:50Z")
**Posts on this page:** 1
**Showing post:** 2

<div class="post-metadata">

### Author: ![Stephen](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/stephen/32/95011_2.png) [@Stephen](https://meta.discourse.org/u/Stephen)
#### Post date: [April 6, 2019, 10:19pm UTC](https://meta.discourse.org/t/how-to-edit-robot-txt-for-indexing-user-profiles/113644/2 "2019-04-06T22:19:33Z")

</div>

User profiles are disallowed as the version which is seen by the crawler contains no useful information.

It’s not a question of whether the server can handle it or “take the hit”, it’s the tens of thousands of effectively empty pages which will waste the crawl time allocated to your site by the search engines.

---

_[View the full topic](https://meta.discourse.org/t/how-to-edit-robot-txt-for-indexing-user-profiles/113644)._