# Needing to edit robots.txt file - where is it?
**URL:** https://meta.discourse.org/t/needing-to-edit-robots-txt-file-where-is-it/93879
**Category:** Support
**Created:** [03.Август.2018 21:11:37 UTC](https://meta.discourse.org/t/needing-to-edit-robots-txt-file-where-is-it/93879 "2018-08-03T21:11:37Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@jerdog](https://meta.discourse.org/u/jerdog)
#### Post date: [03.Август.2018 21:11:37 UTC](https://meta.discourse.org/t/needing-to-edit-robots-txt-file-where-is-it/93879/1 "2018-08-03T21:11:37Z")
So there have been a number of conversations around the robots.txt file, and how some need / don’t need to edit it based on their specific use-case. What I haven’t seen is exactly where to edit it (knowing that after an upgrade one would need to re-edit the file again). I see in `/var/www/discourse/app/views` there is a `robots.txt` directory with some ruby files but not an actual robots.txt file.
So where exactly does this file reside?
---
### Author:  [@gdpelican](https://meta.discourse.org/u/gdpelican)
#### Post date: [03.Август.2018 21:32:33 UTC](https://meta.discourse.org/t/needing-to-edit-robots-txt-file-where-is-it/93879/2 "2018-08-03T21:32:33Z")
Theres a robots\_txt\_controller.rb which returns some text which ends up in the robots.txt file when served.
It mostly runs on site settings, which you should do your best to use over modifying the file directly.
Those settings being
- whitelisted\_crawler\_user\_agents
- blacklisted\_crawler\_user\_agents
- slow\_down\_crawler\_rate
---
### Author:  [@jerdog](https://meta.discourse.org/u/jerdog)
#### Post date: [05.Август.2018 01:00:01 UTC](https://meta.discourse.org/t/needing-to-edit-robots-txt-file-where-is-it/93879/3 "2018-08-05T01:00:01Z")
Yep - those are valid settings. However, with the newest changes Google keeps making, we need to add `/latest*` to the Disallow list. These end up getting classified as “thin links” by Google and in turn negatively affect site ranking for search over time. For instance [this turns up](https://www.google.com/search?ei=OtthW5zILdGUtQXerInwBA&q=site%3Ahttps%3A%2F%2Fcommunity.auth0.com%2Flatest%3F&oq=site%3Ahttps%3A%2F%2Fcommunity.auth0.com%2Flatest%3F&gs_l=psy-ab.3...1640846.1644417.0.1644569.17.14.0.0.0.0.279.1520.0j7j2.9.0....0...1.1.64.psy-ab..10.0.0....0.XC_VaGKr34I) a lot of records which are not valid
It would benefit everyone if we _at least_ could get Latest added to the robots.txt file
---
### Author:  [@Mittineague](https://meta.discourse.org/u/Mittineague)
#### Post date: [05.Август.2018 01:26:41 UTC](https://meta.discourse.org/t/needing-to-edit-robots-txt-file-where-is-it/93879/4 "2018-08-05T01:26:41Z")
I just searched for Google documentation regarding “thin links” but came up empty. Got a link to the Google documentation handy?
---
### Author:  [@jerdog](https://meta.discourse.org/u/jerdog)
#### Post date: [05.Август.2018 01:41:58 UTC](https://meta.discourse.org/t/needing-to-edit-robots-txt-file-where-is-it/93879/5 "2018-08-05T01:41:58Z")
Sorry, I meant _[thin content](https://www.vertical-leap.uk/blog/what-is-thin-content-why-does-it-matter/)_
---
### Author:  [@Mittineague](https://meta.discourse.org/u/Mittineague)
#### Post date: [05.Август.2018 02:28:46 UTC](https://meta.discourse.org/t/needing-to-edit-robots-txt-file-where-is-it/93879/6 "2018-08-05T02:28:46Z")
And you’ve gotten “Thin content with little or no added value” message(s) on your Manual Actions Report page for the “latest” page?
Anyway, getting a bit off-topic, sorry.
You can write a plugin to add rules to the robots.txt file
[https://meta.discourse.org/t/how-to-add-to-robots-txt-host/59718/4?u=mittineague](https://meta.discourse.org/t/how-to-add-to-robots-txt-host/59718/4)
---
### Author:  [@mpalmer](https://meta.discourse.org/u/mpalmer)
#### Post date: [05.Август.2018 04:07:07 UTC](https://meta.discourse.org/t/needing-to-edit-robots-txt-file-where-is-it/93879/7 "2018-08-05T04:07:07Z")
If this is something that would be of benefit to many people, consider making a PR against core to add it either as a default, or behind a site setting.
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [06.Август.2018 00:15:14 UTC](https://meta.discourse.org/t/needing-to-edit-robots-txt-file-where-is-it/93879/8 "2018-08-06T00:15:14Z")
> [@jerdog](#):
>
> It would benefit everyone if we _at least_ could get Latest added to the robots.txt file
I would be extremely against this cause how would google then be able to find new content? Latest is very much required.
@codinghorror what I totally support here is adding [noindex, follow](https://yoast.com/robots-meta-tags/) to meta tags for `/latest` and even category topic lists, it will clean up a bunch of messy results. Thoughts?
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [06.Август.2018 00:39:51 UTC](https://meta.discourse.org/t/needing-to-edit-robots-txt-file-where-is-it/93879/9 "2018-08-06T00:39:51Z")
I am not really following? What’s the problem? Our `meta.discourse.org` Google Search Console has been running for years, we check it every few months in quite some detail, and I’ve never seen any problems with this reported?
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [06.Август.2018 00:43:31 UTC](https://meta.discourse.org/t/needing-to-edit-robots-txt-file-where-is-it/93879/10 "2018-08-06T00:43:31Z")
The problem is that we store “latest” content in the search index which is pointless cause we never want people to land on “latest”. The tricky thing is that any mucking around here can rock the boat and this is not a boat we want to rock.
I have seen rare reports of this kind of stuff with user profiles not have noindex in meta tags, but not the reverse.
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [06.Август.2018 00:45:20 UTC](https://meta.discourse.org/t/needing-to-edit-robots-txt-file-where-is-it/93879/11 "2018-08-06T00:45:20Z")
I’m gonna need to see some screenshot Google Search Console proof to believe this is a problem, because we’ve checked our console pretty closely for _years_.
---
### Author:  [@markersocial](https://meta.discourse.org/u/markersocial)
#### Post date: [07.Август.2018 05:33:05 UTC](https://meta.discourse.org/t/needing-to-edit-robots-txt-file-where-is-it/93879/12 "2018-08-07T05:33:05Z")
As requested, here are screenshots from one site:
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [07.Август.2018 06:31:36 UTC](https://meta.discourse.org/t/needing-to-edit-robots-txt-file-where-is-it/93879/13 "2018-08-07T06:31:36Z")
This is completely unrelated and we should probably add noindex tags on user pages. This was raised before wrt Bing.
---
### Author:  [@markersocial](https://meta.discourse.org/u/markersocial)
#### Post date: [07.Август.2018 09:18:29 UTC](https://meta.discourse.org/t/needing-to-edit-robots-txt-file-where-is-it/93879/14 "2018-08-07T09:18:29Z")
Not sure why we’re operating on the assumption that user pages and other useful pages that have healthy internal linking shouldn’t be indexed when Google’s own forums (e.g. Google Product Forums) allow indexing of user profiles. Same with Reddit, Quora and even Stackoverflow 😉 I guess the biggest forum-like sites in the world and Google themselves must be bad at SEO.
Discourse is free and beggars can’t be choosers, please don’t take this as complaining and I appreciate the work you’re all doing. I’m just attempting to give objective helpful feedback.
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [07.Август.2018 09:27:11 UTC](https://meta.discourse.org/t/needing-to-edit-robots-txt-file-where-is-it/93879/15 "2018-08-07T09:27:11Z")
On what planet would you want a web search to end up on a _user page_ in a discussion forum? That’s beyond useless.
---
### Author:  [@markersocial](https://meta.discourse.org/u/markersocial)
#### Post date: [07.Август.2018 09:56:51 UTC](https://meta.discourse.org/t/needing-to-edit-robots-txt-file-where-is-it/93879/16 "2018-08-07T09:56:51Z")
A huge % of search engine queries are for people, including usernames. On Google Trends, most of the top searches are for people, 14/20 at the time of posting this (70%). [https://trends.google.com/trends/trendingsearches/daily?geo=US](https://trends.google.com/trends/trendingsearches/daily?geo=US)
Yes, most of these people are celebrities of some sort and this is at macro level. At micro level, there are still lots of searches for people, smaller influencers, bloggers etc. and are some of the most popular searches.
If I search for ‘{user} on {site\_name}’ the most relevant result would be that user’s profile, where I can see all their posts. Or if I’m searching for a particular person because I enjoy their content on other platforms, it’d be nice to have the ability to find them on a Discourse forum.
Google knows how to sort results, usually. Having the profiles indexed gives the ability to show the profiles for the searches that they are most relevant for.
The main thing in my opinion, is if not indexing profiles helps stop spammers due to not getting link juice from non-indexed profiles - using the same logic, link juice won’t flow internally through our sites as efficiently. Nofollow tags for all profile links to external URLs would achieve the same result.
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [07.Август.2018 10:58:53 UTC](https://meta.discourse.org/t/needing-to-edit-robots-txt-file-where-is-it/93879/17 "2018-08-07T10:58:53Z")
You are completely hijacking a topic here that is asking for **removal** of items from the Google index.
Feel free to open a dedicated topic to discuss this. There are changes we do support like having a reasonable HTML off version of the user summary page… however you are doing is totally derailing this topic. We have survived fine for the last 5 years without indexing user pages, lots of users **DO NOT** want their forum user page indexed by Google. Tons of profiles are thin and pointless. Turning this on, even just for TL2, can burn a lot of forums in Google cause of the dupe content.
You can see previous discussions here:
> [@Excluding user profiles in robots.txt (or allow edit of file)](https://meta.discourse.org/t/excluding-user-profiles-in-robots-txt-or-allow-edit-of-file/15886):
>
> After the latest Google Panda update, I found myself on the unhappy end of a 15% drop in visitors via Google, which led me to do some research on what exactly Google has indexed for my site. According to a quick search using “site:discuss.howtogeek.com/users/” I noticed that all of the user profiles are being indexed in Google, despite having almost no useful information on any of them. So on my very small forum that’s 4,310 additional URLs that are indexed in Google despite having very little …
> [@Why there are lots of Disallow rule in robots.txt?](https://meta.discourse.org/t/why-there-are-lots-of-disallow-rule-in-robots-txt/84251):
>
> Does it prevent indexing in the SERPs? Never. Why does this happen? What best can be done? Please allow editing of the robots.txt file, we will do our best. wink Thanks
[https://meta.discourse.org/t/seo-compared-to-other-well-known-tools/3914](https://meta.discourse.org/t/seo-compared-to-other-well-known-tools/3914)
---
### Author:  [@markersocial](https://meta.discourse.org/u/markersocial)
#### Post date: [07.Август.2018 11:30:15 UTC](https://meta.discourse.org/t/needing-to-edit-robots-txt-file-where-is-it/93879/18 "2018-08-07T11:30:15Z")
Привет, Сэм,
Спасибо за ответ. Я попытался создать тему #Contribute > Feature, но она была заблокирована и должна быть удалена. Я не думал, что её посчитают дубликатом, так как, похоже, нет других тем с запросами на эту функцию. Есть несколько постов поддержки с вопросами о том, как редактировать файл robots.txt.
Вот пост, который я создал специально для этого:
[https://meta.discourse.org/t/robots-txt-remove-blocking-user-profiles-or-allow-editing/94082](https://meta.discourse.org/t/robots-txt-remove-blocking-user-profiles-or-allow-editing/94082)
Я поискал соответствующую тему и подумал, что это вторая тема, на которую ссылались как на причину закрытия моей темы. Приношу извинения. Наличие возможности редактировать файл robots.txt, как указано в заголовке этой темы, решило бы проблемы с индексацией для всех 🙂
Хорошо, понял. Большинство аргументов за исключение профилей пользователей, которые я видел, касаются в основном SEO, дублирования контента и т. д., с чем я не согласен. Я не видел, чтобы люди не хотели, чтобы их профили индексировались на публичном форуме, называли это одной из основных причин для их исключения. Это веская причина, просто я не думаю, что она была частью процесса принятия решений (по крайней мере, публично), когда их изначально исключили: [Excluding user profiles in robots.txt (or allow edit of file)](https://meta.discourse.org/t/excluding-user-profiles-in-robots-txt-or-allow-edit-of-file/15886)
В любом случае, я больше не буду писать в этой теме. Спасибо.
---
### Author:  [@jerdog](https://meta.discourse.org/u/jerdog)
#### Post date: [07.Август.2018 17:56:31 UTC](https://meta.discourse.org/t/needing-to-edit-robots-txt-file-where-is-it/93879/19 "2018-08-07T17:56:31Z")
The following came as an explanation and supporting use case for why this could/should be edited from our SEO Manager:
> The issue here lies in the fact that this is indexed and seen as a) duplicate content b) thin content c) does not provide value and d) affects page (and domain) authority increases.
>
> There is no Google Search Console error, warning, or penalties placed, _yet_ (“Thin content with low or no added value.” manual penalty is the most common penalty). Utilizing other SEO-related tools, this does surface. If requested, I can absolutely provide screenshots of these reports, just let me know.
>
> Moz Pro site crawl reports, for example, mark these Latest pages as having multiple content errors for a variety reasons and the issues associated with these pages include (based upon our domain reports, yet seemingly relevant to others): URL Too Long, Thin Content, Missing Canonical Tag, Overly Dynamic URL, Duplicate Titles and Descriptions, Missing H1, etc.
>
> The question here now seems to be “remove or improve?”. Yes one option is to simply add the noindex, nofollow meta tag values and include this in robots.txt and/or yes, another option is to simply add canonicalization to these duplicates, referencing back to the main /latest page. Also, yes, another option is to keep all of these pages indexed, perhaps then focusing on improvements and optimizations to these pages so the content truly does provide unique value and addresses correcting those issues noted above.
For reference [here is information from Google](https://support.google.com/webmasters/answer/2604824?visit_id=1-636692603248079019-2265619011&rd=1#thin-content) around this
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [08.Август.2018 00:23:24 UTC](https://meta.discourse.org/t/needing-to-edit-robots-txt-file-where-is-it/93879/20 "2018-08-08T00:23:24Z")
> [@sam](#):
>
> The problem is that we store “latest” content in the search index which is pointless cause we never want people to land on “latest”
This statement makes zero sense to me, “latest” is _the default homepage for Discourse_ so we absolutely _do_ want people to land there.
[Следующая страница](https://meta.discourse.org/t/needing-to-edit-robots-txt-file-where-is-it/93879.md?page=2)