# 需要编辑 robots.txt 文件 - 它在哪里？

**URL:** https://meta.discourse.org/t/needing-to-edit-robots-txt-file-where-is-it/93879
**Category:** Support
**Created:** [2018年八月3日 21:11 UTC](https://meta.discourse.org/t/needing-to-edit-robots-txt-file-where-is-it/93879 "2018-08-03T21:11:37Z")
**Posts on this page:** 20
**Page:** 2

<div class="post-metadata">

### Author: ![jerdog](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/jerdog/32/122843_2.png) [@jerdog](https://meta.discourse.org/u/jerdog)
#### Post date: [2018年八月8日 00:38 UTC](https://meta.discourse.org/t/needing-to-edit-robots-txt-file-where-is-it/93879/21 "2018-08-08T00:38:40Z")

</div>

> [@codinghorror](#):
>
> > [@sam](#):
> >
> > The problem is that we store “latest” content in the search index which is pointless cause we never want people to land on “latest”
> 
> This statement makes zero sense to me, “latest” is _the default homepage for Discourse_ so we absolutely _do_ want people to land there.

Correct me if I am wrong, but _Latest_ is the default display but not the default link, right? This has to do with the actual `/latest` link

---

<div class="post-metadata">

### Author: ![sam](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/sam/32/102149_2.png) [@sam](https://meta.discourse.org/u/sam)
#### Post date: [2018年八月8日 00:46 UTC](https://meta.discourse.org/t/needing-to-edit-robots-txt-file-where-is-it/93879/22 "2018-08-08T00:46:20Z")

</div>

> [@codinghorror](#):
>
> This statement makes zero sense to me, “latest” is _the default homepage for Discourse_ so we absolutely _do_ want people to land there.

We have every single page of latest in the index, the content is like quicksand and there is nothing in the homepage that is “site specific” and not quicksand which is a big problem:

We absolutely do not want people landing on page 2 / 3 etc.. page 1 maybe, but the content on page 1 keeps on changing.

This URL for example `https://meta.discourse.org/latest?no_definitions=true&no_subcategories=false&page=2` is stored in the Google index.

 ![image](https://global.discourse-cdn.com/meta/original/3X/2/d/2df22548359370f73408de3dd143dc7767b6c275.png)

I am reticent to change stuff though cause I do not know how the big Google will deal with us adding “dont store in index” directives here. Also people never land on these pages anyway cause Google automatically detects they are rubbish and do not send people there.

If there is anything super positive here, I guess it would be having a wonderful “HTML off” homepage that has useful enough content that search engines would send people to the page.

For example, it would be super nice if `discourse community discussions` ranked `meta.discourse.org` first cause we had a nice front page.

A simple fix here we can make that can give us lots of mileage is nice expansion of pinned posts:

 ![image](https://global.discourse-cdn.com/meta/original/3X/9/0/90a339c1c19ad1eb02be073c88648e608d7f71b5.png)

They are stable content, we can expand that:

 ![image](https://global.discourse-cdn.com/meta/original/3X/7/2/727abed3925ef439df76aa250246449e1915ea56.png)

In fact we can even expand it a bit further for crawler views. Additionally we could list all the categories on the home page as well in the crawler view… there is a bunch of stuff we can do.

---

<div class="post-metadata">

### Author: ![Pham\_Quyet\_Nghi](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/pham_quyet_nghi/32/100298_2.png) [@Pham\_Quyet\_Nghi](https://meta.discourse.org/u/Pham_Quyet_Nghi)
#### Post date: [2018年八月8日 01:41 UTC](https://meta.discourse.org/t/needing-to-edit-robots-txt-file-where-is-it/93879/23 "2018-08-08T01:41:42Z")

</div>

Hello!  
this is my file

```
# See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
#
User-agent: *
Disallow: /auth/cas
Disallow: /auth/facebook/callback
Disallow: /auth/twitter/callback
Disallow: /auth/google/callback
Disallow: /auth/yahoo/callback
Disallow: /auth/github/callback
Disallow: /auth/cas/callback
Disallow: /assets/browser-update*.js
Disallow: /users/
Disallow: /u/
Disallow: /my/
Disallow: /badges/
Disallow: /search
Disallow: /search/
Disallow: /tags
Disallow: /tags/
Disallow: /email/
Disallow: /session
Disallow: /session/
Disallow: /admin
Disallow: /admin/
Disallow: /user-api-key
Disallow: /user-api-key/
Disallow: /*?api_key*
Disallow: /*?*api_key*
Disallow: /groups
Disallow: /groups/
Disallow: /t/*/*.rss
Disallow: /tags/*.rss
Disallow: /c/*.rss

User-agent: mauibot
Disallow: /

User-agent: bingbot
Crawl-delay: 60
Disallow: /auth/cas
Disallow: /auth/facebook/callback
Disallow: /auth/twitter/callback
Disallow: /auth/google/callback
Disallow: /auth/yahoo/callback
Disallow: /auth/github/callback
Disallow: /auth/cas/callback
Disallow: /assets/browser-update*.js
Disallow: /users/
Disallow: /u/
Disallow: /my/
Disallow: /badges/
Disallow: /search
Disallow: /search/
Disallow: /tags
Disallow: /tags/
Disallow: /email/
Disallow: /session
Disallow: /session/
Disallow: /admin
Disallow: /admin/
Disallow: /user-api-key
Disallow: /user-api-key/
Disallow: /*?api_key*
Disallow: /*?*api_key*
Disallow: /groups
Disallow: /groups/
Disallow: /t/*/*.rss
Disallow: /tags/*.rss
Disallow: /c/*.rss

```

I read the tutorials above but I do not understand how to fix the question “Need to edit robots.txt file - where is it?”. Looking forward to receiving help from the community

**This is the content to be want to update**

```
# See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file
#
User-agent: *
Disallow: /auth/cas
Disallow: /auth/facebook/callback
Disallow: /auth/twitter/callback
Disallow: /auth/google/callback
Disallow: /auth/yahoo/callback
Disallow: /auth/github/callback
Disallow: /auth/cas/callback
Disallow: /assets/browser-update*.js
Disallow: /users/
Disallow: /u/
Disallow: /badges/
Disallow: /search
Disallow: /search/
Disallow: /tags
Disallow: /tags/

```

Thanks all

---

<div class="post-metadata">

### Author: ![Stranik](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/stranik/32/85638_2.png) [@Stranik](https://meta.discourse.org/u/Stranik)
#### Post date: [2018年八月8日 06:58 UTC](https://meta.discourse.org/t/needing-to-edit-robots-txt-file-where-is-it/93879/24 "2018-08-08T06:58:13Z")

</div>

> [@Pham\_Quyet\_Nghi](#):
>
> I do not understand how to fix the question “Need to edit robots.txt file - where is it?”.

I think you can override the file in your own plugin.

> <https://github.com/discourse/discourse/blob/main/app/views/robots_txt/index.erb>

---

<div class="post-metadata">

### Author: ![Pham\_Quyet\_Nghi](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/pham_quyet_nghi/32/100298_2.png) [@Pham\_Quyet\_Nghi](https://meta.discourse.org/u/Pham_Quyet_Nghi)
#### Post date: [2018年八月11日 02:05 UTC](https://meta.discourse.org/t/needing-to-edit-robots-txt-file-where-is-it/93879/25 "2018-08-11T02:05:49Z")

</div>

My archive directory is this

 ![robots%20txt](https://global.discourse-cdn.com/meta/original/3X/a/6/a6620ecd33b2e343d52932be881c098a985c5990.jpeg)

how to override the file in your own plugin

**Thanks**

---

<div class="post-metadata">

### Author: ![cpradio](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/cpradio/32/4970_2.png) [@cpradio](https://meta.discourse.org/u/cpradio)
#### Post date: [2018年八月11日 02:15 UTC](https://meta.discourse.org/t/needing-to-edit-robots-txt-file-where-is-it/93879/26 "2018-08-11T02:15:52Z")

</div>

You will want to read the plugin development topics and then read this  
[https://meta.discourse.org/t/how-to-block-all-crawlers-but-googles/62431/4?u=cpradio](https://meta.discourse.org/t/how-to-block-all-crawlers-but-googles/62431/4)

---

<div class="post-metadata">

### Author: ![Pham\_Quyet\_Nghi](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/pham_quyet_nghi/32/100298_2.png) [@Pham\_Quyet\_Nghi](https://meta.discourse.org/u/Pham_Quyet_Nghi)
#### Post date: [2018年八月11日 02:26 UTC](https://meta.discourse.org/t/needing-to-edit-robots-txt-file-where-is-it/93879/27 "2018-08-11T02:26:40Z")

</div>

I really do not want to block the google search engine that I want to change by content in the `robots.txt` file

Why does my website not find such a directory `/discourse/app/views` ?

---

<div class="post-metadata">

### Author: ![Mittineague](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/mittineague/32/114259_2.png) [@Mittineague](https://meta.discourse.org/u/Mittineague)
#### Post date: [2018年八月11日 03:32 UTC](https://meta.discourse.org/t/needing-to-edit-robots-txt-file-where-is-it/93879/28 "2018-08-11T03:32:04Z")

</div>

There is no robots.txt text file per se. It is a Ruby controller

> <https://github.com/discourse/discourse/blob/main/app/controllers/robots_txt_controller.rb>

---

<div class="post-metadata">

### Author: ![cpradio](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/cpradio/32/4970_2.png) [@cpradio](https://meta.discourse.org/u/cpradio)
#### Post date: [2018年八月11日 03:42 UTC](https://meta.discourse.org/t/needing-to-edit-robots-txt-file-where-is-it/93879/29 "2018-08-11T03:42:00Z")

</div>

> [@Pham\_Quyet\_Nghi](#):
>
> I really do not want to block the google search engine that I want to change by content in the `robots.txt` file
> 
> Why does my website not find such a directory `/discourse/app/views` ?

You really need to read some of the #Development topics, it explains all of that and more. The plugin should be trivial, to be honest. Or you can post something in #Marketplace with a budget to see if someone will build it for you.

---

<div class="post-metadata">

### Author: ![j127](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/j127/32/79093_2.png) [@j127](https://meta.discourse.org/u/j127)
#### Post date: [2018年八月16日 17:53 UTC](https://meta.discourse.org/t/needing-to-edit-robots-txt-file-where-is-it/93879/30 "2018-08-16T17:53:17Z")

</div>

> [@sam](#):
>
> we should probably add noindex tags on user pages

If that is added, could it be made into an overridable setting? I clicked on this link in the newsletter, because getting user pages indexed is also something we need. We’re hoping to add additional information to them and eventually redirect the old (indexed) user pages to the Discourse ones.

---

<div class="post-metadata">

### Author: ![j127](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/j127/32/79093_2.png) [@j127](https://meta.discourse.org/u/j127)
#### Post date: [2019年四月13日 17:08 UTC](https://meta.discourse.org/t/needing-to-edit-robots-txt-file-where-is-it/93879/31 "2019-04-13T17:08:14Z")

</div>

> [@jerdog](#):
>
> we need to add `/latest*` to the Disallow list.

I was just noticing this problem on one of my Discourse sites. The way to block those dynamic URLs from bots while still allowing search engines to crawl `/latest` is this:

```plaintext
Disallow: /latest?

```

That will only block the dynamic ones, but not `/latest`, so search engines would still be able to see the latest content. I tested the rule in [Google’s Webmaster Tools](https://www.google.com/webmasters/tools/robots-testing-tool) and it works.

Here’s an example of some of the dynamic URLs that are getting crawled on my site:

[https://gist.githubusercontent.com/j127/d329c15dab45369b03321cad40448734/raw/300aa579b1386087b903da6aa52c52ff5d95828c/latest.txt](https://gist.githubusercontent.com/j127/d329c15dab45369b03321cad40448734/raw/300aa579b1386087b903da6aa52c52ff5d95828c/latest.txt)

Is it possible to add that one line to `robots.txt`?

( **Edit:** I looked more closely at the file, and I wouldn’t use `noindex` there, at least on that dynamic rule. I’m pretty sure that Google has recommended not to use `noindex` in `robots.txt` though it was several years ago.)

---

<div class="post-metadata">

### Author: ![codinghorror](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/codinghorror/32/110067_2.png) [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [2019年七月9日 23:30 UTC](https://meta.discourse.org/t/needing-to-edit-robots-txt-file-where-is-it/93879/32 "2019-07-09T23:30:20Z")

</div>

现在，您可以通过站点设置封禁或限制恶意网络爬虫，这将间接编辑 `robots.txt` 文件，但我们仍不提供任意编辑功能。

不过我认为我们应该提供该功能……@eviltrout 能否将其纳入 2.4 版本的范围？这回应了许多请求，其中不少我们并不认同，但我的态度是：“后果自负，如果您觉得有必要，那就去做吧 ☠”

---

<div class="post-metadata">

### Author: ![Stephen](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/stephen/32/95011_2.png) [@Stephen](https://meta.discourse.org/u/Stephen)
#### Post date: [2019年七月10日 02:24 UTC](https://meta.discourse.org/t/needing-to-edit-robots-txt-file-where-is-it/93879/34 "2019-07-10T02:24:37Z")

</div>

我们至少可以将编辑 robots.txt 文件明确列为社区支持范围之外的事项吗？

---

<div class="post-metadata">

### Author: ![vinothkannans](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/vinothkannans/32/86465_2.png) [@vinothkannans](https://meta.discourse.org/u/vinothkannans)
#### Post date: [2019年七月10日 05:17 UTC](https://meta.discourse.org/t/needing-to-edit-robots-txt-file-where-is-it/93879/35 "2019-07-10T05:17:43Z")

</div>

顺便一提，任何人都可以通过使用 “`robots_txt_index`” 连接器模板的简单插件轻松添加额外规则。例如：[app/views/connectors/robots\_txt\_index/sitemap.html.erb](https://github.com/discourse/discourse-sitemap/blob/master/app/views/connectors/robots_txt_index/sitemap.html.erb)

---

<div class="post-metadata">

### Author: ![eviltrout](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/eviltrout/32/5275_2.png) [@eviltrout](https://meta.discourse.org/u/eviltrout)
#### Post date: [2019年七月10日 19:25 UTC](https://meta.discourse.org/t/needing-to-edit-robots-txt-file-where-is-it/93879/36 "2019-07-10T19:25:28Z")

</div>

以下是我认为它应该如何工作的流程：

- 在管理区域添加一个新的未直接链接的 URL。例如 `/admin/customize/robots`

---

<div class="post-metadata">

### Author: ![codinghorror](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/codinghorror/32/110067_2.png) [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [2019年七月10日 19:31 UTC](https://meta.discourse.org/t/needing-to-edit-robots-txt-file-where-is-it/93879/38 "2019-07-10T19:31:36Z")

</div>

> [@eviltrout](#):
>
> 在管理部分添加一个新标签页：“管理 \> 自定义 \> robots.txt”

我强烈反对这一提议，因为它将一个晦涩且危险的功能置于用户界面的显著位置。

我认为，目前自定义 robots.txt 的路径应由用户自行输入。如果用户需要此功能，他们应通过搜索 Google 或 Meta 来找到相应路径。

---

<div class="post-metadata">

### Author: ![eviltrout](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/eviltrout/32/5275_2.png) [@eviltrout](https://meta.discourse.org/u/eviltrout)
#### Post date: [2019年七月10日 19:44 UTC](https://meta.discourse.org/t/needing-to-edit-robots-txt-file-where-is-it/93879/39 "2019-07-10T19:44:43Z")

</div>

这就是为什么我把它藏在“高级编辑”后面，但如果我们在让界面变得复杂，我可以进一步简化它（会编辑那篇帖子。）

---

<div class="post-metadata">

### Author: ![Osama](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/osama/32/98013_2.png) [@Osama](https://meta.discourse.org/u/Osama)
#### Post date: [2019年七月11日 20:44 UTC](https://meta.discourse.org/t/needing-to-edit-robots-txt-file-where-is-it/93879/40 "2019-07-11T20:44:41Z")

</div>

我已为此创建了 PR：

> <https://github.com/discourse/discourse/pull/7884>
>
> This allows admins to customize/override the content of the robots.txt
> file at …/admin/customize/robots. That page is not linked to anywhere in
> the UI -- admins have to manually type the URL to access that page.
> 
> Meta topic: https://meta.discourse.org/t/needing-to-edit-robots-txt-file-where-is-it/93879?u=osama
> 
> Screenshots:
> 
> !\[image\](https://user-images.githubusercontent.com/17474474/61083151-b5174800-a433-11e9-890e-283221046b2b.png)
> 
> !\[image\](https://user-images.githubusercontent.com/17474474/61083312-16d7b200-a434-11e9-9cae-dc3a9982c049.png)
> 
> @eviltrout does it make sense to prepend a comment to robots.txt that says something along the lines of "this robots.txt file has been customized at /admin/customize/robots" \*\*if\*\* the file is customized? It might help with figuring out why certain things are in the file and how to remove/change them?

截图：

 ![image](https://global.discourse-cdn.com/meta/original/3X/2/8/28b61f808e6f4fa28f34296b1bfb7cc4bf1054cc.png)

 ![image](https://global.discourse-cdn.com/meta/original/3X/f/7/f778129e5b56b76c8930c6cc38879c436cd3a33b.png)

---

<div class="post-metadata">

### Author: ![codinghorror](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/codinghorror/32/110067_2.png) [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [2019年七月12日 00:14 UTC](https://meta.discourse.org/t/needing-to-edit-robots-txt-file-where-is-it/93879/41 "2019-07-12T00:14:25Z")

</div>

看起来不错！请确保“还原”按钮使用正确的图标，与站点设置中“还原”功能使用的图标一致。此外，我们统一使用“重置”一词，因此你可以直接复用该文案，无需创建新的翻译。

![image](https://global.discourse-cdn.com/meta/original/3X/8/9/89b6bbec15277d756b004ec8763e4793556836d5.png)

另外，我们需要针对少数会修改 `robots.txt` 的站点设置添加警告，因为手动编辑等操作会覆盖这些设置。

---

<div class="post-metadata">

### Author: ![Osama](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/osama/32/98013_2.png) [@Osama](https://meta.discourse.org/u/Osama)
#### Post date: [2019年七月15日 18:44 UTC](https://meta.discourse.org/t/needing-to-edit-robots-txt-file-where-is-it/93879/42 "2019-07-15T18:44:06Z")

</div>

PR 已合并：🎉

> <https://github.com/discourse/discourse/commit/6515ff19e5c8e62ba3aaecb5947eaccdcbbaf0dd>
>
> \* FEATURE: Allow customization of robots.txt
> 
> This allows admins to customize/…override the content of the robots.txt
> file at /admin/customize/robots. That page is not linked to anywhere in
> the UI -- admins have to manually type the URL to access that page.
> 
> \* use Ember.computed.not
> 
> \* Jeff feedback
> 
> \* Feedback
> 
> \* Remove unused import

如果您更新至最新的测试通过版本，即可在 `/admin/customize/robots` 自定义 robots.txt。该页面在 UI 中未提供任何链接，您需要手动将 URL 复制并粘贴到浏览器中访问。

注意：如果您覆盖了该文件，则后续对站点设置的任何修改（例如“白名单爬虫用户代理”等）将不会反映在 robots.txt 文件中（设置会正常保存，但更改不会生效）。您可以恢复为默认版本，此时站点设置将再次对该文件生效。

如果存在覆盖内容，且管理员访问 `/robots.txt` 文件，他们将在文件顶部看到一条注释，提示存在覆盖，并提供链接以便您修改文件或重置为默认版本。

[上一頁](https://meta.discourse.org/t/needing-to-edit-robots-txt-file-where-is-it/93879.md?page=1)

[下一頁](https://meta.discourse.org/t/needing-to-edit-robots-txt-file-where-is-it/93879.md?page=3)