Needing to edit robots.txt file - where is it?

Osama · 2019 年 7 月 11 日午後 8:44

I’ve created a PR for this:

github.com/discourse/discourse

FEATURE: Allow customization of robots.txt

master ← OsamaSayegh:customize-robots-txt

merged 05:47PM - 15 Jul 19 UTC

OsamaSayegh

+282 -7

This allows admins to customize/override the content of the robots.txt file at …/admin/customize/robots. That page is not linked to anywhere in the UI -- admins have to manually type the URL to access that page. Meta topic: https://meta.discourse.org/t/needing-to-edit-robots-txt-file-where-is-it/93879?u=osama Screenshots: ![image](https://user-images.githubusercontent.com/17474474/61083151-b5174800-a433-11e9-890e-283221046b2b.png) ![image](https://user-images.githubusercontent.com/17474474/61083312-16d7b200-a434-11e9-9cae-dc3a9982c049.png) @eviltrout does it make sense to prepend a comment to robots.txt that says something along the lines of "this robots.txt file has been customized at /admin/customize/robots" **if** the file is customized? It might help with figuring out why certain things are in the file and how to remove/change them?

Screenshots:

トピック		返信	表示
Why there are lots of Disallow rule in robots.txt? Support	34	4578	2020 年 12 月 22 日
Issues Google Search Console is throwing at me for wrong discourse structure (or some for wrong administration of my site) Support	18	204	2024 年 12 月 18 日
Excluding user profiles in robots.txt (or allow edit of file) Feature	5	2516	2014 年 5 月 24 日
Pages listed in the robots.txt are crawled and indexed by Google Support	19	3297	2019 年 7 月 30 日
Google indexing issue (robots.txt) Support	9	703	2024 年 5 月 23 日

Needing to edit robots.txt file - where is it?

関連トピック