Needing to edit robots.txt file - where is it?

Osama · 11 ביולי,‏ 2019,‏ 8:44pm

I’ve created a PR for this:

github.com/discourse/discourse

FEATURE: Allow customization of robots.txt

master ← OsamaSayegh:customize-robots-txt

merged 05:47PM - 15 Jul 19 UTC

OsamaSayegh

+282 -7

This allows admins to customize/override the content of the robots.txt file at …/admin/customize/robots. That page is not linked to anywhere in the UI -- admins have to manually type the URL to access that page. Meta topic: https://meta.discourse.org/t/needing-to-edit-robots-txt-file-where-is-it/93879?u=osama Screenshots: ![image](https://user-images.githubusercontent.com/17474474/61083151-b5174800-a433-11e9-890e-283221046b2b.png) ![image](https://user-images.githubusercontent.com/17474474/61083312-16d7b200-a434-11e9-9cae-dc3a9982c049.png) @eviltrout does it make sense to prepend a comment to robots.txt that says something along the lines of "this robots.txt file has been customized at /admin/customize/robots" **if** the file is customized? It might help with figuring out why certain things are in the file and how to remove/change them?

Screenshots:

נושא		תגובות	צפיות
Why there are lots of Disallow rule in robots.txt? Support	34	4633	22 בדצמבר,‏ 2020
Issues Google Search Console is throwing at me for wrong discourse structure (or some for wrong administration of my site) Support	18	240	18 בדצמבר,‏ 2024
Excluding user profiles in robots.txt (or allow edit of file) Feature	5	2531	24 במאי,‏ 2014
Pages listed in the robots.txt are crawled and indexed by Google Support	19	3335	30 ביולי,‏ 2019
Google indexing issue (robots.txt) Support	9	714	23 במאי,‏ 2024

Needing to edit robots.txt file - where is it?

נושאים קשורים