# Block crawlers from embedded topics only?

**URL:** <https://meta.discourse.org/t/block-crawlers-from-embedded-topics-only/268916>\
**Category:** Product Support\
**Created:** [June 19, 2023, 3:34pm UTC](https://meta.discourse.org/t/block-crawlers-from-embedded-topics-only/268916 "2023-06-19T15:34:53Z")\
**Posts on this page:** 1\
**Showing post:** 13

<div class="post-metadata">

**Author:** ![angus](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/angus/32/341715_2.png) [@angus](https://meta.discourse.org/u/angus)\
**Post date:** [June 19, 2023, 6:50pm UTC](https://meta.discourse.org/t/block-crawlers-from-embedded-topics-only/268916/13 "2023-06-19T18:50:38Z")

</div>

> [@haydenjames](#):
>
> Question: If I manually edit my Discourse robots.txt file. Will the change remain persistent?

If you do it via `/admin/customize/robots` it will persist.

 ![Screenshot 2023-06-19 at 20.50.19](https://global.discourse-cdn.com/meta/original/4X/5/9/4/594ff5c632742f2df27eaf7801553623dccedaf8.png)

It gets stored in a hidden site setting called `overridden_robots_txt`. If that’s filled it will always be served as your robots.txt file.

---

_[View the full topic](https://meta.discourse.org/t/block-crawlers-from-embedded-topics-only/268916)._