# Keeping crawlers off a staging site

**URL:** <https://meta.discourse.org/t/keeping-crawlers-off-a-staging-site/128919>\
**Category:** Feature\
**Created:** [September 18, 2019, 3:56pm UTC](https://meta.discourse.org/t/keeping-crawlers-off-a-staging-site/128919 "2019-09-18T15:56:12Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![pfaffman](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/pfaffman/32/120154_2.png) [@pfaffman](https://meta.discourse.org/u/pfaffman)\
**Post date:** [September 18, 2019, 3:56pm UTC](https://meta.discourse.org/t/keeping-crawlers-off-a-staging-site/128919/1 "2019-09-18T15:56:12Z")

</div>

A client with a staging site just raised concern that their site could be crawled. (It would require a crawler to go looking for their domain, but if a crawler somehow stumbled on a link to the site, then the site could be crawled.)

The solutions for this, of course, are to turn on `allow_index_in_robots_txt` and/or `login_required`, but this would require remembering to reset them every time a backup from the production site is restored to the staging site.

Neither of these settings is `shadowed by global`.

Solutions that I have right now are to have `app.yml` modify `site_settings.yml` to add `shadowed_by_global` to one of those variables or to create a plugin that sets them.

Am I missing something?

---

<div class="post-metadata">

**Author:** ![gerhard](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/gerhard/32/119479_2.png) [@gerhard](https://meta.discourse.org/u/gerhard)\
**Post date:** [September 18, 2019, 9:29pm UTC](https://meta.discourse.org/t/keeping-crawlers-off-a-staging-site/128919/2 "2019-09-18T21:29:59Z")

</div>

> [@pfaffman](#):
>
> Neither of these settings is `shadowed by global` .

It’s highly likely that `shadowed_by_global` will be replaced by “every setting can be shadowed” real soon. 🤞

[https://github.com/discourse/discourse/pull/8061](https://github.com/discourse/discourse/pull/8061)