# Missing X-Robots-Tag
**URL:** https://meta.discourse.org/t/missing-x-robots-tag/152593
**Category:** Development
**Created:** [May 25, 2020, 10:26am UTC](https://meta.discourse.org/t/missing-x-robots-tag/152593 "2020-05-25T10:26:52Z")
**Posts on this page:** 3
**Page:** 1
### Author:  [@agmontpetit](https://meta.discourse.org/u/agmontpetit)
#### Post date: [May 25, 2020, 10:26am UTC](https://meta.discourse.org/t/missing-x-robots-tag/152593/1 "2020-05-25T10:26:53Z")
We have `DISCOURSE_ALLOW_INDEX_IN_ROBOTS_TXT` set to `false`, but the `X-Robots-Tag` header is not being sent in the response.
Any ideas of what could be happening? Not sure if it could be related to this [change](https://github.com/discourse/discourse/commit/bb4e8899c41889f7316e512d6ed89a3847fa655b#diff-55c5b7aecfb519d0e4880eaf2788eb6e).
---
### Author:  [@Artem\_Vasiliev](https://meta.discourse.org/u/Artem_Vasiliev)
#### Post date: [May 25, 2020, 11:03am UTC](https://meta.discourse.org/t/missing-x-robots-tag/152593/2 "2020-05-25T11:03:56Z")
I’ve dug it for our Staging site @agmontpetit was referring to, I looks like it’s caused by our custom plugin somehow - will investigate.
---
### Author:  [@Artem\_Vasiliev](https://meta.discourse.org/u/Artem_Vasiliev)
#### Post date: [May 25, 2020, 12:45pm UTC](https://meta.discourse.org/t/missing-x-robots-tag/152593/3 "2020-05-25T12:45:49Z")
Turns out this happens for any Guest GET HTML request to an action not skipping [check\_xhr](https://github.com/discourse/discourse/blob/bb4e8899c41889f7316e512d6ed89a3847fa655b/app/controllers/application_controller.rb#L652) filter for some reason. I couldn’t find such actions in standard Discourse, though, so probably only some plugin developers suffer from this issue. To us this started to happen after we added a custom page with xhr-provided content for Guests, and stopped when we added preloading to it (which requires skipping `check_xhr`).
This tricky issue can be fixed with very simply by running `add_noindex_header` [before `check_xhr`.](https://github.com/discourse/discourse/pull/9868/files#diff-55c5b7aecfb519d0e4880eaf2788eb6eR42)