Missing X-Robots-Tag

We have DISCOURSE_ALLOW_INDEX_IN_ROBOTS_TXT set to false, but the X-Robots-Tag header is not being sent in the response.

Any ideas of what could be happening? Not sure if it could be related to this change.

1 Like

Iā€™ve dug it for our Staging site @agmontpetit was referring to, I looks like itā€™s caused by our custom plugin somehow - will investigate.

1 Like

Turns out this happens for any Guest GET HTML request to an action not skipping check_xhr filter for some reason. I couldnā€™t find such actions in standard Discourse, though, so probably only some plugin developers suffer from this issue. To us this started to happen after we added a custom page with xhr-provided content for Guests, and stopped when we added preloading to it (which requires skipping check_xhr).

This tricky issue can be fixed with very simply by running add_noindex_header before check_xhr.

1 Like