We have DISCOURSE_ALLOW_INDEX_IN_ROBOTS_TXT set to false, but the X-Robots-Tag header is not being sent in the response.
Any ideas of what could be happening? Not sure if it could be related to this change.
1 Like
I’ve dug it for our Staging site @agmontpetit was referring to, I looks like it’s caused by our custom plugin somehow - will investigate.
1 Like
Turns out this happens for any Guest GET HTML request to an action not skipping check_xhr filter for some reason. I couldn’t find such actions in standard Discourse, though, so probably only some plugin developers suffer from this issue. To us this started to happen after we added a custom page with xhr-provided content for Guests, and stopped when we added preloading to it (which requires skipping check_xhr).
This tricky issue can be fixed with very simply by running add_noindex_header before check_xhr.
1 Like