So, I’m Discourse as forum for a website using a subdomain, and I’m noticing that Bing is picking up each post and category rss for indexing in the site sitemap section.
Is this an expected behavior?
Should I disable the rss feed on the forum to prevent this from happening?
Is there a way to disable the feed is Discourse?
If there a way to confirm that the noindex appear in the feed? I ask because, I can’t even find the noindex tag in the source of the rss. For instance, I also opened this forum thread with the .rss, and in the source, I can’t find the noindex either.
Thus far, I only see the feeds in the webmaster tools, in the sitemap section under the feed grouping.
Sure. This assumes you’re using Google Chrome, but you can follow a similar process in other browsers.
Open the Developer Tools panel F12
Switch to the Network tab.
Load an RSS feed, for example this topic - https://meta.discourse.org/t/bing-is-picking-all-the-rss-feed-for-each-post-can-i-disable-the-feed-in-discourse/168025.rss
Select the RSS feed entry from the list.
Look at the response headers on the Headers tab, you should see x-robots-tag: noindex
Hey everybody! Sorry for all the trouble. I’m going to forward this thread to the others on the Bing team. Bing is not supposed to index the RSS with noindex meta tag, so this is likely a bug or crawling issue. Thanks for the report!
Expect this to be fixed only within a few hours. I will then get back to this thread and make a post confirming so. If we experience some issues, I’ll take this to a private message with the OP. Again, thank you all.
@Frédéric has told me about this place and whats going on. I’m speaking on behalf of him while he is dealing with something else that unfortunately had to take his attention.
At the moment, it appears that we have a grasp on the issue. Like Frédéric said, I have direct messaged @mhwebee and will privately take it from here. If you could close the thread, that’d be great!