Bing is picking all the rss feed for each post, can I disable the feed in Discourse

support
1

So, I’m Discourse as forum for a website using a subdomain, and I’m noticing that Bing is picking up each post and category rss for indexing in the site sitemap section.

Is this an expected behavior?
Should I disable the rss feed on the forum to prevent this from happening?
Is there a way to disable the feed is Discourse?

6 Likes
2

No. RSS feeds have a noindex header, which Bing is supposed to respect. See
https://github.com/discourse/discourse/blob/master/app/controllers/application_controller.rb#L43

and

https://github.com/discourse/discourse/blob/master/app/controllers/application_controller.rb#L826

There’s no way to do this (other than making your entire site login required - but that would disable all indexing.)

Are you seeing RSS pages in actual Bing search results? Or only in their webmaster tools?

10 Likes
3

If there a way to confirm that the noindex appear in the feed? I ask because, I can’t even find the noindex tag in the source of the rss. For instance, I also opened this forum thread with the .rss, and in the source, I can’t find the noindex either.

Thus far, I only see the feeds in the webmaster tools, in the sitemap section under the feed grouping.

Thanks,

3 Likes
4

Sure. This assumes you’re using Google Chrome, but you can follow a similar process in other browsers.

  1. Open the Developer Tools panel F12
  2. Switch to the Network tab.
  3. Load an RSS feed, for example this topic - https://meta.discourse.org/t/bing-is-picking-all-the-rss-feed-for-each-post-can-i-disable-the-feed-in-discourse/168025.rss
  4. Select the RSS feed entry from the list.
  5. Look at the response headers on the Headers tab, you should see x-robots-tag: noindex
10 Likes
5

Thanks, I do see the x-robots-tag: noindex
I still don’t get it why Bing will pick up the RSS when it’s noindex tag.

5 Likes
6

Welcome to the exciting world of Bing!™

I wish I could say I was surprised. I really, really do.

6 Likes
7

Trying to catch up to Google? :slightly_smiling_face:

4 Likes
8

I’m going to pretend that you aren’t being sarcastic. :stuck_out_tongue_closed_eyes:

Nah, they’ve already done that, but they have a lot of catching up to do in terms of indexing Discourse forums correctly.

2 Likes
9

Hey everybody! Sorry for all the trouble. I’m going to forward this thread to the others on the Bing team. Bing is not supposed to index the RSS with noindex meta tag, so this is likely a bug or crawling issue. Thanks for the report!

Expect this to be fixed only within a few hours. I will then get back to this thread and make a post confirming so. If we experience some issues, I’ll take this to a private message with the OP. Again, thank you all.

5 Likes
10

@Frédéric has told me about this place and whats going on. I’m speaking on behalf of him while he is dealing with something else that unfortunately had to take his attention. :slight_smile:

At the moment, it appears that we have a grasp on the issue. Like Frédéric said, I have direct messaged @mhwebee and will privately take it from here. If you could close the thread, that’d be great!

Best regards from myself and the Bing developers,

Jordi

5 Likes