So, I’m Discourse as forum for a website using a subdomain, and I’m noticing that Bing is picking up each post and category rss for indexing in the site sitemap section. Is this an expected behavior? Should I disable the rss feed on the forum to prevent this from happening? Is there a way to disable the feed is Discourse?

[image] mhwebee: Is this an expected behavior? No. RSS feeds have a noindex header, which Bing is supposed to respect. See app/controllers/application_controller.rb main end end around_action :ensure_dont_cache_page before_action :check_readonly_mode before_action :handle_theme before_action :set_current_user_for_logs before_action :set_mp_snapshot_fields before_action :clear_notifications around_action :with_resolved_locale before_action :block_if_readonly_mode before_action :authorize_mini_profiler before_action :redirect_to_login_if_required before_action :block_if_requires_login before_action :redirect_to_profile_if_required before_action :preload_json before_action :initialize_application_layout_preloader before_action :check_xhr before_action :set_crawler_header after_action :add_readonly_header after_action :perform_refresh_session and app/controllers/application_controller.rb main if client&.auth_redirect.present? && params[:auth_redirect] != client.auth_redirect return render plain: I18n.t("user_api_key.invalid_auth_redirect") end if UserApiKeyClient.invalid_auth_redirect?(params[:auth_redirect]) return render plain: I18n.t("user_api_key.invalid_auth_redirect") end if UserApiKey.allowed_scopes.superset?(Set.new(["one_time_password"])) redirect_to("#{params[:auth_redirect]}?otp=true", allow_other_host: true) return end end if !current_user && SiteSetting.login_required? flash.keep if (request.format && request.format.json?) || request.xhr? || !request.get? ensure_logged_in else redirect_to_login end [image] mhwebee: Should I disable the rss feed on the forum to prevent this from happening? Is there a way to disable the feed is Discourse? There’s no way to do this (other than making your entire site login required - but that would disable all indexing.) Are you seeing RSS pages in actual Bing search results? Or only in their webmaster tools?

If there a way to confirm that the noindex appear in the feed? I ask because, I can’t even find the noindex tag in the source of the rss. For instance, I also opened this forum thread with the .rss, and in the source, I can’t find the noindex either. Thus far, I only see the feeds in the webmaster tools, in the sitemap section under the feed grouping. Thanks,

[image] mhwebee: If there a way to confirm that the noindex appear in the feed? Sure. This assumes you’re using Google Chrome, but you can follow a similar process in other browsers. Open the Developer Tools panel F12 Switch to the Network tab. Load an RSS feed, for example this topic - https://meta.discourse.org/t/bing-is-picking-all-the-rss-feed-for-each-post-can-i-disable-the-feed-in-discourse/168025.rss Select the RSS feed entry from the list. Look at the response headers on the Headers tab, you should see x-robots-tag: noindex

[image] mhwebee: I still don’t get it why Bing will pick up the RSS when it’s noindex tag. Welcome to the exciting world of Bing!™ I wish I could say I was surprised. I really, really do.

[image] codinghorror: Welcome to the exciting world of Bing!™ Trying to catch up to Google? :slightly_smiling_face:

[image] codinghorror: Welcome to the exciting world of Bing!™ I’m going to pretend that you aren’t being sarcastic. :stuck_out_tongue_closed_eyes: [image] JimPas: Trying to catch up to Google? :slightly_smiling_face: Nah, they’ve already done that, but they have a lot of catching up to do in terms of indexing Discourse forums correctly.

[image] mhwebee: I still don’t get it why Bing will pick up the RSS when it’s noindex tag. Hey everybody! Sorry for all the trouble. I’m going to forward this thread to the others on the Bing team. Bing is not supposed to index the RSS with noindex meta tag, so this is likely a bug or crawling issue. Thanks for the report! Expect this to be fixed only within a few hours. I will then get back to this thread and make a post confirming so. If we experience some issues, I’ll take this to a private message with the OP. Again, thank you all.

@Frédéric has told me about this place and whats going on. I’m speaking on behalf of him while he is dealing with something else that unfortunately had to take his attention. :slight_smile: At the moment, it appears that we have a grasp on the issue. Like Frédéric said, I have direct messaged @mhwebee and will privately take it from here. If you could close the thread, that’d be great! Best regards from myself and the Bing developers, Jordi

Bing is picking all the rss feed for each post, can I disable the feed in Discourse

Product Support

mhwebee (M) October 25, 2020, 6:30pm 5

Thanks, I do see the x-robots-tag: noindex
I still don’t get it why Bing will pick up the RSS when it’s noindex tag.

Topic		Replies	Views
Issues with embedding from RSS Product Support	31	3758	June 26, 2020
Can we add "feed" function on Discourse? Feature	10	4082	March 6, 2017
RSS Polling Plugin official , rss-polling , included-in-core	4	18447	August 14, 2026
Finding Discourse RSS feeds Using Discourse rss , reference	25	11606	January 18, 2025
Way to globally disable all RSS feeds Feature	6	1433	October 26, 2021

Bing is picking all the rss feed for each post, can I disable the feed in Discourse

Related topics