Bing is picking all the rss feed for each post, can I disable the feed in Discourse

mhwebee · October 23, 2020, 12:03am

So, I’m Discourse as forum for a website using a subdomain, and I’m noticing that Bing is picking up each post and category rss for indexing in the site sitemap section.

Is this an expected behavior?
Should I disable the rss feed on the forum to prevent this from happening?
Is there a way to disable the feed is Discourse?

jomaxro · October 23, 2020, 2:30am

No. RSS feeds have a noindex header, which Bing is supposed to respect. See

github.com

discourse/discourse/blob/main/app/controllers/application_controller.rb#L43


      
          before_action :handle_theme
          before_action :set_current_user_for_logs
          before_action :set_mp_snapshot_fields
          before_action :clear_notifications
          around_action :with_resolved_locale
          before_action :set_mobile_view
          before_action :block_if_readonly_mode
          before_action :authorize_mini_profiler
          before_action :redirect_to_login_if_required
          before_action :block_if_requires_login
          before_action :preload_json
          before_action :check_xhr
          after_action :add_readonly_header
          after_action :perform_refresh_session
          after_action :dont_cache_page
          after_action :conditionally_allow_site_embedding
          after_action :ensure_vary_header
          after_action :add_noindex_header,
                       if: -> { is_feed_request? || !SiteSetting.allow_index_in_robots_txt }
          after_action :add_noindex_header_to_non_canonical, if: :spa_boot_request?
          after_action :set_cross_origin_opener_policy_header, if: :spa_boot_request?

and

github.com

discourse/discourse/blob/main/app/controllers/application_controller.rb#L826


      
            @requires_login_arg = arg
          end
          
          def self.requires_login_arg
            @requires_login_arg
          end
          
          def block_if_requires_login
            if arg = self.class.requires_login_arg
              check =
                if except = arg[:except]
                  !except.include?(action_name.to_sym)
                elsif only = arg[:only]
                  only.include?(action_name.to_sym)
                else
                  true
                end
              ensure_logged_in if check
            end
          end

There’s no way to do this (other than making your entire site login required - but that would disable all indexing.)

Are you seeing RSS pages in actual Bing search results? Or only in their webmaster tools?

mhwebee · October 23, 2020, 6:19pm

If there a way to confirm that the noindex appear in the feed? I ask because, I can’t even find the noindex tag in the source of the rss. For instance, I also opened this forum thread with the .rss, and in the source, I can’t find the noindex either.

Thus far, I only see the feeds in the webmaster tools, in the sitemap section under the feed grouping.

Thanks,

jomaxro · October 23, 2020, 8:06pm

Sure. This assumes you’re using Google Chrome, but you can follow a similar process in other browsers.

Open the Developer Tools panel F12
Switch to the Network tab.
Load an RSS feed, for example this topic - https://meta.discourse.org/t/bing-is-picking-all-the-rss-feed-for-each-post-can-i-disable-the-feed-in-discourse/168025.rss
Select the RSS feed entry from the list.
Look at the response headers on the Headers tab, you should see x-robots-tag: noindex

mhwebee · October 25, 2020, 6:30pm

Thanks, I do see the x-robots-tag: noindex
I still don’t get it why Bing will pick up the RSS when it’s noindex tag.

codinghorror · October 25, 2020, 10:43pm

Welcome to the exciting world of Bing!™

I wish I could say I was surprised. I really, really do.

JimPas · October 26, 2020, 4:31am

Trying to catch up to Google?

anon23393886 · November 20, 2020, 4:33pm

I’m going to pretend that you aren’t being sarcastic.

Nah, they’ve already done that, but they have a lot of catching up to do in terms of indexing Discourse forums correctly.

Frédéric · November 20, 2020, 9:02pm

Hey everybody! Sorry for all the trouble. I’m going to forward this thread to the others on the Bing team. Bing is not supposed to index the RSS with noindex meta tag, so this is likely a bug or crawling issue. Thanks for the report!

Expect this to be fixed only within a few hours. I will then get back to this thread and make a post confirming so. If we experience some issues, I’ll take this to a private message with the OP. Again, thank you all.

Jordi · November 20, 2020, 11:17pm

@Frédéric has told me about this place and whats going on. I’m speaking on behalf of him while he is dealing with something else that unfortunately had to take his attention.

At the moment, it appears that we have a grasp on the issue. Like Frédéric said, I have direct messaged @mhwebee and will privately take it from here. If you could close the thread, that’d be great!

Best regards from myself and the Bing developers,

Jordi

Topic		Replies	Views
Way to globally disable all RSS feeds Feature	6	1141	October 26, 2021
Why there are lots of Disallow rule in robots.txt? Support	34	4430	December 22, 2020
Bingbot is (no longer) default throttled Announcements	17	18710	November 20, 2020
Handling Bingbot Feature	29	7275	November 20, 2020
Google complaining – Indexed, though blocked by robots.txt Support	24	2410	September 28, 2023

Bing is picking all the rss feed for each post, can I disable the feed in Discourse

Related topics