왜 semrushbot과 ahrefsbot이 기본적으로 차단되나요?

neounix · 12월 3, 2020, 9:35오전

이 방법을 시도해 보셨나요? (다만, 컨테이너 이름을 사용하세요)

참고: Admin UI에서도 이 설정을 구성할 수 있습니다:

# docker exec -it socket-only bash
root@socket-only:/# rails c
[1] pry(main)> SiteSetting.blocked_crawler_user_agents
=> "mauibot|semrushbot|ahrefsbot|blexbot|seo spider"
[2] pry(main)> SiteSetting.blocked_crawler_user_agents = ""
=> ""
[3] pry(main)> SiteSetting.blocked_crawler_user_agents
=> ""
[4] pry(main)>

다음도 참고하세요:

github.com/discourse/discourse

config/site_settings.yml

d1d87b6fa

# Available options:
#
# default            - The default value of the setting. For upload site settings, use the id of the upload seeded in db/fixtures/010_uploads.rb.
# client             - Set to true if the javascript should have access to this setting's value.
# refresh            - Set to true if clients should refresh when the setting is changed.
# min                - For a string setting, the minimum length. For an integer setting, the minimum value.
# max                - For a string setting, the maximum length. For an integer setting, the maximum value.
# regex              - A regex that the value must match.
# validator          - The name of the class that will be use to validate the value of the setting.
# allow_any          - For choice settings allow items not specified in the choice list (default true)
# secret             - Set to true if input type should be password and value needs to be scrubbed from logs (default false).
# enum               - The setting has a fixed set of allowed values, and only one can be chosen.
#                      Set to the class name that defines the set.
# locale_default     - A hash which overrides according to `SiteSetting.default_locale`.
#                      The key should be as the same as possible value of default_locale.
#
#
# type: email    - Must be a valid email address.
# type: username - Must match the username of an existing user.
# type: list     - A list of values, chosen from a set of valid values defined in the choices option.

This file has been truncated. show original

다음도 참고하세요:

  def self.allow_crawler?(user_agent)
    return true if SiteSetting.allowed_crawler_user_agents.blank? &&
      SiteSetting.blocked_crawler_user_agents.blank?
...
...

github.com/discourse/discourse

lib/crawler_detection.rb

e0d923225

# frozen_string_literal: true

module CrawlerDetection
  WAYBACK_MACHINE_URL = "archive.org"

  def self.to_matcher(string, type: nil)
    escaped = string.split('|').map { |agent| Regexp.escape(agent) }.join('|')

    if type == :real && Rails.env == "test"
      # we need this bypass so we properly render views
      escaped << "|Rails Testing"
    end

    Regexp.new(escaped, Regexp::IGNORECASE)
  end

  def self.crawler?(user_agent, via_header = nil)
    return true if user_agent.nil? || user_agent&.include?(WAYBACK_MACHINE_URL) || via_header&.include?(WAYBACK_MACHINE_URL)

    # this is done to avoid regenerating regexes

This file has been truncated. show original

코드를 보면 이 두 사이트 설정을 "빈 값(blank)"으로 설정하면 차단이 이루어지지 않음을 알 수 있습니다:

SiteSetting.allowed_crawler_user_agents
SiteSetting.blocked_crawler_user_agents

Discourse 코어에서 기본적으로 차단하는 이 봇들은 robots.txt를 준수하지 않기 때문에 변경하지 않는 것을 권장합니다. 하지만 이는 사용자의 사이트이므로 원하시는 대로 설정할 수 있습니다. 코어에서 이들을 차단하는 데는 좋은 이유가 있습니다.

그럼에도 불구하고, Discourse는 UI의 SiteSettings를 사용하여 이들을 "차단 해제"할 수 있는 옵션을 제공합니다.

글		댓글	조회수
Handling Bingbot Feature	28	7764	11월 20, 2020
MegaIndex bot did about 4,000 pageviews on one day Community Building	40	5064	12월 2, 2023
Issues Google Search Console is throwing at me for wrong discourse structure (or some for wrong administration of my site) Product Support	18	404	12월 18, 2024
Why there are lots of Disallow rule in robots.txt? Product Support	34	4964	12월 22, 2020
How to protect myself from bots crawling my Discourse instance? Product Support	4	1719	1월 21, 2019

왜 semrushbot과 ahrefsbot이 기본적으로 차단되나요?

Related topics