neounix
3 Diciembre, 2020 09:35
12
¿Probaste esto (pero usando el nombre de tu contenedor)?
Nota: también puedes configurar esto en la interfaz de administración (Admin UI):
# docker exec -it socket-only bash
root@socket-only:/# rails c
[1] pry(main)> SiteSetting.blocked_crawler_user_agents
=> "mauibot|semrushbot|ahrefsbot|blexbot|seo spider"
[2] pry(main)> SiteSetting.blocked_crawler_user_agents = ""
=> ""
[3] pry(main)> SiteSetting.blocked_crawler_user_agents
=> ""
[4] pry(main)>
Ver también:
# Available options:
#
# default - The default value of the setting. For upload site settings, use the id of the upload seeded in db/fixtures/010_uploads.rb.
# client - Set to true if the javascript should have access to this setting's value.
# refresh - Set to true if clients should refresh when the setting is changed.
# min - For a string setting, the minimum length. For an integer setting, the minimum value.
# max - For a string setting, the maximum length. For an integer setting, the maximum value.
# regex - A regex that the value must match.
# validator - The name of the class that will be use to validate the value of the setting.
# allow_any - For choice settings allow items not specified in the choice list (default true)
# secret - Set to true if input type should be password and value needs to be scrubbed from logs (default false).
# enum - The setting has a fixed set of allowed values, and only one can be chosen.
# Set to the class name that defines the set.
# locale_default - A hash which overrides according to `SiteSetting.default_locale`.
# The key should be as the same as possible value of default_locale.
#
#
# type: email - Must be a valid email address.
# type: username - Must match the username of an existing user.
# type: list - A list of values, chosen from a set of valid values defined in the choices option.
show original
Ver también:
def self.allow_crawler?(user_agent)
return true if SiteSetting.allowed_crawler_user_agents.blank? &&
SiteSetting.blocked_crawler_user_agents.blank?
...
...
# frozen_string_literal: true
module CrawlerDetection
WAYBACK_MACHINE_URL = "archive.org"
def self.to_matcher(string, type: nil)
escaped = string.split('|').map { |agent| Regexp.escape(agent) }.join('|')
if type == :real && Rails.env == "test"
# we need this bypass so we properly render views
escaped << "|Rails Testing"
end
Regexp.new(escaped, Regexp::IGNORECASE)
end
def self.crawler?(user_agent, via_header = nil)
return true if user_agent.nil? || user_agent&.include?(WAYBACK_MACHINE_URL) || via_header&.include?(WAYBACK_MACHINE_URL)
# this is done to avoid regenerating regexes
show original
Puedes ver en el código que si estableces estas dos configuraciones del sitio en “vacío” (blank), no habrá bloqueo:
SiteSetting.allowed_crawler_user_agents
SiteSetting.blocked_crawler_user_agents
Recomiendo que no cambies esto, ya que estos bots que Discourse bloquea por defecto en su núcleo no respetan robots.txt; sin embargo, es tu sitio y puedes hacer lo que quieras. Hay una buena razón por la que están bloqueados en el núcleo.
Dicho esto, Discourse te ofrece la opción de “desbloquear” estos usando tus SiteSettings en la interfaz de usuario.