About 1/3 of our traffic is from crawlers (about 250K last month). Is there a way to block these but allow Google’s crawlers?
AI-generated summary
A user, Roy_ILGM, is seeking a way to block crawlers while allowing Google’s crawlers on their site, which receives a significant amount of crawler traffic. codinghorror suggests using site settings, where you can find options to block crawler user agents, or specify allowed crawler user agents to only permit specific crawlers, such as Google’s.
أصبح هذا ممكنًا الآن عبر إعدادات الموقع، ابحث عن “crawler”، وتحديدًا blocked crawler user agents. يوجد أيضًا allowed crawler user agents إذا كنت ترغب في أن تكون صارمًا للغاية وعدم السماح بأي شيء باستثناء crawler محدد.
12 إعجابًا