Why I'm getting lots of searches from Chinese websites?

Product Support
14

@Yassine_Yousfi

Here you go…

In HTTP, the User-Agent string is often used for content negotiation, where the origin server selects suitable content or operating parameters for the response. For example, the User-Agent string might be used by a web server to choose variants based on the known capabilities of a particular version of client software. The concept of content tailoring is built into the HTTP standard in RFC 1945 "for the sake of tailoring responses to avoid particular user agent limitations.”

The User-Agent string is one of the criteria by which Web crawlers may be excluded from accessing certain parts of a website using the Robots Exclusion Standard( robots.txt file).

As with many other HTTP request headers, the information in the “User-Agent” string contributes to the information that the client sends to the server, since the string can vary considerably from user to user.[5]

Reference:

@Yassine_Yousfi. there are myriad references on the Internet about HTTP user agent (UA) strings and how to use them in various ways, including as a sensor when detecting bots and other malicious cyber activity.

Happy Bot Hunting!

Notes:

  1. You can see the Discourse view of bot user agents here (some UA strings are truncated):
https://discourse.your-great-domain.com/admin/reports/web_crawlers

  1. No detection algorithm can detect all bots with 100% accuracy.

  2. You can also get the UA strings from your web server log files and other methods.

See also: