Why I'm getting lots of searches from Chinese websites?

Product Support
6

We have seen, addressed and mitigated this issue at a large scale for years and have found the more reliable way (over the past few years) to block rouge bots in to block based on the user agent (UA) string (sometimes in combo with geoip information).

We have blocked 100s of millions of Chinese bots hits over the years, and we have rarely found blocking IP addresses works over time nearly as well as blocking clients based on UA strings.

Here is a snippet of one piece of code we use on one of our sites as an example:

$user_agents = explode('|',$string_of_bad_user_agents,-1);
$hide_content_useragent = $_SERVER['HTTP_USER_AGENT'];
$IS_A_BAD_BOT = FALSE;

foreach($user_agents as $hcag) {
    trim($hcag);
    if (preg_match("/$hcag/i", "$hide_content_useragent")) {
        $IS_A_BAD_BOT = TRUE;
        break;
    }
}

Most all (not all) rouge bots use UA strings which can be fairly easily identified and blocked (in this era, not sure about in the future as things evolve); and so we abandoned the method of trying to block rouge bots based on IP addresses years ago. The reason we abandoned blocking based on IPs is because many countries, like China, Russia, N. Korea, and many more, now run their bot farms from servers in other countries. IP addresses are not a good indicator of actual origin or intent. In addition, by blocking massive IP address blocks, good addresses can be blocked denying access by legitimate users.

For example, China runs huge server farms of bots out of Brazil and other countries closer (geographically to the US) to disguise their origin and to retrieve data faster (shorter Internet reach).

Sometimes the WHOIS data will back match a Chinese, N. Korean or Russian (examples) physical address but other times they will not and will use local country physical addresses. We have seen a lot of rouge Chinese bots registered to Brazilian companies (over the past few years) where we could see (and confirm) the user agent strings matched rouge bots out of China. In addition, when we perform doing Google searches on those UA strings, we see others have also identified many of the same UA strings as Chinese (for example).

In summary, while many people immediate go to blocking IP addresses to block rouge bot activity; most sophisticated bot farms are very good at running their bots out of other countries. It’s easy to set up a VPS in most countries and of course, the closer the bot is to the targeted country, the more data the bot can scrape. VPS can come and go in minutes and bot software can be deployed very quickly in just about any VPS data center globally.

For the past few years, blocking based on UA string has proven to be the more reliable method (sometimes in combo with geoip info, sometimes not); but of course spammers, bot masters, and their agents are also beginning to disguise the UA strings as they have their IP addresses (for many years).

Hope this helps.

Cheers & Happy Bot Hunting!