[Screenshot 2020-07-01 at 23.04.02] there are way more than that in the list, any ideas?

I saw this in the logs of one of our clients today so this is more than a coincidence. [image] EDIT no I think it is a coincidence, searching for ymwears .cn shows more complaints about referral spam, for example these (over one year old) Relevanssi shows weird search queries on my page | WordPress.org and Block specific referrer or agent to enter url | WordPress.org

I had a client complaining about these last month. I blocked a few IPs and considered configuring fail2ban to block IPs that searched for some of those URLs, but never actual did anything about it. I looked in to blocking by geographic region, but it seemed that you’d need a $20/month database to accomplish that.

interesting are you guys aware of any solution that might work without having to mess around with the server itself.. @pfaffman @RGJ

Referrer spam is a pretty big issue which even the big guys (i.e. Google Analytics) are not fighting 100% successfully. For now all I can think of is removing these entries manually. Since these sites are apparently - at least partially - the same on multiple independent Discourse sites (given the fact that our screenshots are showing pretty much the same list) maybe a (dynamic) blacklist would be an idea? @codinghorror do you have a suggestion?

[image] Yassine_Yousfi: Interesting are you guys aware of any solution that might work without having to mess around with the server itself… We have seen, addressed and mitigated this issue at a large scale for years and have found the more reliable way (over the past few years) to block rouge bots in to block based on the user agent (UA) string (sometimes in combo with geoip information). We have blocked 100s of millions of Chinese bots hits over the years, and we have rarely found blocking IP addresses works over time nearly as well as blocking clients based on UA strings. Here is a snippet of one piece of code we use on one of our sites as an example: $user_agents = explode('|',$string_of_bad_user_agents,-1); $hide_content_useragent = $_SERVER['HTTP_USER_AGENT']; $IS_A_BAD_BOT = FALSE; foreach($user_agents as $hcag) { trim($hcag); if (preg_match("/$hcag/i", "$hide_content_useragent")) { $IS_A_BAD_BOT = TRUE; break; } } Most all (not all) rouge bots use UA strings which can be fairly easily identified and blocked (in this era, not sure about in the future as things evolve); and so we abandoned the method of trying to block rouge bots based on IP addresses years ago. The reason we abandoned blocking based on IPs is because many countries, like China, Russia, N. Korea, and many more, now run their bot farms from servers in other countries. IP addresses are not a good indicator of actual origin or intent. In addition, by blocking massive IP address blocks, good addresses can be blocked denying access by legitimate users. For example, China runs huge server farms of bots out of Brazil and other countries closer (geographically to the US) to disguise their origin and to retrieve data faster (shorter Internet reach). Sometimes the WHOIS data will back match a Chinese, N. Korean or Russian (examples) physical address but other times they will not and will use local country physical addresses. We have seen a lot of rouge Chinese bots registered to Brazilian companies (over the past few years) where we could see (and confirm) the user agent strings matched rouge bots out of China. In addition, when we perform doing Google searches on those UA strings, we see others have also identified many of the same UA strings as Chinese (for example). In summary, while many people immediate go to blocking IP addresses to block rouge bot activity; most sophisticated bot farms are very good at running their bots out of other countries. It’s easy to set up a VPS in most countries and of course, the closer the bot is to the targeted country, the more data the bot can scrape. VPS can come and go in minutes and bot software can be deployed very quickly in just about any VPS data center globally. For the past few years, blocking based on UA string has proven to be the more reliable method (sometimes in combo with geoip info, sometimes not); but of course spammers, bot masters, and their agents are also beginning to disguise the UA strings as they have their IP addresses (for many years). Hope this helps. Cheers & Happy Bot Hunting!

Yes, I absolutely agree that IP blocking is not effective. User agent blocking tends to work pretty well, except when the spammers are constantly changing it. That’s why my thoughts went to just blacklisting the actual URL that is being referrer spammed. It just “feels better” because we’re not blocking something based on an underlying assumption (i.e. “this user agent supplies a bad referrer so we don’t trust it”) but we are actually blocking what we want to block (“we see this website being referrer spammed on more Discourse sites, let’s not put it in our database”). At least this is harder to circumvent.

[image] RGJ: User agent blocking tends to work pretty well, except when the spammers are constantly changing it. That’s why my thoughts went to just blacklisting the actual URL that is being referrer spammed. Good thoughts. There is no one size fits all to stop rouge and malicious bots; and each site must evaluate which controls work best for them. On a similar note… Sites which rely mainly on blacklists and spam or rogue bot databases can also have problems because let’s say someone does not like site www.our-arch-rival.com because that site is a competitor (or simply made us angry or offended us). Some people will then submit the site www.our-arch-rival.com to a blacklist or DB and then other sites will filter a legitimate site because of this kind of “bad consequence” from the method of blacklisting DBs. Then, of course advocates of blacklists will stay, “you can go to the blacklisting sites and submit a report and asked to be removed”, but for many busy, long standing sites, that can be a time killer. Generally, it is important to analyze the problem and create a mitigation strategy based on the scenario because every “adversary” is different. It’s the old “Know Your Enemy” from Sun Tzu and the Art of War. Every situation is a bit different in the real-world and unfortunately, it does take analysis skills for sys admins to create optimal mitigation strategies against malicious or unwanted cyber activity. This is also a good reason to run Discourse behind a reverse proxy because the reverse proxy is a good place for analyzing, classifying and controlling malicious activity before this traffic hits the Discourse app. It can be a full time job in the year 2020 trying to control and mitigate rouge bots and other malicious activity in cyberspace. As soon as admins come up with one good detection and mitigation strategy, the spammers and scrappers will adjust and find ways around it. I tend to advise people to oversize their servers to insure they have enough headroom since these kinds of problems in cyberspace are only going to get worse over time, not better. Ready Player One!

[image] neounix: As soon as admins come up with one good detection and mitigation strategy, the spammers and scrappers will adjust and find ways around it. Which is another reason to stay away from IP blacklisting: the spammers will know you are taking measures.

I think can block most spammers through Cloudflare but not sure what to put in rules for the browser Agent @neounix what do you mean by “UA strings”? and how can they be used in Cloudflare firewall rules?

[image] RGJ: Referrer spam is a pretty big issue which even the big guys But this isn’t even referrer spam, is it? It’s just that they are doing a search for that URL, so it’s not actually doing anything, is it? Do I totally misunderstand what that report is? It’s not available to anyone but admins, right?

Why I'm getting lots of searches from Chinese websites?

Product Support

RGJ (Richard - Communiteq) July 3, 2020, 11:52am 9

Which is another reason to stay away from IP blacklisting: the spammers will know you are taking measures.

Topic		Replies	Views
How do I stop someone from spamming "Trending Search Terms"? Product Support	28	2535	February 26, 2026
Pageviews from Anonymous Users have exploded but Google Analytics showed no traffic growth. How to find about where the increase come from? Data & reporting	23	2711	January 5, 2021
View IP address of guests / anonymous visitors? Data & reporting	13	1702	January 13, 2022
User-agent: lfc-discourse-public-crawler Bug	3	244	July 9, 2026
MegaIndex bot did about 4,000 pageviews on one day Community Building	40	5112	December 2, 2023

Why I'm getting lots of searches from Chinese websites?

Related topics