Prevent scraping script running on my discourse platform

Marketplace

AI-generated summary

Preventing Scraping Script on Discourse Platform

AnnaNaumova416 wants to prevent scraping scripts from accessing content on her Discourse platform. She’s looking for a plugin that can detect if a user is accessing the site using a browser or not, and block content if it’s not a browser.

@PabloC notes that detecting scrapers can be challenging due to the use of headless browsers. He suggests blocking known scraper IPs and asks about the harm caused by scrapers and the type of content on the site.

@mbcahyono recommends using a browser integrity check, similar to Cloudflare’s, to detect non-browser users.

However, @fefrei states that currently, there isn’t a way to block content to non-browser users, and creating a plugin wouldn’t be effective due to the sophistication of modern scrapers.

AnnaNaumova416 has decided to give up on the idea, citing the lack of benefits and the challenges involved.

1

Hello. My name’s Anna and I want to have my own server running Discourse platform.

However, I have installed discourse platform on my server and it’s working fine.

One concern is that if the user access to the site using scraping script, for example, using PHP or Python, then I want to prevent sending content to those IP.

Idea is that if the user doesn’t use the browser then doesn’t send him the content.

I want this to be done by a plugin but I am open to your suggestion.

I am also open to any budget suggestion from you. Thanks in advance. Anna.

2

Its not simple to detect scrappers because toda headless browsers are used to scrape.

A simple first step would be to block all the ips you know are scrappers.

Also, how much harm are scrapers doing? What kind of content do you have?

1 Like
3

Hello @PabloC.

Thanks for your reply. For now, I am just opening the site so there is no content at all~

But I just want to detect if the user is using a browser or not.

If he is not using a browser then block sending content at the server side. Is it possible?

4

Something like Cloudflare’s browser integrity check?

5

Thanks, @mbcahyono for your reply.

Sure very similar to that.

Is there anyway to block content to non-browser users?

6

Currently, there isn’t. Something like this would have to be a plugin, and wouldn’t be effective – as @PabloC mentioned, modern scrapers are browsers.

1 Like
7

Thanks, @fefrei for your reply.

I am going to give up making that kind of plugin.

Because there is not much benefit having that, even though I can have one!

I have posted many posts to get help, but I finally give up… However thanks.

8

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.