How to protect myself from bots crawling my Discourse instance?

支持
1

Currently I have faced issued in one of my wordpress site that someone has setup bot to crawl my whole site.

Q: How I know that bot crawl my site?
Ans: I have seen +500% clicks on my affiliate dashboard from my website. After that, I have login to server and check logs of last days and found one ip that are crawling my site and click to affiliate links.

Solution: I have blocked that ip through security plugin.

So is it possible same thing will happens in discourse website?

What i have to do for setup precaution for above situation?

2

If you set your login_required setting to true, bots will not be able to crawl your forum.

4 个赞
3

Thank you for your suggestion.

If i set to true of login_required setting, then all users has to do login the forum.

May be it hurt in seo like if someone come from google search engine then they has to login first to read the content.

Also, i have doubt regarding how google, bing bot crawl if i set to true. (they stop crawling my forum?)

It will be great if someone has set to true and share their experience regarding google and yahoo bot crawling.

4

Search for “crawler” in site settings.

Use the blocked crawler user agents and allowed crawler user agents site settings.

7 个赞
5

Thank you to all your support and suggestion for guiding me to my small queries.

I have read below thread and read other topic but could not find exact “how to check logs for bots”.

If, I will make all changes as above suggestion. Now i have to check generate logs for google and bing bot crawling my discourse forum.

Where i have to find logs for that in my dashboard?