How to protect myself from bots crawling my Discourse instance?

Support

AI-generated summary

User PoojaPatel is seeking advice on how to protect their Discourse instance from bots crawling their site, citing a previous experience on a WordPress site where a bot crawled and clicked on affiliate links.

Mittineague suggests setting the login_required setting to true, but PoojaPatel is concerned this may negatively impact SEO and wonders how Google and Bing bots will crawl the site.

codinghorror recommends using the blocked crawler user agents and allowed crawler user agents site settings, found by searching for “crawler” in site settings.

PoojaPatel is now seeking guidance on how to check logs for bot activity, specifically for Google and Bing bots, and where to find these logs in the dashboard.

1

Currently I have faced issued in one of my wordpress site that someone has setup bot to crawl my whole site.

Q: How I know that bot crawl my site?
Ans: I have seen +500% clicks on my affiliate dashboard from my website. After that, I have login to server and check logs of last days and found one ip that are crawling my site and click to affiliate links.

Solution: I have blocked that ip through security plugin.

So is it possible same thing will happens in discourse website?

What i have to do for setup precaution for above situation?

2

If you set your login_required setting to true, bots will not be able to crawl your forum.

3

Thank you for your suggestion.

If i set to true of login_required setting, then all users has to do login the forum.

May be it hurt in seo like if someone come from google search engine then they has to login first to read the content.

Also, i have doubt regarding how google, bing bot crawl if i set to true. (they stop crawling my forum?)

It will be great if someone has set to true and share their experience regarding google and yahoo bot crawling.

4

Search for “crawler” in site settings.

Use the blocked crawler user agents and allowed crawler user agents site settings.

5

Thank you to all your support and suggestion for guiding me to my small queries.

I have read below thread and read other topic but could not find exact “how to check logs for bots”.

If, I will make all changes as above suggestion. Now i have to check generate logs for google and bing bot crawling my discourse forum.

Where i have to find logs for that in my dashboard?