# How to protect myself from bots crawling my Discourse instance?
**URL:** https://meta.discourse.org/t/how-to-protect-myself-from-bots-crawling-my-discourse-instance/106962
**Category:** Support
**Created:** [19 בינואר, 2019, 4:55am UTC](https://meta.discourse.org/t/how-to-protect-myself-from-bots-crawling-my-discourse-instance/106962 "2019-01-19T04:55:19Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@PoojaPatel](https://meta.discourse.org/u/PoojaPatel)
#### Post date: [19 בינואר, 2019, 4:55am UTC](https://meta.discourse.org/t/how-to-protect-myself-from-bots-crawling-my-discourse-instance/106962/1 "2019-01-19T04:55:19Z")
Currently I have faced issued in one of my wordpress site that someone has setup bot to crawl my whole site.
Q: How I know that bot crawl my site?
Ans: I have seen +500% clicks on my affiliate dashboard from my website. After that, I have login to server and check logs of last days and found one ip that are crawling my site and click to affiliate links.
Solution: I have blocked that ip through security plugin.
So is it possible same thing will happens in discourse website?
What i have to do for setup precaution for above situation?
---
### Author:  [@Mittineague](https://meta.discourse.org/u/Mittineague)
#### Post date: [19 בינואר, 2019, 8:16am UTC](https://meta.discourse.org/t/how-to-protect-myself-from-bots-crawling-my-discourse-instance/106962/2 "2019-01-19T08:16:13Z")
If you set your `login_required` setting to true, bots will not be able to crawl your forum.
---
### Author:  [@PoojaPatel](https://meta.discourse.org/u/PoojaPatel)
#### Post date: [19 בינואר, 2019, 10:41am UTC](https://meta.discourse.org/t/how-to-protect-myself-from-bots-crawling-my-discourse-instance/106962/3 "2019-01-19T10:41:53Z")
Thank you for your suggestion.
If i set to true of `login_required` setting, then all users has to do login the forum.
May be it hurt in seo like if someone come from google search engine then they has to login first to read the content.
Also, i have doubt regarding how google, bing bot crawl if i set to true. (they stop crawling my forum?)
It will be great if someone has set to true and share their experience regarding google and yahoo bot crawling.
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [19 בינואר, 2019, 11:21am UTC](https://meta.discourse.org/t/how-to-protect-myself-from-bots-crawling-my-discourse-instance/106962/4 "2019-01-19T11:21:38Z")
Search for “crawler” in site settings.
Use the `blocked crawler user agents` and `allowed crawler user agents` site settings.
---
### Author:  [@PoojaPatel](https://meta.discourse.org/u/PoojaPatel)
#### Post date: [21 בינואר, 2019, 5:38am UTC](https://meta.discourse.org/t/how-to-protect-myself-from-bots-crawling-my-discourse-instance/106962/5 "2019-01-21T05:38:10Z")
Thank you to all your support and suggestion for guiding me to my small queries.
I have read below thread and read other topic but could not find exact “how to check logs for bots”.
> [@Search Logs Page](https://meta.discourse.org/t/search-logs-page/73281/):
>
> Now that we have [search logging](https://meta.discourse.org/t/effectively-logging-search-queries/63838) in place it’s time to put that data into action. I propose creating a new page under “Admin” → “Logs” → “Search Logs” tab, like so: The initial version of search logs page will contain following data points: term: the search term. searches: number of times the term has been searched for. continued from search: number of times user has clicked on topic from search results. most viewed topic: the topic that has been visited most for a particular…
If, I will make all changes as above suggestion. Now i have to check generate logs for google and bing bot crawling my discourse forum.
Where i have to find logs for that in my dashboard?
---
### Author:  [@system](https://meta.discourse.org/u/system)
#### Post date: [15 בינואר, 2022, 3:52pm UTC](https://meta.discourse.org/t/how-to-protect-myself-from-bots-crawling-my-discourse-instance/106962/6 "2022-01-15T15:52:31Z")
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [17 בינואר, 2022, 6:01pm UTC](https://meta.discourse.org/t/how-to-protect-myself-from-bots-crawling-my-discourse-instance/106962/7 "2022-01-17T18:01:49Z")