PoojaPatel
January 19, 2019, 4:55am
1
Currently I have faced issued in one of my wordpress site that someone has setup bot to crawl my whole site.
Q: How I know that bot crawl my site?
Solution: I have blocked that ip through security plugin.
So is it possible same thing will happens in discourse website?
What i have to do for setup precaution for above situation?
If you set your login_required setting to true, bots will not be able to crawl your forum.
4 Likes
PoojaPatel
January 19, 2019, 10:41am
3
Thank you for your suggestion.
If i set to true of login_required setting, then all users has to do login the forum.
May be it hurt in seo like if someone come from google search engine then they has to login first to read the content.
Also, i have doubt regarding how google, bing bot crawl if i set to true. (they stop crawling my forum?)
It will be great if someone has set to true and share their experience regarding google and yahoo bot crawling.
Search for “crawler” in site settings.
Use the blocked crawler user agents and allowed crawler user agents site settings.
7 Likes
PoojaPatel
January 21, 2019, 5:38am
5
Thank you to all your support and suggestion for guiding me to my small queries.
I have read below thread and read other topic but could not find exact “how to check logs for bots”.
Now that we have search logging in place it’s time to put that data into action.
I propose creating a new page under “Admin” → “Logs” → “Search Logs” tab, like so:
[39]
The initial version of search logs page will contain following data points:
term: the search term.
searches: number of times the term has been searched for.
continued from search: number of times user has clicked on topic from search results.
most viewed topic: the topic that has been visited most for a particular…
If, I will make all changes as above suggestion. Now i have to check generate logs for google and bing bot crawling my discourse forum.
Where i have to find logs for that in my dashboard?