Discourse Meta
What can crawlers see?
Data & reporting
HAWK
(Hawk)
February 4, 2024, 2:41am
6
Controlling Web Crawlers For a Site
1 Like
show post in topic
Related topics
Topic
Replies
Views
Activity
How to protect myself from bots crawling my Discourse instance?
Support
6
1581
January 17, 2022
Controlling Web Crawlers For a Site
Site Management
how-to
10
2212
July 19, 2025
Smarter handling of random crawler traffic
Feature
2
3486
March 29, 2018
MegaIndex bot did about 4,000 pageviews on one day
Community
40
4440
December 2, 2023
How to allow user-agent access to private discourse?
Support
10
1999
November 25, 2018