I don’t think we’ve ever encountered this situation before. You would need to authenticate the crawler as a regular user, probably through the Discourse API. Do we have any examples of this @techapj?
codinghorror
(Jeff Atwood)
7
Related topics
| Topic | Replies | Views | Activity | |
|---|---|---|---|---|
| Robots.txt to completely block all indexing of private site | 4 | 994 | March 6, 2020 | |
| Can't login after removing all bot user agents from site setting | 5 | 981 | November 24, 2017 | |
| Discourse SEO overview (sitemap / robots.txt) | 0 | 2587 | October 18, 2023 | |
| How to protect myself from bots crawling my Discourse instance? | 4 | 1713 | January 21, 2019 | |
| Menu, title collumn, and a lot a stuff are gone | 14 | 1371 | December 11, 2017 |