I don’t think we’ve ever encountered this situation before. You would need to authenticate the crawler as a regular user, probably through the Discourse API. Do we have any examples of this @techapj?
codinghorror
(Jeff Atwood)
7
関連トピック
| トピック | 返信 | 表示 | アクティビティ | |
|---|---|---|---|---|
| Robots.txt to completely block all indexing of private site | 4 | 994 | 2020 年 3 月 6 日 | |
| Can't login after removing all bot user agents from site setting | 5 | 981 | 2017 年 11 月 24 日 | |
| Discourse SEO overview (sitemap / robots.txt) | 0 | 2587 | 2023 年 10 月 18 日 | |
| How to protect myself from bots crawling my Discourse instance? | 4 | 1714 | 2019 年 1 月 21 日 | |
| Menu, title collumn, and a lot a stuff are gone | 14 | 1371 | 2017 年 12 月 11 日 |