Indexing Discourse Community Content in Glean AI

It is not matter of AI, but crawlers. And AFAIK I know, the answer is no, and yes. If a category is visible for everyone it can be scraped. That`s how googlebot is working. It a forum is behind log in, or visibility of a category is limited by trust levels, scraping is impossible. And I really hope that wounldn’t be breaked ever, because it is one of the most important security measurements.

But sure, you can scrape such ”hidden” content, if

  • you get a system where a bot can log in and read content, or
  • you’ll index the content from inside using Discourse AI connected to wanted model (or similar system)