Generating sitemap for a private discourse instance


#1

I’ve integrated discourse into my site https://reviews.anytreatment.com as a private forum, and I was hoping to generate a sitemap to facilitate an internal site-wide search (I use swifttype for this).

Currently a user has to be on discourse to search topics, and I want them to find topics from the non-discourse portions of my site as well.

Anyone know how I can generate this sitemap?

EDIT: It seems like swifttype might be able to handle this with some config that allow their crawler to access password-protected sites. Will update once I get this going.


#2

Update: Allowing Swifttype’s crawler (Swiftbot) access to a private discourse instance requires editing htaccess to allow user-agent Swiftbot to pass without authentication. Here’s their documentation for this: Can Swiftype index password protected sites? - Setup & Indexing - Swiftype Community Forums (super short)

I’m on a docker install which I believe uses nginx, and I’m not sure I should begin editing nginx config to match the htaccess rewrite rules. Additionally I’m using SSO, so I’m not sure if the editing should be happening on the discourse side or the SSO side, or both.

Swifttype doesn’t know the answer. Any suggestions would be much appreciated.

P.S. If there were an internal sitemap generating plugin, it would definitely suit this use case.