If you set their user agent to be identified as a crawler bot, Discourse will render a basic HTML view which is much easier to index.
Alternatively, add their user agent to the hidden site setting crawler_user_agents.
3 Likes