Good to know. Sounds like a big step in the right direction.
For what it’s worth, we’ve actually seen some instances in our current forum (back when it was still public) where we had added the robots.txt file to certain subfolders of our forum, and Google was crawling them anyway. I believe this is highly irregular… but our discovery was that there are some cases where Google doesn’t follow this directive. The only way to be 100% sure a page doesn’t get indexed is for the page to be hidden behind a login screen or for the page to not exist at all.
From an outsider’s perspective, it seems like it should be simple to tell Discourse not to go through the extra motions of creating these additional URLs for every reply. The software would be doing less work and creating less complexity this way, wouldn’t it?
It would be nice to have a feature in the admin settings to just turn off these extra URLs altogether.