It’s possible to override robots.txt in /admin/customize/robots (which is linked in the settings). The page provides a form to have a custom robots.txt. Example:
When pressing that ‘Reset’ button however, the form is immediately filled with what seems to be the website’s own html code and… appears to immediately save that as the new robots.txt
After a refresh however, it seems to actually have the Discourse’s default robots.txt, so it was reset to the default. But still, that filling with html probably needs a look.
On that note, form reset buttons usually restore the form to initial values (and don’t immediately submit the form), so maybe a “Restore to default robots.txt” link next to the buttons would be more clear.
