PR was just merged: 
https://github.com/discourse/discourse/commit/6515ff19e5c8e62ba3aaecb5947eaccdcbbaf0dd
If you update to latest tests-passed, you’ll be able to customize robots.txt at /admin/customize/robots. The page is not linked to from anywhere in the UI, you’ll have to copy and paste the URL manually into your browser.
Note: if you override the file, any later changes to the site settings that modify robots.txt (e.g. whitelisted crawler user agents etc.) won’t apply to the file (settings will save correctly, but changes won’t reflect on robots.txt). You can restore to the default version and the site settings will apply to the file again.
If there are overrides AND an admin views the file at /robots.txt, they’ll see a comment on the top that says there’re overrides and links to where they can modify the file or reset to the default version.