robots.txt is only a pollite suggestion. It doesn’t stop anything per se. But if a user agent follows those rules, it tells that bot isn’t awful malicious.
We should do such operations on nginx level 
2 Likes