There shouldn’t be anything additional you need to do… that crawler traffic is likely from crawlers hitting community.yoursite.com/login. If you check community.example.com/admin/reports/web_crawlers you can see how often specific crawlers are hitting your site.
There are a couple things you can do to reduce the crawler traffic…
-
Try disallowing
/loginfrom crawlers within robots.txt (community.example.com/admin/customize/robots)… you’d probably see some crawler traffic drop (though probably not completely because there are crawlers out there not obeying robots.txt) -
Take a look at the worst offenders from
/admin/reports/web_crawlersand add their user-agents to theblocked crawler user agentssite setting