Hi, In our forum we got too much Too many Crawlers, it’ like 81k to 90k as average! So Is that will cause any problems? if yes how can I disable Bad Crawlers from crawling?
I listed blacklisted crawler user agents as
- mauibot
- semruhbot
- ahrefsbot
- blexbot
- SEO spider
also slow down crawler user agents as
- bingbot
Thanks.