Just to clarify, is there no way to unblock semrushbot and seo spider? We need them for SEO audit. Tried removing both from /admin/customize/robots (also tried Allow: ) but we get 429 error in Screaming Frog. Or is this 429 error a separate issue? Your insights are highly appreciated.
trying2survive
(John)
10
Related topics
| Topic | Replies | Views | Activity | |
|---|---|---|---|---|
| Handling Bingbot | 28 | 7762 | November 20, 2020 | |
| MegaIndex bot did about 4,000 pageviews on one day | 40 | 5064 | December 2, 2023 | |
| Issues Google Search Console is throwing at me for wrong discourse structure (or some for wrong administration of my site) | 18 | 403 | December 18, 2024 | |
| Why there are lots of Disallow rule in robots.txt? | 34 | 4961 | December 22, 2020 | |
| How to protect myself from bots crawling my Discourse instance? | 4 | 1719 | January 21, 2019 |