The site has always required login, and does not permit user self-registration. However, it has still been showing up in search engines – just with no content indexed since it couldn’t log in. My hope is that with robots.txt blocking explicitly, any mention will cease. Thanks.
L30110
5
Related topics
| Topic | Replies | Views | Activity | |
|---|---|---|---|---|
| How to disable indexing by crawlers | 15 | 5464 | January 18, 2017 | |
| Pages listed in the robots.txt are crawled and indexed by Google | 18 | 3584 | July 30, 2019 | |
| Excluding user profiles in robots.txt (or allow edit of file) | 4 | 2627 | May 24, 2014 | |
| Need to find a solution to block indexing topics from a category | 2 | 169 | November 27, 2024 | |
| Malformed robots.txt causing issues with indexing | 7 | 2019 | September 14, 2019 |