# Add to robots.txt some CloudFlare path to avoid crawlers inspect incorrectly

**URL:** <https://meta.discourse.org/t/add-to-robots-txt-some-cloudflare-path-to-avoid-crawlers-inspect-incorrectly/242561>\
**Category:** Development\
**Created:** [October 21, 2022, 7:59am UTC](https://meta.discourse.org/t/add-to-robots-txt-some-cloudflare-path-to-avoid-crawlers-inspect-incorrectly/242561 "2022-10-21T07:59:04Z")\
**Posts on this page:** 1\
**Showing post:** 1

<div class="post-metadata">

**Author:** ![albertoiNET](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/albertoinet/32/277536_2.png) [@albertoiNET](https://meta.discourse.org/u/albertoiNET)\
**Post date:** [October 21, 2022, 7:59am UTC](https://meta.discourse.org/t/add-to-robots-txt-some-cloudflare-path-to-avoid-crawlers-inspect-incorrectly/242561/1 "2022-10-21T07:59:04Z")

</div>

First, I want to explain that some sites trust on CloudFlare over Discourse to add extra layer of security. This layer provides some features like Email Ofuscation, and Anti-Bots challenges, that uses a virtual path like:

- [JavaScript detection](https://developers.cloudflare.com/bots/reference/javascript-detections/) used by Cloudflare bot products (`example.com/cdn-cgi/challenge-platform/`)
- [Web analytics](https://developers.cloudflare.com/analytics/web-analytics/getting-started/#sites-proxied-through-cloudflare) for a website proxied through Cloudflare (`example.com/cdn-cgi/rum`)
- [Image resizing](https://developers.cloudflare.com/images/image-resizing/url-format/) in the new URLs you would use for images (`example.com/cdn-cgi/image/`)
- [Email address obfuscationOpen external link](https://support.cloudflare.com/hc/articles/200170016) used to hide email addresses from malicious bots (`example.com/cdn-cgi/l/email-protection`)

In CloudFlare [documentations](https://developers.cloudflare.com/fundamentals/get-started/reference/cdn-cgi-endpoint/), recomed disallow that virtual path `/cdn-cgi/` to avoid that crawlers, search engines, inspect that path.

The main purpouse of this topic, is that I want to contribute on Discourse develop, adding a new PR, but **first I should ask if this is a properly feature on Discourse**.

Thanks for all

---

_[View the full topic](https://meta.discourse.org/t/add-to-robots-txt-some-cloudflare-path-to-avoid-crawlers-inspect-incorrectly/242561)._