Add to robots.txt some CloudFlare path to avoid crawlers inspect incorrectly

AI-generated summary

A discussion on adding CloudFlare paths to Discourse’s robots.txt to prevent crawlers from incorrectly inspecting virtual paths. albertoiNET explains that CloudFlare uses virtual paths for features like JavaScript detection, web analytics, image resizing, and email address obfuscation, and recommends disallowing the /cdn-cgi/ path to avoid crawler issues. However, pfaffman is skeptical, stating that Cloudflare’s optimizations often break Discourse and these paths may not be relevant.

First, I want to explain that some sites trust on CloudFlare over Discourse to add extra layer of security. This layer provides some features like Email Ofuscation, and Anti-Bots challenges, that uses a virtual path like:

In CloudFlare documentations, recomed disallow that virtual path /cdn-cgi/ to avoid that crawlers, search engines, inspect that path.

The main purpouse of this topic, is that I want to contribute on Discourse develop, adding a new PR, but first I should ask if this is a properly feature on Discourse.

Thanks for all

Cloudflare’s speedups often break Discourse. I don’t think any of those things matter.