# Web Crawlers

**URL:** <https://meta.discourse.org/t/web-crawlers/271767>\
**Category:** Data & reporting\
**Created:** [July 16, 2023, 6:41am UTC](https://meta.discourse.org/t/web-crawlers/271767 "2023-07-16T06:41:00Z")\
**Posts on this page:** 1\
**Showing post:** 5

<div class="post-metadata">

**Author:** ![Ed\_S](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/ed_s/32/134015_2.png) [@Ed\_S](https://meta.discourse.org/u/Ed_S)\
**Post date:** [July 16, 2023, 6:58pm UTC](https://meta.discourse.org/t/web-crawlers/271767/5 "2023-07-16T18:58:17Z")

</div>

Worth noting that user agents are totally easy to spoof. If it’s Google, you can be pretty sure it will say so. But just because it says it’s Google, that means nothing.

(Same situation as robots.txt, these are mechanisms which assume trust. Untrustworthy parties can just play by different rules.)

---

_[View the full topic](https://meta.discourse.org/t/web-crawlers/271767)._