# Web Crawlers

**URL:** <https://meta.discourse.org/t/web-crawlers/271767>\
**Category:** Data & reporting\
**Created:** [July 16, 2023, 6:41am UTC](https://meta.discourse.org/t/web-crawlers/271767 "2023-07-16T06:41:00Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![pfaffman](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/pfaffman/32/120154_2.png) [@pfaffman](https://meta.discourse.org/u/pfaffman)\
**Post date:** [July 16, 2023, 9:57am UTC](https://meta.discourse.org/t/web-crawlers/271767/2 "2023-07-16T09:57:46Z")

</div>

Search site settings for “crawler”. You can block or slow down by user agent.

> [@anon65426961](#):
>
> don’t like these little buggers what are they doing?

At least some index your site to have it appear in search engines. You probably do like that.

---

_[View the full topic](https://meta.discourse.org/t/web-crawlers/271767)._