# What exactly are crawlers?

**URL:** https://meta.discourse.org/t/what-exactly-are-crawlers/362811
**Category:** Data & reporting
**Created:** [April 21, 2025, 2:41pm UTC](https://meta.discourse.org/t/what-exactly-are-crawlers/362811 "2025-04-21T14:41:05Z")
**Posts on this page:** 6
**Page:** 1

<div class="post-metadata">

### Author: ![Mittens](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/mittens/32/485149_2.png) [@Mittens](https://meta.discourse.org/u/Mittens)
#### Post date: [April 21, 2025, 2:41pm UTC](https://meta.discourse.org/t/what-exactly-are-crawlers/362811/1 "2025-04-21T14:41:05Z")

</div>

So im seeing this

 ![image](https://global.discourse-cdn.com/meta/original/4X/c/2/5/c25be6cabad2fb596808c75282653fb83d09c03b.png)  
and I was wondering what crawlers are?

---

<div class="post-metadata">

### Author: ![nat](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/nat/32/235063_2.png) [@nat](https://meta.discourse.org/u/nat)
#### Post date: [April 21, 2025, 2:51pm UTC](https://meta.discourse.org/t/what-exactly-are-crawlers/362811/2 "2025-04-21T14:51:46Z")

</div>

Crawlers would be things like Google Search automating “crawling” your website for content so that Google Search can present your site when you search for it.

There are also other crawlers besides Google’s.

You should check out this post (and the link in it) for more info: [Better pageview metrics with the new site traffic report](https://meta.discourse.org/t/better-pageview-metrics-with-the-new-site-traffic-report/324068)

---

<div class="post-metadata">

### Author: ![Hacxx](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/hacxx/32/370225_2.png) [@Hacxx](https://meta.discourse.org/u/Hacxx)
#### Post date: [April 22, 2025, 2:09pm UTC](https://meta.discourse.org/t/what-exactly-are-crawlers/362811/3 "2025-04-22T14:09:28Z")

</div>

For simplest understanding, crawlers are bots that crawl pages online and follow links to crawl even more pages. Mostly used by search engines to build the index, there is other crawlers online for example wayback machine, etc.

---

<div class="post-metadata">

### Author: ![Jagster](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/jagster/32/192154_2.png) [@Jagster](https://meta.discourse.org/u/Jagster)
#### Post date: [April 22, 2025, 2:18pm UTC](https://meta.discourse.org/t/what-exactly-are-crawlers/362811/4 "2025-04-22T14:18:18Z")

</div>

Most of them are actually SEO and AI companies, not search engines or other useful actors. And part X of those search bots are actually scrapers seaching weaknesses and stealing content.

On my forum and sites anyway.

---

<div class="post-metadata">

### Author: ![Mittens](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/mittens/32/485149_2.png) [@Mittens](https://meta.discourse.org/u/Mittens)
#### Post date: [April 22, 2025, 6:18pm UTC](https://meta.discourse.org/t/what-exactly-are-crawlers/362811/5 "2025-04-22T18:18:26Z")

</div>

> [@Jagster](#):
>
> seaching weaknesses and stealing content.

typical,people are always trying to steal other people’s work.

---

<div class="post-metadata">

### Author: ![tobiaseigen](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/tobiaseigen/32/539204_2.png) [@tobiaseigen](https://meta.discourse.org/u/tobiaseigen)
#### Post date: [April 26, 2025, 4:26pm UTC](https://meta.discourse.org/t/what-exactly-are-crawlers/362811/6 "2025-04-26T16:26:39Z")

</div>