# Web crawler question

**URL:** https://meta.discourse.org/t/web-crawler-question/199178
**Category:** Support
**Created:** [August 4, 2021, 8:03pm UTC](https://meta.discourse.org/t/web-crawler-question/199178 "2021-08-04T20:03:58Z")
**Posts on this page:** 3
**Page:** 1

<div class="post-metadata">

### Author: ![Jim.Morrison](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/jim.morrison/32/162161_2.png) [@Jim.Morrison](https://meta.discourse.org/u/Jim.Morrison)
#### Post date: [August 4, 2021, 8:03pm UTC](https://meta.discourse.org/t/web-crawler-question/199178/1 "2021-08-04T20:03:59Z")

</div>

Good afternoon,

is there any specific way an HTTP request can identify as a web crawler request to the Discourse server? The reason is the website uses elements like infinite scroll and so had a question on whether the page would look different when a crawler request is made. Thanks in advance!

---

<div class="post-metadata">

### Author: ![Falco](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/falco/32/179432_2.png) [@Falco](https://meta.discourse.org/u/Falco)
#### Post date: [August 4, 2021, 8:12pm UTC](https://meta.discourse.org/t/web-crawler-question/199178/2 "2021-08-04T20:12:49Z")

</div>

You can include the `bot` string in the User-Agent header in order to get our special crawler view.

---

<div class="post-metadata">

### Author: ![system](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/system/32/443519_2.png) [@system](https://meta.discourse.org/u/system)
#### Post date: [September 3, 2021, 8:13pm UTC](https://meta.discourse.org/t/web-crawler-question/199178/3 "2021-09-03T20:13:46Z")

</div>

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.