Web crawler question

Support
1

Good afternoon,

is there any specific way an HTTP request can identify as a web crawler request to the Discourse server? The reason is the website uses elements like infinite scroll and so had a question on whether the page would look different when a crawler request is made. Thanks in advance!

2

You can include the bot string in the User-Agent header in order to get our special crawler view.

1 Like
3

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.