# Controlling Web Crawlers For a Site

**URL:** https://meta.discourse.org/t/controlling-web-crawlers-for-a-site/244925
**Category:** Site Management
**Tags:** how-to
**Created:** [November 9, 2022, 12:04pm UTC](https://meta.discourse.org/t/controlling-web-crawlers-for-a-site/244925 "2022-11-09T12:04:55Z")
**Posts on this page:** 1
**Showing post:** 18

<div class="post-metadata">

### Author: ![Jagster](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/jagster/32/192154_2.png) [@Jagster](https://meta.discourse.org/u/Jagster)
#### Post date: [July 19, 2025, 7:51am UTC](https://meta.discourse.org/t/controlling-web-crawlers-for-a-site/244925/18 "2025-07-19T07:51:39Z")

</div>

There can be several reasons, but googlebot has its budget, and when sitemaps are the most important way to find links, it never reaches internal links when daily/weekly/monthly budget is used.

And in a forum internal links are important for users, not for Google.

But I don’t know if googlebot sees internal links. It should, though.

---

_[View the full topic](https://meta.discourse.org/t/controlling-web-crawlers-for-a-site/244925)._