# Web Crawler user agents - Mastodon agents growing in number

**URL:** https://meta.discourse.org/t/web-crawler-user-agents-mastodon-agents-growing-in-number/386506
**Category:** Community Building
**Created:** [October 23, 2025, 9:04am UTC](https://meta.discourse.org/t/web-crawler-user-agents-mastodon-agents-growing-in-number/386506 "2025-10-23T09:04:45Z")
**Posts on this page:** 1
**Showing post:** 2

<div class="post-metadata">

### Author: ![pmusaraj](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/pmusaraj/32/119489_2.png) [@pmusaraj](https://meta.discourse.org/u/pmusaraj)
#### Post date: [October 23, 2025, 6:22pm UTC](https://meta.discourse.org/t/web-crawler-user-agents-mastodon-agents-growing-in-number/386506/2 "2025-10-23T18:22:44Z")

</div>

> [@EricGT](#):
>
> After some more research and talking with other admins on our site, we welcome these agents as they are used to generate link previews.

Cool, that likely means that something in your community got reposted by users on Mastodon. Because Mastodon is federated, the link preview crawlers will have different user agents. Both because the instances will be on different Mastodon versions and because it looks like Mastodon includes the URL of the community as part of the user agent.

> [@EricGT](#):
>
> it would be nice to see a new report that shows just the fetcher agents

Agreed. It could also be neat to group the user agents, so that you can see Mastodon link preview totals, Facebook onebox totals, Discourse onebox totals (from other communities) and so on.

---

_[View the full topic](https://meta.discourse.org/t/web-crawler-user-agents-mastodon-agents-growing-in-number/386506)._