# Dark Visitors
**URL:** https://meta.discourse.org/t/dark-visitors/365158
**Category:** Plugin
**Created:** [May 7, 2025, 3:30pm UTC](https://meta.discourse.org/t/dark-visitors/365158 "2025-05-07T15:30:42Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@elmuerte](https://meta.discourse.org/u/elmuerte)
#### Post date: [May 7, 2025, 3:30pm UTC](https://meta.discourse.org/t/dark-visitors/365158/1 "2025-05-07T15:30:42Z")
| | | |
| --- | --- | --- |
| ℹ | **Summary** | Integrates the Known Agents service with Discourse to keep track of undesired crawlers and scrapers visiting your forum. |
| 🛠 | **Repository Link** | [https://github.com/magicball-network/discourse-darkvisitors](https://github.com/magicball-network/discourse-darkvisitors) |
| 📖 | **Install Guide** | [How to install plugins in Discourse](https://meta.discourse.org/t/install-plugins-in-discourse/19157) |
> ℹ As of [February 3rd 2026](https://knownagents.com/posts/dark-visitors-is-now-known-agents) the _Dark Visitors_ service has been rebranded to _Known Agents_. The Discourse plugin has not be renamed.
### Features
[Known Agents](https://knownagents.com/) is a service that keeps track of [agents](https://darkvisitors.com/agents) (crawlers, scrapers, and other kinds of bots) visiting your websites. Their main attention is towards analyzing AI agents.
It offers two services:
- robots.txt generation and monitoring
- agent analytics; both server side and client side
The robots.txt service is gratis. The analytics services provides a free tier. I suggest you visit [their website](https://darkvisitors.com/) for more information.
This Discourse plugin connects to all these services, all of them optional and configurable to a degree.
#### robots.txt generation
Discourse already provides an option to configure the robots.txt. This plugin extends it. When enabled the plugin retrieves a list of agents in different categories (currently only AI categories are supported) and those which are missing in the already configured agents, it will add them. The list will be updated daily. This way when a new AI scrapper is recognized it will be added to your robots.txt.
This feature only works if Discourse manages the robots.txt and you have not manually overridden it. The plugin does not change the _Blocked crawler user agents_ setting, it augments the robots.txt new missing agents. So you are still in full control of managing this list.
When you visit your site’s robots.txt you will see a leading comment of the last update, and the number of agents returned by Known Agents. The not already configured agents are added to the end of the list. They should be between Googlebot and the sitemap directive (if configured).
#### Agent analytics
Both server side and client side analytics can be individually enabled. It can be enabled for all visitors, or only unauthenticated visitors.
The server side analytics reports _tracked visits_ to Known Agents. It will send the request path, the remote address of the visitor, the User-Agent header, and a few [more browser headers](https://github.com/magicball-network/discourse-darkvisitors/blob/main/lib/server_analytics.rb#L8).
There are some additional settings to which request are reported, see the settings below. By default only request which Discourse marks to track as views will be tracked. The following requests to Discourse will never be reported:
- Request to the Admin section
- Background and API requests
The client side analytics are handled by adding [Javascript](https://darkvisitors.com/tracker.js) to your page which calls home to Known Agents under certain conditions:
- The browser appears to be automated, or an AI browser
- The user came from an AI chat service
All tracked requests count towards the events which affects your payment plan.
### Configuration
You need to sign up with [Known Agents](https://knownagents.com/) in order to use this plugin. The free tier gives you 100,000 events per month. When that limit is reached you will not see any newer events in their analytics, but you can still send new information and keep using the robots.txt service.
After you sign up you must create a project (i.e. a site to track). This will provide you with an _access token_ which is needed for the robots.txt and server side analytics functionality.
When you enable the robots.txt functionality it takes a short while before it is updated. Visit `https://yoursite/robots.txt` to see if it is working. It should have a comment at the top
```plaintext
# Augmented by Dark Visitors on 2025-05-07T12:46:00+00:00 with 28 agents
```
When you enable the service side analytics you can test to see if it works by requesting a _test visit_ from the Known Agents’s project settings. It can take a few seconds. You should see the result in the _Realtime_ page on Known Agents.
### Settings
| Name | Description |
| --- | --- |
| darkvisitors enabled | Global flag to enable the whole plugin |
| darkvisitors access token | The secret access token needed for the robots.txt and server side analytics in order to communicate with Known Agents. You will find this in your Known Agents’s project under _settings_. |
| darkvisitors robots txt enabled | When enabled the Disocurse robots.txt will be augmented with additional agents |
| darkvisitors robots txt agents | The kind of agents to add to the robots.txt. |
| darkvisitors robots txt path | The path to deny the agents access to. It is probably best to leave this at `/` so access to the whole site is rejected. |
| darkvisitors server analytics | Enables server side analytics. I recommend to only enable it for anonymous users. |
| darkvisitors server analytics include | Additional requests track. You can also track requests to the uploaded files, or even 404 Not Found request. |
| darkvisitors server analytics ignore | Sub-strings in the user agents to ignore (case sensitive). If you use uptime monitoring I strong suggest to include their identifying user agent in this list. |
| darkvisitors client analytics | Enable client side analytics. This will also give you insights of normal users visiting your forum while coming from an AI chat service. |
| darkvisitors client analytics project key | For client side analytics you must configure the (public) project key. You can find this in your Known Agents project settings in the section _JavaScript Tag_, it is the code after `project_key=` |
---
### Author:  [@ToddZ](https://meta.discourse.org/u/ToddZ)
#### Post date: [September 23, 2025, 4:16am UTC](https://meta.discourse.org/t/dark-visitors/365158/4 "2025-09-23T04:16:05Z")
Thanks for this, elmuerte! I’ve set it up and it’s working great.
I see that in the plugin settings, the agent types that can be selected for exclusion via robots.txt are:
- AI Data Scraper _[selected by default]_
- Undocumented AI Agent _[selected by default]_
- AI Agent
- AI Assistant
- AI Search Crawler
But the complete list of Dark Visitors agent types, per `darkvisitors.com`, is:
( **bold** = additional)
Crawlers & Scrapers…
- AI Assistant
- **AI Data Scraper**
- AI Search Crawler
- **Archiver**
- **Developer Helper**
- **Fetcher**
- **Intelligence Gatherer**
- **Scraper**
- **Search Engine Crawler**
- **Security Scanner**
- **SEO Crawler**
- **Uncategorized Agent**
- Undocumented AI Agent
AI Agents…
- AI Agent
- **Headless Agent**
Not all of these agent types are things one would want to block, but I’d like to include a few like Scraper, AI Data Scraper, SEO Crawler…
Are these additional agent types just newer than your plugin? Could they be added to the current list choices in settings.yml?
---
### Author:  [@Jagster](https://meta.discourse.org/u/Jagster)
#### Post date: [September 23, 2025, 4:47am UTC](https://meta.discourse.org/t/dark-visitors/365158/5 "2025-09-23T04:47:13Z")
> [@ToddZ](#):
>
> selected for exclusion via robots.txt are
Except robots.txt is just a request. A bot follows it or not. Firewall is the only way to stop those.
---
### Author:  [@ToddZ](https://meta.discourse.org/u/ToddZ)
#### Post date: [September 23, 2025, 4:55am UTC](https://meta.discourse.org/t/dark-visitors/365158/6 "2025-09-23T04:55:23Z")
Yep, I understand that – but since Dark Visitors only works with robots.txt, I’d like to make it work as well as it can.
(I’m actually reading a couple of posts right now where you suggest real blocking with Nginx reverse proxy, but I’m not sure if I need to go that far yet.)
---
### Author:  [@Jagster](https://meta.discourse.org/u/Jagster)
#### Post date: [September 23, 2025, 5:00am UTC](https://meta.discourse.org/t/dark-visitors/365158/7 "2025-09-23T05:00:38Z")
> [@ToddZ](#):
>
> you suggest real blocking with Nginx reverse proxy, but I’m not sure if I need to go that far yet
That is a bit hard core. But Dark Visitor should work with banlist of Discourse to be usefull at some level. Sure, with that you don’t need to add manually i.e. OpenAI or else that follows robots.txt.
---
### Author:  [@elmuerte](https://meta.discourse.org/u/elmuerte)
#### Post date: [September 23, 2025, 8:04pm UTC](https://meta.discourse.org/t/dark-visitors/365158/8 "2025-09-23T20:04:46Z")
> [@ToddZ](#):
>
> Not all of these agent types are things one would want to block, but I’d like to include a few like Scraper, AI Data Scraper, SEO Crawler…
I contacted Dark Visitors about this on may 3rd this year, and their response was “Not at the moment”. But I see the current documentation lists even more types now.
At this moment, the following types are supported by the Dark Visitors API:
> [@](#):
>
> - AI Agent
> - AI Assistant
> - AI Data Scraper
> - AI Search Crawler
> - Archiver
> - Developer Helper
> - Fetcher
> - Headless Agent
> - Intelligence Gatherer
> - Scraper
> - SEO Crawler
> - Search Engine Crawler
> - Security Scanner
> - Undocumented AI Agent
> - Uncategorized
I made sure the setting in Discourse can be extended with additional agent types by just adding it.
After adding the new type and saving the setting the robots.txt should be updated right away will all the new agents.
---
### Author:  [@ToddZ](https://meta.discourse.org/u/ToddZ)
#### Post date: [September 23, 2025, 9:22pm UTC](https://meta.discourse.org/t/dark-visitors/365158/9 "2025-09-23T21:22:24Z")
OMG, I totally missed the “Search or create” field. My theme has a really low contrast there and it escaped my eyes. Thank you for the clarification!
