I was looking at our forum in ahrefs which reports on “AI Citations” and saw this: [image] I checked in my settings and found this: [image] Does anyone in here have experience with this? Does throttling the bots mean they don’t index the forum at all? Has anyone removed throttling that can speak to the impact on resources and AI citations? Thanks.

No, it doesn’t mean. Throttling stops, depending on how it has been done, because robots.txt is toothless, bots to act as mini DDOS attacks. There are three reasons why you don’t get any AI citations: The same information is found in more popular places No one has even asked something that would lead to a citation to your forum URL Ahrefs is wrong Ahrefs is one of those companies whose bot should be banned in the first place. That is one reason among many why the assumption ”Ahrefs is wrong” is one of those three.

And as far as I know, they are blocked by default in Discourse: [Screenshot 2025-08-08 at 09.58.18] So yeah, the ahrefs report is probably unreliable. Why they choose not to tell you that they weren’t able to get any data, but instead erronously report “0”.. now that’s a question for them. You can unblock their bot, but be prepared for the occasional mini-ddos, increased pageviews and server load.

The ahrefs bot doesn’t need to be able to index our forum for this to work as it’s only reporting on what it sees on Google.

It would be very unusual if that were the case as our forum gets a couple million hits a year … which is why I’m wondering about active AI bot blocking. Or maybe the framework is otherwise technically unfriendly to the AI scrapers, like say the js rendering etc.

Can you get the bots to actually cite your forums? If so, the problem is ahrefs. This remains my guess, the AI Citations feature is fairly new and might not work yet. If you cannot, something else might be amiss. In that case, try a few other Discourse forums. If they do show up, then the problem isn’t the framework, but something specific to your site. For what it’s worth, here’s what ChatGPT gives me for an example query about Meta: [Screenshot 2025-08-08 at 13.40.30]

[תמונה] abrambailey: It would be very unusual if that were the case as our forum gets a couple million hits a year No it isn’t. And that isn’t that many. I got 3,5 million a year, if your hits means search hits from Google. If it means user visits, it isn’t that much either. If it means bot visits, it is even low. Plus hits doensn’t mean any of AI would tell your forum is defenetly used as a cited source. It depends what type content you have there. But yet, it isn’t such thing what you can ask here. Only Ahrefs knows how they are counting theirs metric. Plus Ahrefs doesn’t see via Google how many times an AI has cited

[תמונה] Jagster: And that isn’t that many. I got 3,5 million a year So you saying 3.5M is also not “that many” or does the number become significant somewhere between 2 and 3.5 million? Lol. Unique visitors from google btw.. and our other site with far fewer visitors is all over AI Answers… and ahrefs is reporting it (who knows how accurately)… it’s just weird to have a bigger site with 0 for all citations when I have a smaller one with a lot more… Anyway, I’d love to hear from any Discourse webmasters who have any direct knowledge of this topic. Is anyone else using ahrefs and seeing non-zeroes for AI citations for your forum(s)?? Thank you.

[תמונה] abrambailey: So you saying 3.5M is also not “that many” or does the number become significant somewhere between 2 and 3.5 million? Lol. Or should I say it was very dumb counting error :man_facepalming: But my defense is that it was night here when I calculated that. Sorry. I get 10 000 per month, and that isn’t that much — but it is significant less per year that 3,5 million in deed.

Possibly relevant: https://www.gsqi.com/marketing-blog/ai-search-javascript-rendering/

This is not likely to be a problem for Discourse forums as it will render a crawler view for any crawlers using a bot-like user-agent. If they’re allowed, of course.

ציטוטים של AI - האם Discourse מאפשר סריקת בוטים של AI?

תמיכה

abrambailey (Abram Bailey) 8 באוגוסט,‏ 2025,‏ 11:16am 4

בוט ahrefs אינו צריך להיות מסוגל לאנדקס את הפורום שלנו כדי שזה יעבוד מכיוון שהוא מדווח רק על מה שהוא רואה בגוגל.

נושא		תגובות	צפיות
Discourse is Agent Ready: Here’s How Blog	9	844	24 במאי,‏ 2026
Tracking LLM Crawler Activity: Can we identify which topics are being indexed/crawled most frequently? Data & reporting data-explorer	3	236	21 באפריל,‏ 2026
I populated my (mostly-dead) Discourse forum with bots! General show-and-tell	15	845	3 במאי,‏ 2026
MegaIndex bot did about 4,000 pageviews on one day Community Building	40	5113	2 בדצמבר,‏ 2023
AI Forum Moderation: Seeking Insights and Experiences Development ai	8	2236	27 בספטמבר,‏ 2025

ציטוטים של AI - האם Discourse מאפשר סריקת בוטים של AI?

נושאים קשורים