I checked the audit logs and it returned an empty response. I re-triggered the rule and it worked this time.
5 Likes
Hi,
Not sure where to report this or if it’s worth reporting, however, it seems that the Discourse AI summary feature hallucinates - quite a few times.
Example:
A topic was created, there were no replies nor interactions from users. No solutions were discussed nor investigated. See post.
Yet, the bot confidently states “Some options were discussed but no definitive solution.”
Is there no way to filter this so that if a post does not have replies, the summary is not imagined?
1 Like
If you mean ‘can you get Bert to stop hallucinating’ then no, that is not something I can consistently do. 
I think the underlying tools get better all the time though, and accuracy improves along with them.
5 Likes
I do plan to eventually move to amalgamation of topic summaries for the topic summaries section
It can eliminate a bunch of problems, but increasing reliability here with the current tools will certainly increase cost a lot, you would need to generate and then run rounds of validation, could get very costly
2 Likes
On Daily Summary (5am UTC) - #149 by Discourse and Daily Summary (5am UTC) - #148 by Discourse, it seems that some topics have been spelt incorrectly. Instead of #site-feedback, it was #site feedback, hence the topic did not get the link. Then, instead of #dev:translations, it said #translations. Sinilarly, it entered #Self-hosting instead of #documentation:self-hosting.
Yes, this is a known thing. It’s not really worth the effort to manually edit them or spend a lot of time tweaking the prompt to work around it. 
3 Likes
Bert has been eating the wrong mushrooms again
5 Likes
But… but you didn’t even post in that topic lol.
(Also, what Bert and mushrooms?)
Bert generally seems to be the nickname used for Meta’s AI bot.
1 Like
Seems like this was fixed? AND formatted the same way as the original few posts? (For this one, but seems like the 1pm one has the caregories fixed, though it doesn’t look like the first few (where caregories are bolded))
We moved to Claude 3.5 Sonnet so the quality is a bit better.
3 Likes
Did AI just give up putting them (categories) in a class ‘mention’? Or was the prompt changed?
EDIT: Looks like it fixed itself in the next post.
It seems that AI did not group topics of the same category together, so the post has many many topics, with it in no apparent order.
(Not sure if this post belongs here)