Disclaimer: I’m a de facto end user and I don’t even understand how AIs really work. And I use OpenAI.
There are some reasons why AI isn’t giving the wanted answer.
- Prompting dictates where and how it can search. One bad wording and it will do whatever it wants
- AI isn’t like Google with steroids and skill to explain things, even if it kind of is. It can find the right hits as well, and I mean as badly, as Google. And it doesn’t actually read and analyze everything, but it just… thinks so.
- RAG and embeddings work, but need extremely tight prompting. But those give only a direction, quite often not a steady base to build up an answer. Just embeddings need a lot of manual labor and quite often topics, again per se, are not enough. A topic or a post (even worse) can be accurate and logical enough, but out there in real life? No. That’s why ask.discourse.com fails quite often, if not asked a very limited and targeted question. How do I allow only specific email domains in registrations? Boom, you have the answer. How do I get notifications when a group PM-box has new messages? A lot of hallucination and wrong refs.
The most disturbing idea per OpenAI is that wrong answers are acceptable. It is a matter of amount and specifically how much those hallucinated and factually wrong answers will cost to a company.
Very true for companies, but really bad for that one user.
AIs can be very accurate. All that is needed is a lot of manpower to code and take care of that, and so much computing power that mining bitcoins is a cheap hobby.
My very weak point is that just dropping manuals into topics isn’t enough.