### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [June 14, 2024, 5:25am UTC](https://meta.discourse.org/t/improving-quality-of-search-filters-in-discourse-ai/311499/10 "2024-06-14T05:25:31Z")
> [@aas](#):
>
> Should I not expect to see all of the context there?
Yes, debugging tool shows everything we send the LLM.
I just made a fix to number of results per:
>
>
> \- Display filtered search correctly, so it is not confusing
> \- When XML strippingâŠ, if a chunk was \`\<\` it would crash
> \- SQL Helper improved to be better aware of Data Explorer
We were not properly linking to a filtered query.
> [@aas](#):
>
> When calling the search function, always use this value for the category parameter: âdocs,faqsâ
Recommend against this, use base\_query, just tested it works fine: `categories:docs,faqs`
* * *
I do recommend against using GPT4 these days be sure to use GPT4o or Turbo, they are so much cheaper and faster. â
Just double checking ⊠you did enable
`ai_bot_debugging_allowed_groups` so you see the debug button?
---
### Author:  [@aas](https://meta.discourse.org/u/aas)
#### Post date: [June 14, 2024, 11:45pm UTC](https://meta.discourse.org/t/improving-quality-of-search-filters-in-discourse-ai/311499/11 "2024-06-14T23:45:46Z")
> [@sam](#):
>
> Yes, debugging tool shows everything we send the LLM.
> [@sam](#):
>
> Just double checking ⊠you did enable
>
> `ai_bot_debugging_allowed_groups` so you see the debug button?
Yes. Iâll PM you a quick screen share to show that the results of the read function call arenât used in the prompt.
I also noticed that the read function is never called unless I add a note in the system prompt: âalways call the read function.â It would be awesome if there was a way for us to have more control over function calling. For example, [set `tool_choice: "required"`](https://platform.openai.com/docs/guides/function-calling) in the case of OpenAI.
> [@sam](#):
>
> Recommend against this, use base\_query, just tested it works fine: `categories:docs,faqs`
I just tried again with this base query and the the correct search parameters were used. This is the first time Iâve seen it work. đ€·
> [@sam](#):
>
> I do recommend against using GPT4 these days be sure to use GPT4o or Turbo, they are so much cheaper and faster. â
I know, Iâm just experimenting with different models atm đ Was wondering if GPT4 would decide to make different function calls. Haiku sounds very promising.
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [June 15, 2024, 2:34am UTC](https://meta.discourse.org/t/improving-quality-of-search-filters-in-discourse-ai/311499/12 "2024-06-15T02:34:09Z")
Thanks heaps
oh, that is correct, system prompt injection will only happen if the persona has uploads. If there are no uploads then we donât inject anything
Without uploads we donât use unconditional system prompt injection and instead rely on tool calls with tool call results, you can see them when scroll through.
A tool free mode targeted at topics is certainly interesting, it would eat a fair bit of tokens, the challenge though is fishing the right content out of a topic.
If a topic for example is 5000 tokens, which tokens would we pick? how would we pick the right tokens.
As it stands we only have N tokens embedded per topic depending on the embedding model you are using.
So we would need to start embedding more chunks per topic or come up with an algorithm to figure out what part of the topic is the most interesting.
---
### Author:  [@aas](https://meta.discourse.org/u/aas)
#### Post date: [June 26, 2024, 4:29pm UTC](https://meta.discourse.org/t/improving-quality-of-search-filters-in-discourse-ai/311499/13 "2024-06-26T16:29:59Z")
> [@sam](#):
>
> oh, that is correct, system prompt injection will only happen if the persona has uploads. If there are no uploads then we donât inject anything
>
> Without uploads we donât use unconditional system prompt injection and instead rely on tool calls with tool call results, you can see them when scroll through.
Iâm confused. If the results of invoking the read tool arenât used for RAG, what does invoking the read tool even do? I see the results of the read tool call are put into the [raw context](https://github.com/discourse/discourse-ai/blob/e39e0bdb4a9c5683d2baacf6f8863caf0015e7cc/lib/ai_bot/bot.rb#L141-L142), but it doesnât seem like this is actually being used by the LLM in any way.
> [@sam](#):
>
> A tool free mode targeted at topics is certainly interesting, it would eat a fair bit of tokens, the challenge though is fishing the right content out of a topic.
>
> If a topic for example is 5000 tokens, which tokens would we pick? how would we pick the right tokens.
My desired use case is simple: **enable RAG on a subset of topics as per the `base_query`.**
I think it would be best not to rely on Discourse to handle splitting my topics into multiple chunks, each with their own embedding. Rather, I want to be responsible for crafting my Docs and FAQs such that they are optimized for RAG. I.e., the entirety of the topics main post should be embeddable. Iâm talking about wiki topics here, the replies to which can and should be disregarded.
ada-002 has a [max token input of 8191](https://platform.openai.com/docs/guides/embeddings/embedding-models), which is plenty for any well-written doc or faq topic. Such topics should be well under this limit. [voyage ai](https://docs.voyageai.com/docs/embeddings#model-choices) has a much larger context length. of 16000.
> [@sam](#):
>
> So we would need to start embedding more chunks per topic or come up with an algorithm to figure out what part of the topic is the most interesting.
What about something as simple as this:
- `(tokens in the system prompt - tokens in user message) = total remaining tokens available`
- inject as many topics into the prompt as possible based on `total remaining tokens available`
- use only the first post in the found topics
I guess the near-term workaround could be the following:
- create a persona that uses the search tool
- export all the topics that could be found with the search tool
- I wonât need to split these docs up because
- I donât care about topic replies
- The main wiki topic is already small enough to embed the entire thing.
- Upload those documents to the persona for RAG
This way, the persona will actually do RAG with the search results.
I hope this makes sense! LMK if anything is unclear.
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [June 27, 2024, 1:45am UTC](https://meta.discourse.org/t/improving-quality-of-search-filters-in-discourse-ai/311499/14 "2024-06-27T01:45:30Z")
> [@aas](#):
>
> Iâm confused. If the results of invoking the read tool arenât used for RAG
When you read stuff it is injected into the conversation stream.
Eg:
- Can you read X
- Tool call read X
- Tool result âthe pageâ
- LLM reply
RAG is a very overloaded term, but at the end reading a topic adds it to the context, the llm at the end is responsible for how to use it, various llms may make better or worse usage out of it.
Eg: [[Untitled AI bot PM] - AI Conversation - Discourse Meta](https://meta.discourse.org/discourse-ai/ai-bot/shared-ai-conversations/UrUqWYwu9Lzst5qrYwrbzA)
Search tool already inserts excerpts of topics, having it and a RAG on the same content does not make sense.
What you are saying, I guess, is that âsearch is giving context, but not **enough context** â.
So one workaround is to export all the important topics into a file, then use it as a RAG. Do not use search tool.
* * *
Simplest thing for now is:
`categories:cat1,cat2,cat3 in:first`
This will restrict the search tool to first posts only.
It is very efficient, see:
> **[Twitter Authentication Setup: A Quick Configuration Guide](https://meta.discourse.org/discourse-ai/ai-bot/shared-ai-conversations/xw47vzgTQExpyYD777J5mQ)**
>
> sam: how do I configure twitter auth?
This works cause the search tool injects the following into the context:
* * *
I also want an unconditional âragâ mode where you can opt to inject even more data (eg: find top 5 documents and inject up to 5000 tokens per document, trying to find most relevant sections)
Its a complicated change, we will get there, but for now ⊠search + read gets you quite far.
---
### Author:  [@Jagster](https://meta.discourse.org/u/Jagster)
#### Post date: [June 27, 2024, 7:15am UTC](https://meta.discourse.org/t/improving-quality-of-search-filters-in-discourse-ai/311499/15 "2024-06-27T07:15:26Z")
Just saying⊠you guys are very close to that point where you must to do a introducing what are basic rules how to use AI. I mean you have already similar one of CSS and a couple other thing.
---
### Author:  [@aas](https://meta.discourse.org/u/aas)
#### Post date: [June 27, 2024, 9:32pm UTC](https://meta.discourse.org/t/improving-quality-of-search-filters-in-discourse-ai/311499/16 "2024-06-27T21:32:34Z")
> [@sam](#):
>
> RAG is a very overloaded term, but at the end reading a topic adds it to the context, the llm at the end is responsible for how to use it, various llms may make better or worse usage out of it.
I see. That makes sense. From [OpenAIâs docs on function calling](https://platform.openai.com/docs/guides/function-calling):
> The basic sequence of steps for function calling is as follows:
> 1. Call the model with the user query and a set of functions defined in the [functions parameter](https://platform.openai.com/docs/api-reference/chat/create#chat/create-functions).
> 2. The model can choose to call one or more functions; if so, the content will be a stringified JSON object adhering to your custom schema (note: the model may hallucinate parameters).
> 3. Parse the string into JSON in your code, and call your function with the provided arguments if they exist.
> 4. Call the model again by appending the function response as a new message, and let the model summarize the results back to the user.
Critically, I wasnât aware of #4. I didnât realize that the results of invoking the tool were appended as messages. I thought it was entirely up to the client if the results were used in subsequent calls! So I was confused when I didnât see tool call results in the prompt itself. :facepalm: This all makes a lot more sense now. Thank you!
> [@sam](#):
>
> So one workaround is to export all the important topics into a file, then use it as a RAG. Do not use search tool.
đ đ Sounds like the only downside here is that bot replies wonât include those nice search results. Would you strongly recommend using doc uploads _and_ the search tool for any reason, even though it will be a bit redundant? I wonder if the benefits of getting the search results in the convo will be worth it.
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [June 28, 2024, 12:28am UTC](https://meta.discourse.org/t/improving-quality-of-search-filters-in-discourse-ai/311499/17 "2024-06-28T00:28:09Z")
> [@aas](#):
>
> . Would you strongly recommend using doc uploads _and_ the search tool for any reason, even though it will be a bit redundant?
You need to test it, it could work, it could give you surprising results and so much depends on the LLM you are using.