FOOTNOTE:
I was able to rerun the above test with GPT4o (128k context) , making sure to use large token / chunk settings… but it’s still very flaky for my white paper Q/A use case… (lost in the middle, lost at the end , etc.) …here’s my settings if anyone wants to duplicate and refine. .Would love it if we can find the right settings for this case :
| CUSTOM AI PERSONA | |
|---|---|
| Enabled? | Yes |
| Priority | Yes |
| Allow Chat | Yes |
| Allow Mentions | Yes |
| Vision Enabled | No |
| Name | Rag Testing Bot 3 |
| Description | Test RAG vs Long Context prompt injection |
| Default Language Model | GPT-4o-custom |
| User | Rag_Testing_Bot_bot |
| Enabled Commands | Categories, Read, Summary |
| Allowed Groups | trust_level_4 |
| System Prompt | Answer as comprehensively as possible from the provided context on Equatic Carbon Removal Research in the attached file. Do not invent content. Do not use content external to this session. Focus on content provided and create answers from it as accurately and completely as possible. |
| Max Context Posts | 50 |
| Temperature | 0.1 |
| Top P | 1 |
| Uploads | Equatics-paper1-with-unique-haystack-needles-v116.txt |
| Upload Chunk Tokens | 1024 |
| Upload Chunk Overlap Tokens | 10 |
| Search Conversation Chunks | 10 |
| Language Model for Question Consolidator | GPT-4o-custom |
| CUSTOM BOT | |
| Name to display | GPT-4o-custom |
| Model name | gpt-4o |
| Service hosting the model | OpenAI |
| URL of the service hosting the model | https://api.openai.com/v1/chat/completions |
| API Key of the service hosting the model | D20230943sdf_fake_Qqxo2exWa91 |
| Tokenizer | OpenAITokenizer |
| Number of tokens for the prompt | 30000 |