Yeah it is a very tricky problem.
Fitting the entire world into 6000 or so words that GPT-4 8k is allowed to know about is a very very hard problem.
I am borderline on just bumping lola to use 32k tokens here, but the costs are really really high and I want to be testing stuff that is closer to what the general public are using.
The current workaround/solution for this problem is function calling, you get GPT-4 to reason about what information it will need and then a few round trips later it finds the right context. This can involve searching (either using embeddings or just pure keyword search)
I don’t anticipate being able to solve “I am not a programmer, make a plugin for me” solution any time soon.
That said I can see lola getting better at helping people who have a reasonable foundation around programming - especially stuff like semantically searching through our code base and so on.
We also have access to Anthropic Claude here which comes with 80 thousand or so words of context, but sadly its performance is much closer to GPT 3.5 than to 4 and it is just very very very hard to steer it.
Making progress slowly in this uncharted territory…