An interesting conclusion I am coming to with our daily reports (which are now wired for TL3 on meta)
The piece that people love most that is LLM driven is the simple 1 paragraph recap of changes in a topic from date X to Y.
Instead of getting the LLM to read through the entire corpus and create a report like this we could just keep a “fragment cache” where say we store, paragraphs per topic:
| topic id | date range for summary | very short summary |
|---|---|---|
| 2343 | 12-1 → 12-9 | … |
With that we can construct custom summaries per user. So we are basically splitting this into multiple problems:
-
Identifying interesting topics → traditional programming (watched tags / categories or maybe “background” watched based on activity - exclude already read)
-
Recapping topics → LLM powered feature
This has multiple advantages. It scales very well, summaries can be much more accurate cause the amount of context is far more limited and there is less risk of context bleed and hallucination.