benchmarks evals
A Correct Answer Can Hide a Broken Memory
FEB-30 inspects the store after each fact change instead of using downstream question answering as the only proxy.
Summary
FEB-30 inspects the store after each fact change instead of using downstream question answering as the only proxy.
Question-answering accuracy can blame a good memory for a retrieval failure or reward a broken memory when generation guesses correctly. FEB-30 inverts the usual benchmark: known fact-evolution trajectories generate the documents, emails and conversations, producing deterministic ground truth for direct store inspection. On one representative system, intrinsic and downstream evaluation differed by 16 points on average, with 44 percent of trajectories crossing the authors' meaningful-divergence threshold. The system classified create, update, reinforce and supersede operations correctly only 20 percent of the time. Thirty trajectories are not a comprehensive memory benchmark, but the result exposes failures that answer-only scoring cannot see.
Why it matters
FEB-30 inspects the store after each fact change instead of using downstream question answering as the only proxy.
Limits and context
- The system classified create, update, reinforce and supersede operations correctly only 20 percent of the time.
- Thirty trajectories are not a comprehensive memory benchmark, but the result exposes failures that answer-only scoring cannot see.
Key claims
FEB-30 inspects the store after each fact change instead of using downstream question answering as the only proxy.
Qualification: The system classified create, update, reinforce and supersede operations correctly only 20 percent of the time.
Evidence: source-2026-10-17-016
Sources
- arXiv preprint 2610.22209, original v1arXiv · primary research
Corrections
No corrections have been recorded for this story.