benchmarks evals
Agentic Search Missed the Papers Scientists Said Inspired Them
Embedding retrieval reached 0.48 Recall@20; an agent calling that retriever scored 0.42.
Summary
Embedding retrieval reached 0.48 Recall@20; an agent calling that retriever scored 0.42.
ScholarCatalyst asked 184 lead authors of 207 computer-science papers which earlier work did or could have advanced their projects, with rationales and a time-bounded literature pool. Agentic search reached 0.42 Recall@20, below embedding retrieval at 0.48, while the strongest tested agent reached 0.51 despite possible training exposure to completed papers. The benchmark turns scientific inspiration into an author-judged retrieval task; it does not establish that the labeled papers were the only valid catalysts.
Why it matters
Embedding retrieval reached 0.48 Recall@20; an agent calling that retriever scored 0.42.
Limits and context
- The benchmark turns scientific inspiration into an author-judged retrieval task; it does not establish that the labeled papers were the only valid catalysts.
Key claims
Embedding retrieval reached 0.48 Recall@20; an agent calling that retriever scored 0.42.
Qualification: The benchmark turns scientific inspiration into an author-judged retrieval task; it does not establish that the labeled papers were the only valid catalysts.
Evidence: source-2026-10-02-004
Sources
- arXiv preprint 2610.02202arXiv · primary research
Corrections
No corrections have been recorded for this story.