benchmarks evals
The Best Agent Delivered 14 of 95 Customer Jobs
FDE-Gym replaces tidy tickets with ambiguous requirements, scripted stakeholders and production-like systems—and scores the customer's outcome.
Summary
FDE-Gym replaces tidy tickets with ambiguous requirements, scripted stakeholders and production-like systems—and scores the customer's outcome.
Agent benchmarks usually begin after somebody has already turned a messy need into a clean task. FDE-Gym restores the missing work: 95 delivery projects from 20 customer scenarios across 18 sectors require models to question stakeholders, modify live-like systems and uncover unstated constraints. The best of 13 evaluated configurations averaged 0.56 on the benchmark's normalized business-uplift score and cleared the authors' success threshold on only 14 tasks. Even when it found every key fact on 23 tasks, it reached at least 0.8 uplift on just 10. The simulated customers and grading system do not reproduce every deployment, but the result isolates a real distinction between solving a ticket and delivering a useful system.
Why it matters
FDE-Gym replaces tidy tickets with ambiguous requirements, scripted stakeholders and production-like systems—and scores the customer's outcome.
Limits and context
- The best of 13 evaluated configurations averaged 0.56 on the benchmark's normalized business-uplift score and cleared the authors' success threshold on only 14 tasks.
- The simulated customers and grading system do not reproduce every deployment, but the result isolates a real distinction between solving a ticket and delivering a useful system.
Key claims
FDE-Gym replaces tidy tickets with ambiguous requirements, scripted stakeholders and production-like systems—and scores the customer's outcome.
Qualification: The best of 13 evaluated configurations averaged 0.56 on the benchmark's normalized business-uplift score and cleared the authors' success threshold on only 14 tasks.
Evidence: source-2026-10-16-005
Sources
- arXiv preprint 2610.22350, original v1arXiv / submitting research authors · primary research
Corrections
No corrections have been recorded for this story.