benchmarks evals
Tool Diversity Beats More of the Same Task
Across 458,752 synthetic tasks and 849 trained models, environment composition predicted how tool-use agents generalized beyond training.
Summary
Across 458,752 synthetic tasks and 849 trained models, environment composition predicted how tool-use agents generalized beyond training.
SynthAPI generates tool-use environments while independently varying tools, tasks and underlying databases. Training 849 models across four sizes on 458,752 tasks produced a scaling law linking downstream generalization to both data volume and composition. As budgets grew, adding tool and database diversity helped more than merely adding new tasks, and harder data produced larger gains. Agents trained with those allocation rules outperformed prior synthetic-environment approaches on four benchmarks. The relationship is empirical for the generated environments and evaluated models, not a universal law for every agent deployment.
Why it matters
Across 458,752 synthetic tasks and 849 trained models, environment composition predicted how tool-use agents generalized beyond training.
Limits and context
- The relationship is empirical for the generated environments and evaluated models, not a universal law for every agent deployment.
Key claims
Across 458,752 synthetic tasks and 849 trained models, environment composition predicted how tool-use agents generalized beyond training.
Qualification: The relationship is empirical for the generated environments and evaluated models, not a universal law for every agent deployment.
Evidence: source-2026-10-29-003
Sources
- arXiv preprint 2610.37648, original v1arXiv · primary research
Corrections
No corrections have been recorded for this story.