benchmarks evals
The Agent Had Spare CPUs and Still Left Work Waiting
AOS-Bench turns coding agents into workload managers; under heavy streams, every tested system lost 13 to 23 success points.
Summary
AOS-Bench turns coding agents into workload managers; under heavy streams, every tested system lost 13 to 23 success points.
Persistent assistants must decide when to accept, overlap and schedule incoming jobs under shared CPU and memory limits. AOS-Bench compares that stream behavior with the same verified tasks run one at a time across five model-harness pairs. Heavy workloads cost every system 13 to 23 success points, while two systems separated by only 0.6 points in isolation delivered 54 and 68 percent of the same medium-workload tasks. Serial dispatch left 73 percent of heavy tasks unclaimed, yet elaborate policies were not automatically better: fixed concurrency of two came within four points of the best agent-managed policy. Median CPU use remained below 30 percent while tasks waited, locating much of the failure in intake and scheduling rather than raw compute.
Why it matters
AOS-Bench turns coding agents into workload managers; under heavy streams, every tested system lost 13 to 23 success points.
Limits and context
- Heavy workloads cost every system 13 to 23 success points, while two systems separated by only 0.6 points in isolation delivered 54 and 68 percent of the same medium-workload tasks.
- Serial dispatch left 73 percent of heavy tasks unclaimed, yet elaborate policies were not automatically better: fixed concurrency of two came within four points of the best agent-managed policy.
Key claims
AOS-Bench turns coding agents into workload managers; under heavy streams, every tested system lost 13 to 23 success points.
Qualification: Heavy workloads cost every system 13 to 23 success points, while two systems separated by only 0.6 points in isolation delivered 54 and 68 percent of the same medium-workload tasks.
Evidence: source-2026-10-27-003
Sources
- arXiv preprint 2610.33178, original v1arXiv · primary research
Corrections
No corrections have been recorded for this story.