infrastructure
Holding Back a Ready Agent Turn Cut Tail Latency
A scheduler reduced P95 workflow completion time by as much as 3.5× under contention by controlling when ready turns entered the queue.
Summary
A scheduler reduced P95 workflow completion time by as much as 3.5× under contention by controlling when ready turns entered the queue.
Agent workflows alternate model turns with tools, so releasing every ready turn immediately can fill a busy inference queue with work that can no longer be reordered. The proposed scheduler jointly chooses the next ready turn and the amount of released-but-unfinished work, optimizing a mean-plus-tail-risk objective with online work estimates. On real software-engineering agent traces, it matched eager release under light load and cut P95 flow time by up to 3.5 times under contention. The gain depends on the evaluated traces and arrival rates, but it identifies workflow release policy as a separate lever from model speed.
Why it matters
A scheduler reduced P95 workflow completion time by as much as 3.5× under contention by controlling when ready turns entered the queue.
Limits and context
No additional limitation was separately recorded.
Key claims
A scheduler reduced P95 workflow completion time by as much as 3.5× under contention by controlling when ready turns entered the queue.
Evidence: source-2026-09-11-004
Sources
- arXiv preprint 2609.10964arXiv · primary research
Corrections
No corrections have been recorded for this story.