benchmarks evals
Repository Migration Exposes the Agent Gap
XStack asks coding agents to move complete PyTorch projects on NVIDIA GPUs into JAX repositories running on Google TPUs.
Summary
XStack asks coding agents to move complete PyTorch projects on NVIDIA GPUs into JAX repositories running on Google TPUs.
Each task begins with a PyTorch repository and a clean TPU machine; a migration passes only when it reproduces authenticated criteria derived from the source project’s published result. In the released benchmark, the strongest reported agent passes 75 percent, GPT-5.6-sol passes 45 percent and every other tested agent reaches at most 40 percent. The scores describe this benchmark’s repositories, infrastructure and grading—not all framework migration work.
Why it matters
XStack asks coding agents to move complete PyTorch projects on NVIDIA GPUs into JAX repositories running on Google TPUs.
Limits and context
- Each task begins with a PyTorch repository and a clean TPU machine; a migration passes only when it reproduces authenticated criteria derived from the source project’s published result.
- The scores describe this benchmark’s repositories, infrastructure and grading—not all framework migration work.
Key claims
XStack asks coding agents to move complete PyTorch projects on NVIDIA GPUs into JAX repositories running on Google TPUs.
Qualification: Each task begins with a PyTorch repository and a clean TPU machine; a migration passes only when it reproduces authenticated criteria derived from the source project’s published result.
Evidence: source-2026-11-06-004
Sources
- arXiv preprint 2611.08472, original v1arXiv · primary research
Corrections
No corrections have been recorded for this story.