developer tools
The Best Coding Base Model Could Be Ranked Before It Learned the Harness
Three screens evaluated decisive actions inside successful trajectories instead of demanding cold-start tool use from an untrained checkpoint.
Summary
Three screens evaluated decisive actions inside successful trajectories instead of demanding cold-start tool use from an untrained checkpoint.
Base checkpoints often fail an agent benchmark because they cannot yet format tool calls, hiding whether their weights contain useful coding behavior. This work replays successful post-training trajectories, finds the first cumulative patch that flips tests from failing to passing and evaluates base models at that decisive step. Bits-per-byte on the certified action, multiple-choice discrimination against verifier-rejected alternatives and prefix-conditioned pass-at-k all ranked ten base/post-trained model pairs in close agreement with eventual SWE-bench Verified performance. The screens still depend on successful trajectories and a verifier, but they may prevent expensive post-training runs on checkpoints whose apparent weakness is only harness fluency—or whose underlying coding distribution truly lacks the needed action.
Why it matters
Three screens evaluated decisive actions inside successful trajectories instead of demanding cold-start tool use from an untrained checkpoint.
Limits and context
- Base checkpoints often fail an agent benchmark because they cannot yet format tool calls, hiding whether their weights contain useful coding behavior.
- The screens still depend on successful trajectories and a verifier, but they may prevent expensive post-training runs on checkpoints whose apparent weakness is only harness fluency—or whose underlying coding distribution truly lacks the needed action.
Key claims
Three screens evaluated decisive actions inside successful trajectories instead of demanding cold-start tool use from an untrained checkpoint.
Qualification: Base checkpoints often fail an agent benchmark because they cannot yet format tool calls, hiding whether their weights contain useful coding behavior.
Evidence: source-2026-10-08-013
Sources
- arXiv preprint 2610.10478arXiv · primary research
Corrections
No corrections have been recorded for this story.