developer tools
Code Became the Better Tool Call
Across fourteen models, typed programs usually matched or beat rigid JSON calls—and held up better when the surrounding context decayed.
Summary
Across fourteen models, typed programs usually matched or beat rigid JSON calls—and held up better when the surrounding context decayed.
Researchers compared programmatic tool calling, where a model writes against typed Python stubs and receives execution results in one turn, with native JSON tool calls across 14 language models on Berkeley Function Calling Leaderboard v4. The programmatic approach matched or exceeded JSON in 11 models; the authors report a 10.6 percent gain for the GPT-5.6 family, parity or improvement for 13 models under parallel fan-out, and stable performance in a context-rot test where the JSON baseline fell 2.3 percent on average. These are author-reported benchmark results, not proof that arbitrary generated code is safe or cheaper to execute in production.
Why it matters
Across fourteen models, typed programs usually matched or beat rigid JSON calls—and held up better when the surrounding context decayed.
Limits and context
- These are author-reported benchmark results, not proof that arbitrary generated code is safe or cheaper to execute in production.
Key claims
Across fourteen models, typed programs usually matched or beat rigid JSON calls—and held up better when the surrounding context decayed.
Qualification: These are author-reported benchmark results, not proof that arbitrary generated code is safe or cheaper to execute in production.
Evidence: source-2026-08-08-001
Sources
- arXiv preprint 2608.06370arXiv · primary research
Corrections
No corrections have been recorded for this story.